2026-02-20T11:07:26.312 INFO:root:teuthology version: 1.2.4.dev11+ge6d9099ce 2026-02-20T11:07:26.328 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-20T11:07:26.410 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/jjperez-2026-02-20_10:21:31-crimson-rados-wip-perezjos-crimson-only-19-02-2026-PR64439-distro-debug-trial/60477 branch: wip-perezjos-crimson-only-19-02-2026-PR64439 description: crimson-rados/thrash/{0-crimson_install 0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} clusters/{crimson-fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm} thrashers/{default} thrashosds-health workloads/pool-snaps-few-objects} email: null first_in_suite: false flavor: debug job_id: '60477' kernel: branch: distro kdb: 1 sha1: distro last_in_suite: false machine_type: trial name: jjperez-2026-02-20_10:21:31-crimson-rados-wip-perezjos-crimson-only-19-02-2026-PR64439-distro-debug-trial no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-perezjos-crimson-only-19-02-2026-PR64439 ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc osd_max_pg_log_entries: 2 osd_min_pg_log_entries: 1 osd_pg_log_trim_min: 0 osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 osd pool default crimson: true paxos service trim min: 10 osd: crimson cpu num: 4 crimson osd obc lru size: 10 debug ms: 20 debug osd: 20 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore omap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd max markdown count: 1000 osd objectstore: seastore osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 osd target transaction size: 5 seastore cachepin size pershard: 64M seastore main device type: RANDOM_BLOCK_SSD seastore max concurrent transactions: 8 crimson_compat: true flavor: debug log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - SLOW_OPS - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub - Degraded data redundancy - is down - osds down - pg .*? is .*?degraded.*?, acting - pg .*? is stuck - pg degraded - PG_BACKFILL_FULL - Low space hindering backfill .*? backfill_toofull - OSD_ROOT_DOWN - pgs degraded - pgs undersized - is active.*backfill_toofull.* sha1: 0c3148ca19c827ca2a7a9b92c2554d99507fd1f4 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} conf: osd: osd deep scrub update digest min age: 0 install: ceph: flavor: debug sha1: 0c3148ca19c827ca2a7a9b92c2554d99507fd1f4 thrashosds: min_in: 4 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: 445d5c9ba18a4040e70e25303230c36c77773acb owner: scheduled_jjperez@soko04.front.sepia.ceph.com priority: 101 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - osd.0 - osd.1 - client.0 - node-exporter.a - - mgr.x - osd.2 - osd.3 - client.1 - prometheus.a - node-exporter.b seed: 1736 sha1: 0c3148ca19c827ca2a7a9b92c2554d99507fd1f4 sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: 445d5c9ba18a4040e70e25303230c36c77773acb targets: trial043.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC9KROEZxDEnRzwsYioaSBZAaVX82KUX6mpdTEBGlCxJ1/oKHlwjv0iX6bqUNS8vM7byIN+eIeisLxGilvuexMo= trial109.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMbRy1ZZTQoDxKIW3KAcBEW3M+EMItJWa8K9r3kapqYEYab49qlHZdbYtkkIjSY3EjWrQ0PL7DjwuV79dEIDcOE= tasks: - install: extra_packages: deb: - ceph-osd-crimson - ceph-osd-crimson-dbg rpm: - ceph-osd-crimson - ceph-osd-crimson-debuginfo - pexec: all: - sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson - install: null - ceph: conf: osd: debug monc: 20 - thrashosds: ceph_objectstore_tool: false chance_inject_pause_short: 0 chance_reset_purged_snaps_last: 0 chance_thrash_cluster_full: 0 chance_thrash_pg_upmap: 0 chance_trim_stale_osdmaps: 0 dump_ops_enable: false min_in: 3 noscrub_toggle_delay: 0 reweight_osd: 0 sighup_delay: 0 thrash_primary_affinity: false timeout: 2400 - rados: clients: - client.0 objects: 50 op_weights: copy_from: 0 delete: 50 read: 100 rollback: 0 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 pool_snaps: true teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: reimage-ssh-down teuthology_sha1: e6d9099ce03dc21193fda653159ec82465c3a94a timestamp: 2026-02-20_10:21:31 tube: trial user: jjperez verbose: false worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.1540699 2026-02-20T11:07:26.410 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa; will attempt to use it 2026-02-20T11:07:26.411 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks 2026-02-20T11:07:26.411 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-20T11:07:26.412 INFO:teuthology.task.internal:Checking packages... 2026-02-20T11:07:26.413 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'debug' and ceph hash '0c3148ca19c827ca2a7a9b92c2554d99507fd1f4' 2026-02-20T11:07:26.413 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-20T11:07:26.413 INFO:teuthology.packaging:ref: None 2026-02-20T11:07:26.413 INFO:teuthology.packaging:tag: None 2026-02-20T11:07:26.413 INFO:teuthology.packaging:branch: wip-perezjos-crimson-only-19-02-2026-PR64439 2026-02-20T11:07:26.413 INFO:teuthology.packaging:sha1: 0c3148ca19c827ca2a7a9b92c2554d99507fd1f4 2026-02-20T11:07:26.413 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&ref=wip-perezjos-crimson-only-19-02-2026-PR64439 2026-02-20T11:07:26.614 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5357.g0c3148ca 2026-02-20T11:07:26.615 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-20T11:07:26.627 INFO:teuthology.task.internal:no buildpackages task found 2026-02-20T11:07:26.627 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-20T11:07:26.640 INFO:teuthology.task.internal:Saving configuration 2026-02-20T11:07:26.656 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-20T11:07:26.671 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-20T11:07:26.751 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial043.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/jjperez-2026-02-20_10:21:31-crimson-rados-wip-perezjos-crimson-only-19-02-2026-PR64439-distro-debug-trial/60477', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-20 11:03:59.752449', 'locked_by': 'scheduled_jjperez@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC9KROEZxDEnRzwsYioaSBZAaVX82KUX6mpdTEBGlCxJ1/oKHlwjv0iX6bqUNS8vM7byIN+eIeisLxGilvuexMo='} 2026-02-20T11:07:26.814 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial109.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/jjperez-2026-02-20_10:21:31-crimson-rados-wip-perezjos-crimson-only-19-02-2026-PR64439-distro-debug-trial/60477', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-20 11:03:59.753059', 'locked_by': 'scheduled_jjperez@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMbRy1ZZTQoDxKIW3KAcBEW3M+EMItJWa8K9r3kapqYEYab49qlHZdbYtkkIjSY3EjWrQ0PL7DjwuV79dEIDcOE='} 2026-02-20T11:07:26.814 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-20T11:07:26.825 INFO:teuthology.task.internal:roles: ubuntu@trial043.front.sepia.ceph.com - ['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'] 2026-02-20T11:07:26.825 INFO:teuthology.task.internal:roles: ubuntu@trial109.front.sepia.ceph.com - ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b'] 2026-02-20T11:07:26.825 INFO:teuthology.run_tasks:Running task console_log... 2026-02-20T11:07:26.959 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7c11eae258a0>, signals=[15]) 2026-02-20T11:07:26.959 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-20T11:07:26.970 INFO:teuthology.task.internal:Opening connections... 2026-02-20T11:07:26.970 DEBUG:teuthology.task.internal:connecting to ubuntu@trial043.front.sepia.ceph.com 2026-02-20T11:07:26.971 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-20T11:07:27.030 DEBUG:teuthology.task.internal:connecting to ubuntu@trial109.front.sepia.ceph.com 2026-02-20T11:07:27.030 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial109.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-20T11:07:27.093 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-20T11:07:27.105 DEBUG:teuthology.orchestra.run.trial043:> uname -m 2026-02-20T11:07:27.120 INFO:teuthology.orchestra.run.trial043.stdout:x86_64 2026-02-20T11:07:27.120 DEBUG:teuthology.orchestra.run.trial043:> cat /etc/os-release 2026-02-20T11:07:27.176 INFO:teuthology.orchestra.run.trial043.stdout:NAME="CentOS Stream" 2026-02-20T11:07:27.176 INFO:teuthology.orchestra.run.trial043.stdout:VERSION="9" 2026-02-20T11:07:27.176 INFO:teuthology.orchestra.run.trial043.stdout:ID="centos" 2026-02-20T11:07:27.176 INFO:teuthology.orchestra.run.trial043.stdout:ID_LIKE="rhel fedora" 2026-02-20T11:07:27.176 INFO:teuthology.orchestra.run.trial043.stdout:VERSION_ID="9" 2026-02-20T11:07:27.176 INFO:teuthology.orchestra.run.trial043.stdout:PLATFORM_ID="platform:el9" 2026-02-20T11:07:27.176 INFO:teuthology.orchestra.run.trial043.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-20T11:07:27.176 INFO:teuthology.orchestra.run.trial043.stdout:ANSI_COLOR="0;31" 2026-02-20T11:07:27.176 INFO:teuthology.orchestra.run.trial043.stdout:LOGO="fedora-logo-icon" 2026-02-20T11:07:27.176 INFO:teuthology.orchestra.run.trial043.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-20T11:07:27.176 INFO:teuthology.orchestra.run.trial043.stdout:HOME_URL="https://centos.org/" 2026-02-20T11:07:27.176 INFO:teuthology.orchestra.run.trial043.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-20T11:07:27.176 INFO:teuthology.orchestra.run.trial043.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-20T11:07:27.176 INFO:teuthology.orchestra.run.trial043.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-20T11:07:27.177 INFO:teuthology.lock.ops:Updating trial043.front.sepia.ceph.com on lock server 2026-02-20T11:07:27.251 DEBUG:teuthology.orchestra.run.trial109:> uname -m 2026-02-20T11:07:27.266 INFO:teuthology.orchestra.run.trial109.stdout:x86_64 2026-02-20T11:07:27.266 DEBUG:teuthology.orchestra.run.trial109:> cat /etc/os-release 2026-02-20T11:07:27.321 INFO:teuthology.orchestra.run.trial109.stdout:NAME="CentOS Stream" 2026-02-20T11:07:27.322 INFO:teuthology.orchestra.run.trial109.stdout:VERSION="9" 2026-02-20T11:07:27.322 INFO:teuthology.orchestra.run.trial109.stdout:ID="centos" 2026-02-20T11:07:27.322 INFO:teuthology.orchestra.run.trial109.stdout:ID_LIKE="rhel fedora" 2026-02-20T11:07:27.322 INFO:teuthology.orchestra.run.trial109.stdout:VERSION_ID="9" 2026-02-20T11:07:27.322 INFO:teuthology.orchestra.run.trial109.stdout:PLATFORM_ID="platform:el9" 2026-02-20T11:07:27.322 INFO:teuthology.orchestra.run.trial109.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-20T11:07:27.322 INFO:teuthology.orchestra.run.trial109.stdout:ANSI_COLOR="0;31" 2026-02-20T11:07:27.322 INFO:teuthology.orchestra.run.trial109.stdout:LOGO="fedora-logo-icon" 2026-02-20T11:07:27.322 INFO:teuthology.orchestra.run.trial109.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-20T11:07:27.322 INFO:teuthology.orchestra.run.trial109.stdout:HOME_URL="https://centos.org/" 2026-02-20T11:07:27.322 INFO:teuthology.orchestra.run.trial109.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-20T11:07:27.323 INFO:teuthology.orchestra.run.trial109.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-20T11:07:27.323 INFO:teuthology.orchestra.run.trial109.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-20T11:07:27.323 INFO:teuthology.lock.ops:Updating trial109.front.sepia.ceph.com on lock server 2026-02-20T11:07:27.401 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-20T11:07:27.413 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-20T11:07:27.424 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-20T11:07:27.424 DEBUG:teuthology.orchestra.run.trial043:> test '!' -e /home/ubuntu/cephtest 2026-02-20T11:07:27.426 DEBUG:teuthology.orchestra.run.trial109:> test '!' -e /home/ubuntu/cephtest 2026-02-20T11:07:27.440 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-20T11:07:27.451 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-20T11:07:27.451 DEBUG:teuthology.orchestra.run.trial043:> test -z $(ls -A /var/lib/ceph) 2026-02-20T11:07:27.482 DEBUG:teuthology.orchestra.run.trial109:> test -z $(ls -A /var/lib/ceph) 2026-02-20T11:07:27.498 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-20T11:07:27.664 INFO:teuthology.run_tasks:Running task kernel... 2026-02-20T11:07:27.686 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'} 2026-02-20T11:07:27.686 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}, 'mgr.x': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, timeout 300 2026-02-20T11:07:27.686 DEBUG:teuthology.orchestra.run.trial043:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-20T11:07:27.686 DEBUG:teuthology.orchestra.run.trial109:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-20T11:07:27.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-20T11:07:27.701 DEBUG:teuthology.orchestra.run.trial043:> uname -r 2026-02-20T11:07:27.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-20T11:07:27.701 DEBUG:teuthology.orchestra.run.trial109:> uname -r 2026-02-20T11:07:27.758 INFO:teuthology.orchestra.run.trial043.stdout:5.14.0-681.el9.x86_64 2026-02-20T11:07:27.758 INFO:teuthology.task.kernel:Running kernel on trial043: 5.14.0-681.el9.x86_64 2026-02-20T11:07:27.758 DEBUG:teuthology.orchestra.run.trial043:> sudo yum install -y kernel 2026-02-20T11:07:27.759 INFO:teuthology.orchestra.run.trial109.stdout:5.14.0-681.el9.x86_64 2026-02-20T11:07:27.759 INFO:teuthology.task.kernel:Running kernel on trial109: 5.14.0-681.el9.x86_64 2026-02-20T11:07:27.759 DEBUG:teuthology.orchestra.run.trial109:> sudo yum install -y kernel 2026-02-20T11:07:28.835 INFO:teuthology.orchestra.run.trial109.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.9 MB 00:00 2026-02-20T11:07:31.367 INFO:teuthology.orchestra.run.trial043.stdout:CentOS Stream 9 - BaseOS 2.8 MB/s | 8.9 MB 00:03 2026-02-20T11:07:32.040 INFO:teuthology.orchestra.run.trial109.stdout:CentOS Stream 9 - AppStream 10 MB/s | 27 MB 00:02 2026-02-20T11:07:35.034 INFO:teuthology.orchestra.run.trial043.stdout:CentOS Stream 9 - AppStream 8.6 MB/s | 27 MB 00:03 2026-02-20T11:07:35.541 INFO:teuthology.orchestra.run.trial109.stdout:CentOS Stream 9 - CRB 6.9 MB/s | 7.8 MB 00:01 2026-02-20T11:07:36.728 INFO:teuthology.orchestra.run.trial109.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 20 kB 00:00 2026-02-20T11:07:38.044 INFO:teuthology.orchestra.run.trial109.stdout:Extra Packages for Enterprise Linux 16 MB/s | 20 MB 00:01 2026-02-20T11:07:39.142 INFO:teuthology.orchestra.run.trial043.stdout:CentOS Stream 9 - CRB 4.5 MB/s | 7.8 MB 00:01 2026-02-20T11:07:40.031 INFO:teuthology.orchestra.run.trial043.stdout:CentOS Stream 9 - Extras packages 113 kB/s | 20 kB 00:00 2026-02-20T11:07:40.809 INFO:teuthology.orchestra.run.trial043.stdout:Extra Packages for Enterprise Linux 28 MB/s | 20 MB 00:00 2026-02-20T11:07:41.211 INFO:teuthology.orchestra.run.trial109.stdout:lab-extras 2.7 MB/s | 50 kB 00:00 2026-02-20T11:07:42.223 INFO:teuthology.orchestra.run.trial109.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-20T11:07:42.223 INFO:teuthology.orchestra.run.trial109.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-20T11:07:42.242 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:07:42.243 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:07:42.243 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:07:42.268 DEBUG:teuthology.orchestra.run.trial109:> echo no | sudo yum reinstall kernel || true 2026-02-20T11:07:42.606 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:00:01 ago on Fri 20 Feb 2026 11:07:41 AM UTC. 2026-02-20T11:07:42.686 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:07:42.686 INFO:teuthology.orchestra.run.trial109.stdout:================================================================================ 2026-02-20T11:07:42.687 INFO:teuthology.orchestra.run.trial109.stdout: Package Architecture Version Repository Size 2026-02-20T11:07:42.687 INFO:teuthology.orchestra.run.trial109.stdout:================================================================================ 2026-02-20T11:07:42.687 INFO:teuthology.orchestra.run.trial109.stdout:Reinstalling: 2026-02-20T11:07:42.687 INFO:teuthology.orchestra.run.trial109.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-20T11:07:42.687 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:07:42.687 INFO:teuthology.orchestra.run.trial109.stdout:Transaction Summary 2026-02-20T11:07:42.687 INFO:teuthology.orchestra.run.trial109.stdout:================================================================================ 2026-02-20T11:07:42.687 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:07:42.687 INFO:teuthology.orchestra.run.trial109.stdout:Total download size: 803 k 2026-02-20T11:07:42.687 INFO:teuthology.orchestra.run.trial109.stdout:Installed size: 0 2026-02-20T11:07:42.687 INFO:teuthology.orchestra.run.trial109.stderr:Operation aborted. 2026-02-20T11:07:42.708 INFO:teuthology.orchestra.run.trial109.stdout:Is this ok [y/N]: 2026-02-20T11:07:42.708 DEBUG:teuthology.orchestra.run.trial109:> sudo yum reinstall -y kernel || true 2026-02-20T11:07:43.010 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:00:02 ago on Fri 20 Feb 2026 11:07:41 AM UTC. 2026-02-20T11:07:43.089 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:07:43.090 INFO:teuthology.orchestra.run.trial109.stdout:================================================================================ 2026-02-20T11:07:43.090 INFO:teuthology.orchestra.run.trial109.stdout: Package Architecture Version Repository Size 2026-02-20T11:07:43.090 INFO:teuthology.orchestra.run.trial109.stdout:================================================================================ 2026-02-20T11:07:43.090 INFO:teuthology.orchestra.run.trial109.stdout:Reinstalling: 2026-02-20T11:07:43.090 INFO:teuthology.orchestra.run.trial109.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-20T11:07:43.090 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:07:43.090 INFO:teuthology.orchestra.run.trial109.stdout:Transaction Summary 2026-02-20T11:07:43.090 INFO:teuthology.orchestra.run.trial109.stdout:================================================================================ 2026-02-20T11:07:43.090 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:07:43.090 INFO:teuthology.orchestra.run.trial109.stdout:Total download size: 803 k 2026-02-20T11:07:43.090 INFO:teuthology.orchestra.run.trial109.stdout:Installed size: 0 2026-02-20T11:07:43.090 INFO:teuthology.orchestra.run.trial109.stdout:Downloading Packages: 2026-02-20T11:07:43.932 INFO:teuthology.orchestra.run.trial043.stdout:lab-extras 2.6 MB/s | 50 kB 00:00 2026-02-20T11:07:44.164 INFO:teuthology.orchestra.run.trial109.stdout:kernel-5.14.0-681.el9.x86_64.rpm 1.1 MB/s | 803 kB 00:00 2026-02-20T11:07:44.164 INFO:teuthology.orchestra.run.trial109.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:07:44.165 INFO:teuthology.orchestra.run.trial109.stdout:Total 748 kB/s | 803 kB 00:01 2026-02-20T11:07:44.175 INFO:teuthology.orchestra.run.trial109.stdout:Running transaction check 2026-02-20T11:07:44.208 INFO:teuthology.orchestra.run.trial109.stdout:Transaction check succeeded. 2026-02-20T11:07:44.208 INFO:teuthology.orchestra.run.trial109.stdout:Running transaction test 2026-02-20T11:07:44.213 INFO:teuthology.orchestra.run.trial109.stdout:Transaction test succeeded. 2026-02-20T11:07:44.213 INFO:teuthology.orchestra.run.trial109.stdout:Running transaction 2026-02-20T11:07:44.237 INFO:teuthology.orchestra.run.trial109.stdout: Preparing : 1/1 2026-02-20T11:07:44.241 INFO:teuthology.orchestra.run.trial109.stdout: Reinstalling : kernel-5.14.0-681.el9.x86_64 1/2 2026-02-20T11:07:44.241 INFO:teuthology.orchestra.run.trial109.stdout: Cleanup : kernel-5.14.0-681.el9.x86_64 2/2 2026-02-20T11:07:44.351 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: kernel-5.14.0-681.el9.x86_64 2/2 2026-02-20T11:07:44.351 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 1/2 2026-02-20T11:07:44.404 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 2/2 2026-02-20T11:07:44.404 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:07:44.404 INFO:teuthology.orchestra.run.trial109.stdout:Reinstalled: 2026-02-20T11:07:44.404 INFO:teuthology.orchestra.run.trial109.stdout: kernel-5.14.0-681.el9.x86_64 2026-02-20T11:07:44.405 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:07:44.405 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:07:44.433 DEBUG:teuthology.orchestra.run.trial109:> rpm -q kernel | sort -rV | head -n 1 2026-02-20T11:07:44.457 INFO:teuthology.orchestra.run.trial109.stdout:kernel-5.14.0-681.el9.x86_64 2026-02-20T11:07:44.459 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-681.el9.x86_64 2026-02-20T11:07:44.459 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2026-02-20T11:07:44.459 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2026-02-20T11:07:44.459 DEBUG:teuthology.orchestra.run.trial109:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-20T11:07:44.523 INFO:teuthology.orchestra.run.trial109.stdout:ttyS1 2026-02-20T11:07:44.549 DEBUG:teuthology.parallel:result is None 2026-02-20T11:07:44.946 INFO:teuthology.orchestra.run.trial043.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-20T11:07:44.946 INFO:teuthology.orchestra.run.trial043.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-20T11:07:44.965 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:07:44.966 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:07:44.966 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:07:44.989 DEBUG:teuthology.orchestra.run.trial043:> echo no | sudo yum reinstall kernel || true 2026-02-20T11:07:45.287 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:02 ago on Fri 20 Feb 2026 11:07:43 AM UTC. 2026-02-20T11:07:45.366 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:07:45.368 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-20T11:07:45.368 INFO:teuthology.orchestra.run.trial043.stdout: Package Architecture Version Repository Size 2026-02-20T11:07:45.368 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-20T11:07:45.368 INFO:teuthology.orchestra.run.trial043.stdout:Reinstalling: 2026-02-20T11:07:45.368 INFO:teuthology.orchestra.run.trial043.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-20T11:07:45.368 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:07:45.368 INFO:teuthology.orchestra.run.trial043.stdout:Transaction Summary 2026-02-20T11:07:45.368 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-20T11:07:45.368 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:07:45.369 INFO:teuthology.orchestra.run.trial043.stdout:Total download size: 803 k 2026-02-20T11:07:45.369 INFO:teuthology.orchestra.run.trial043.stdout:Installed size: 0 2026-02-20T11:07:45.369 INFO:teuthology.orchestra.run.trial043.stderr:Operation aborted. 2026-02-20T11:07:45.387 INFO:teuthology.orchestra.run.trial043.stdout:Is this ok [y/N]: 2026-02-20T11:07:45.388 DEBUG:teuthology.orchestra.run.trial043:> sudo yum reinstall -y kernel || true 2026-02-20T11:07:45.686 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:02 ago on Fri 20 Feb 2026 11:07:43 AM UTC. 2026-02-20T11:07:45.767 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:07:45.767 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-20T11:07:45.767 INFO:teuthology.orchestra.run.trial043.stdout: Package Architecture Version Repository Size 2026-02-20T11:07:45.767 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-20T11:07:45.767 INFO:teuthology.orchestra.run.trial043.stdout:Reinstalling: 2026-02-20T11:07:45.767 INFO:teuthology.orchestra.run.trial043.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-20T11:07:45.768 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:07:45.768 INFO:teuthology.orchestra.run.trial043.stdout:Transaction Summary 2026-02-20T11:07:45.768 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-20T11:07:45.768 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:07:45.768 INFO:teuthology.orchestra.run.trial043.stdout:Total download size: 803 k 2026-02-20T11:07:45.768 INFO:teuthology.orchestra.run.trial043.stdout:Installed size: 0 2026-02-20T11:07:45.768 INFO:teuthology.orchestra.run.trial043.stdout:Downloading Packages: 2026-02-20T11:07:46.248 INFO:teuthology.orchestra.run.trial043.stdout:kernel-5.14.0-681.el9.x86_64.rpm 4.0 MB/s | 803 kB 00:00 2026-02-20T11:07:46.249 INFO:teuthology.orchestra.run.trial043.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:07:46.249 INFO:teuthology.orchestra.run.trial043.stdout:Total 1.6 MB/s | 803 kB 00:00 2026-02-20T11:07:46.259 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction check 2026-02-20T11:07:46.294 INFO:teuthology.orchestra.run.trial043.stdout:Transaction check succeeded. 2026-02-20T11:07:46.294 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction test 2026-02-20T11:07:46.299 INFO:teuthology.orchestra.run.trial043.stdout:Transaction test succeeded. 2026-02-20T11:07:46.299 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction 2026-02-20T11:07:46.323 INFO:teuthology.orchestra.run.trial043.stdout: Preparing : 1/1 2026-02-20T11:07:46.327 INFO:teuthology.orchestra.run.trial043.stdout: Reinstalling : kernel-5.14.0-681.el9.x86_64 1/2 2026-02-20T11:07:46.327 INFO:teuthology.orchestra.run.trial043.stdout: Cleanup : kernel-5.14.0-681.el9.x86_64 2/2 2026-02-20T11:07:46.436 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: kernel-5.14.0-681.el9.x86_64 2/2 2026-02-20T11:07:46.436 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 1/2 2026-02-20T11:07:46.487 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 2/2 2026-02-20T11:07:46.487 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:07:46.487 INFO:teuthology.orchestra.run.trial043.stdout:Reinstalled: 2026-02-20T11:07:46.487 INFO:teuthology.orchestra.run.trial043.stdout: kernel-5.14.0-681.el9.x86_64 2026-02-20T11:07:46.487 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:07:46.487 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:07:46.515 DEBUG:teuthology.orchestra.run.trial043:> rpm -q kernel | sort -rV | head -n 1 2026-02-20T11:07:46.578 INFO:teuthology.orchestra.run.trial043.stdout:kernel-5.14.0-681.el9.x86_64 2026-02-20T11:07:46.579 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-681.el9.x86_64 2026-02-20T11:07:46.579 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2026-02-20T11:07:46.579 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-20T11:07:46.580 DEBUG:teuthology.orchestra.run.trial043:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-20T11:07:46.643 INFO:teuthology.orchestra.run.trial043.stdout:ttyS1 2026-02-20T11:07:46.664 DEBUG:teuthology.parallel:result is None 2026-02-20T11:07:46.664 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-20T11:07:46.678 INFO:teuthology.task.internal:Creating test directory... 2026-02-20T11:07:46.678 DEBUG:teuthology.orchestra.run.trial043:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-20T11:07:46.706 DEBUG:teuthology.orchestra.run.trial109:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-20T11:07:46.724 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-20T11:07:46.735 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-20T11:07:46.748 INFO:teuthology.task.internal:Creating archive directory... 2026-02-20T11:07:46.748 DEBUG:teuthology.orchestra.run.trial043:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-20T11:07:46.762 DEBUG:teuthology.orchestra.run.trial109:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-20T11:07:46.790 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-20T11:07:46.802 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-20T11:07:46.802 DEBUG:teuthology.orchestra.run.trial043:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-20T11:07:46.820 DEBUG:teuthology.orchestra.run.trial109:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-20T11:07:46.842 INFO:teuthology.orchestra.run.trial043.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T11:07:46.845 INFO:teuthology.orchestra.run.trial109.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T11:07:46.851 INFO:teuthology.orchestra.run.trial043.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T11:07:46.853 INFO:teuthology.orchestra.run.trial109.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T11:07:46.855 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-20T11:07:46.867 INFO:teuthology.task.internal:Configuring sudo... 2026-02-20T11:07:46.867 DEBUG:teuthology.orchestra.run.trial043:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-20T11:07:46.894 DEBUG:teuthology.orchestra.run.trial109:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-20T11:07:46.919 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-20T11:07:46.931 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-20T11:07:46.931 DEBUG:teuthology.orchestra.run.trial043:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-20T11:07:46.959 DEBUG:teuthology.orchestra.run.trial109:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-20T11:07:46.975 DEBUG:teuthology.orchestra.run.trial043:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-20T11:07:47.131 DEBUG:teuthology.orchestra.run.trial043:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-20T11:07:47.156 DEBUG:teuthology.orchestra.run.trial043:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-20T11:07:47.292 DEBUG:teuthology.orchestra.run.trial043:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-20T11:07:47.316 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-20T11:07:47.316 DEBUG:teuthology.orchestra.run.trial043:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-20T11:07:47.379 DEBUG:teuthology.orchestra.run.trial109:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-20T11:07:47.474 DEBUG:teuthology.orchestra.run.trial109:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-20T11:07:47.498 DEBUG:teuthology.orchestra.run.trial109:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-20T11:07:47.629 DEBUG:teuthology.orchestra.run.trial109:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-20T11:07:47.653 DEBUG:teuthology.orchestra.run.trial109:> set -ex 2026-02-20T11:07:47.653 DEBUG:teuthology.orchestra.run.trial109:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-20T11:07:47.719 DEBUG:teuthology.orchestra.run.trial043:> sudo service rsyslog restart 2026-02-20T11:07:47.721 DEBUG:teuthology.orchestra.run.trial109:> sudo service rsyslog restart 2026-02-20T11:07:47.748 INFO:teuthology.orchestra.run.trial043.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-20T11:07:47.786 INFO:teuthology.orchestra.run.trial109.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-20T11:07:48.163 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-20T11:07:48.177 INFO:teuthology.task.internal:Starting timer... 2026-02-20T11:07:48.177 INFO:teuthology.run_tasks:Running task pcp... 2026-02-20T11:07:48.202 INFO:teuthology.run_tasks:Running task selinux... 2026-02-20T11:07:48.220 DEBUG:teuthology.orchestra.run.trial043:> sudo service auditd rotate 2026-02-20T11:07:48.250 INFO:teuthology.orchestra.run.trial043.stdout:Rotating logs: 2026-02-20T11:07:48.252 DEBUG:teuthology.orchestra.run.trial109:> sudo service auditd rotate 2026-02-20T11:07:48.281 INFO:teuthology.orchestra.run.trial109.stdout:Rotating logs: 2026-02-20T11:07:48.282 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-20T11:07:48.283 DEBUG:teuthology.orchestra.run.trial043:> /usr/sbin/getenforce 2026-02-20T11:07:48.307 INFO:teuthology.orchestra.run.trial043.stdout:Permissive 2026-02-20T11:07:48.308 DEBUG:teuthology.orchestra.run.trial109:> /usr/sbin/getenforce 2026-02-20T11:07:48.337 INFO:teuthology.orchestra.run.trial109.stdout:Permissive 2026-02-20T11:07:48.337 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial043.front.sepia.ceph.com': 'permissive', 'ubuntu@trial109.front.sepia.ceph.com': 'permissive'} 2026-02-20T11:07:48.338 DEBUG:teuthology.orchestra.run.trial043:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-20T11:07:48.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-20T11:07:48.374 DEBUG:teuthology.orchestra.run.trial109:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-20T11:07:48.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-20T11:07:48.401 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-20T11:07:48.401 DEBUG:teuthology.orchestra.run.trial043:> sudo /usr/sbin/setenforce permissive 2026-02-20T11:07:48.438 DEBUG:teuthology.orchestra.run.trial109:> sudo /usr/sbin/setenforce permissive 2026-02-20T11:07:48.465 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-20T11:07:48.483 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-20T11:07:48.495 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-20T11:07:48.496 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial043.front.sepia.ceph.com,trial109.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-20T11:10:32.063 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial043.front.sepia.ceph.com'), Remote(name='ubuntu@trial109.front.sepia.ceph.com')] 2026-02-20T11:10:32.064 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial043.front.sepia.ceph.com' 2026-02-20T11:10:32.065 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-20T11:10:32.173 DEBUG:teuthology.orchestra.run.trial043:> true 2026-02-20T11:10:32.249 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial043.front.sepia.ceph.com' 2026-02-20T11:10:32.249 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial109.front.sepia.ceph.com' 2026-02-20T11:10:32.250 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial109.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-20T11:10:32.343 DEBUG:teuthology.orchestra.run.trial109:> true 2026-02-20T11:10:32.418 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial109.front.sepia.ceph.com' 2026-02-20T11:10:32.418 INFO:teuthology.run_tasks:Running task clock... 2026-02-20T11:10:32.431 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-20T11:10:32.431 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-20T11:10:32.431 DEBUG:teuthology.orchestra.run.trial043:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-20T11:10:32.434 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-20T11:10:32.434 DEBUG:teuthology.orchestra.run.trial109:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-20T11:10:32.461 INFO:teuthology.orchestra.run.trial043.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-20T11:10:32.473 INFO:teuthology.orchestra.run.trial043.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-20T11:10:32.484 INFO:teuthology.orchestra.run.trial109.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-20T11:10:32.492 INFO:teuthology.orchestra.run.trial043.stderr:sudo: ntpd: command not found 2026-02-20T11:10:32.495 INFO:teuthology.orchestra.run.trial109.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-20T11:10:32.502 INFO:teuthology.orchestra.run.trial043.stdout:506 Cannot talk to daemon 2026-02-20T11:10:32.515 INFO:teuthology.orchestra.run.trial043.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-20T11:10:32.517 INFO:teuthology.orchestra.run.trial109.stderr:sudo: ntpd: command not found 2026-02-20T11:10:32.525 INFO:teuthology.orchestra.run.trial043.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-20T11:10:32.526 INFO:teuthology.orchestra.run.trial109.stdout:506 Cannot talk to daemon 2026-02-20T11:10:32.538 INFO:teuthology.orchestra.run.trial109.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-20T11:10:32.549 INFO:teuthology.orchestra.run.trial109.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-20T11:10:32.574 INFO:teuthology.orchestra.run.trial043.stderr:bash: line 1: ntpq: command not found 2026-02-20T11:10:32.595 INFO:teuthology.orchestra.run.trial109.stderr:bash: line 1: ntpq: command not found 2026-02-20T11:10:32.921 INFO:teuthology.orchestra.run.trial043.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-20T11:10:32.921 INFO:teuthology.orchestra.run.trial043.stdout:=============================================================================== 2026-02-20T11:10:32.922 INFO:teuthology.orchestra.run.trial043.stdout:^? ntp2.zeus.frumentum.media 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-20T11:10:32.922 INFO:teuthology.orchestra.run.trial043.stdout:^? time3.lshiy.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-20T11:10:32.922 INFO:teuthology.orchestra.run.trial043.stdout:^? lithium.constant.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-20T11:10:32.922 INFO:teuthology.orchestra.run.trial043.stdout:^? ip216-250-115-174.pbiaas> 2 6 1 0 -4833us[-4833us] +/- 110ms 2026-02-20T11:10:32.940 INFO:teuthology.orchestra.run.trial109.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-20T11:10:32.940 INFO:teuthology.orchestra.run.trial109.stdout:=============================================================================== 2026-02-20T11:10:32.940 INFO:teuthology.orchestra.run.trial109.stdout:^? 149.28.200.179.vultruser> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-20T11:10:32.940 INFO:teuthology.orchestra.run.trial109.stdout:^? chi2.us.ntp.li 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-20T11:10:32.940 INFO:teuthology.orchestra.run.trial109.stdout:^? 155.248.196.28 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-20T11:10:32.940 INFO:teuthology.orchestra.run.trial109.stdout:^? ntp.netlinkify.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-20T11:10:32.942 INFO:teuthology.run_tasks:Running task install... 2026-02-20T11:10:32.959 DEBUG:teuthology.task.install:project ceph 2026-02-20T11:10:32.959 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': '0c3148ca19c827ca2a7a9b92c2554d99507fd1f4'}} 2026-02-20T11:10:32.959 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': '0c3148ca19c827ca2a7a9b92c2554d99507fd1f4'} 2026-02-20T11:10:32.959 INFO:teuthology.task.install:Using flavor: debug 2026-02-20T11:10:32.968 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-20T11:10:32.968 INFO:teuthology.task.install:extra packages: {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']} 2026-02-20T11:10:32.969 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0c3148ca19c827ca2a7a9b92c2554d99507fd1f4', 'tag': None, 'wait_for_package': False} 2026-02-20T11:10:32.969 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=0c3148ca19c827ca2a7a9b92c2554d99507fd1f4 2026-02-20T11:10:32.972 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0c3148ca19c827ca2a7a9b92c2554d99507fd1f4', 'tag': None, 'wait_for_package': False} 2026-02-20T11:10:32.972 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=0c3148ca19c827ca2a7a9b92c2554d99507fd1f4 2026-02-20T11:10:33.167 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-19-02-2026-PR64439/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/centos/9/flavors/debug/ 2026-02-20T11:10:33.167 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5357.g0c3148ca 2026-02-20T11:10:33.177 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-19-02-2026-PR64439/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/centos/9/flavors/debug/ 2026-02-20T11:10:33.177 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5357.g0c3148ca 2026-02-20T11:10:33.323 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-19-02-2026-PR64439/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-19-02-2026-PR64439/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-19-02-2026-PR64439/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-20T11:10:33.323 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-20T11:10:33.323 DEBUG:teuthology.orchestra.run.trial043:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-20T11:10:33.335 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-19-02-2026-PR64439/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-19-02-2026-PR64439/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-19-02-2026-PR64439/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-20T11:10:33.335 DEBUG:teuthology.orchestra.run.trial109:> set -ex 2026-02-20T11:10:33.335 DEBUG:teuthology.orchestra.run.trial109:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-20T11:10:33.348 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo on remote rpm x86_64 2026-02-20T11:10:33.348 DEBUG:teuthology.orchestra.run.trial043:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-20T11:10:33.360 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo on remote rpm x86_64 2026-02-20T11:10:33.360 DEBUG:teuthology.orchestra.run.trial109:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-20T11:10:33.412 DEBUG:teuthology.orchestra.run.trial043:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-20T11:10:33.425 DEBUG:teuthology.orchestra.run.trial109:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-20T11:10:33.485 DEBUG:teuthology.orchestra.run.trial043:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-20T11:10:33.499 DEBUG:teuthology.orchestra.run.trial109:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-20T11:10:33.549 INFO:teuthology.orchestra.run.trial043.stdout:check_obsoletes = 1 2026-02-20T11:10:33.551 DEBUG:teuthology.orchestra.run.trial043:> sudo yum clean all 2026-02-20T11:10:33.562 INFO:teuthology.orchestra.run.trial109.stdout:check_obsoletes = 1 2026-02-20T11:10:33.564 DEBUG:teuthology.orchestra.run.trial109:> sudo yum clean all 2026-02-20T11:10:33.709 INFO:teuthology.orchestra.run.trial043.stdout:41 files removed 2026-02-20T11:10:33.721 INFO:teuthology.orchestra.run.trial109.stdout:41 files removed 2026-02-20T11:10:33.723 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-radosgw 2026-02-20T11:10:33.735 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install ceph-radosgw 2026-02-20T11:10:33.991 INFO:teuthology.orchestra.run.trial043.stdout:ceph packages for x86_64 741 kB/s | 94 kB 00:00 2026-02-20T11:10:34.016 INFO:teuthology.orchestra.run.trial109.stdout:ceph packages for x86_64 672 kB/s | 94 kB 00:00 2026-02-20T11:10:34.136 INFO:teuthology.orchestra.run.trial043.stdout:ceph noarch packages 208 kB/s | 24 kB 00:00 2026-02-20T11:10:34.177 INFO:teuthology.orchestra.run.trial109.stdout:ceph noarch packages 182 kB/s | 24 kB 00:00 2026-02-20T11:10:34.264 INFO:teuthology.orchestra.run.trial043.stdout:ceph source packages 19 kB/s | 2.1 kB 00:00 2026-02-20T11:10:34.302 INFO:teuthology.orchestra.run.trial109.stdout:ceph source packages 20 kB/s | 2.1 kB 00:00 2026-02-20T11:10:35.344 INFO:teuthology.orchestra.run.trial043.stdout:CentOS Stream 9 - BaseOS 8.4 MB/s | 8.9 MB 00:01 2026-02-20T11:10:38.523 INFO:teuthology.orchestra.run.trial043.stdout:CentOS Stream 9 - AppStream 10 MB/s | 27 MB 00:02 2026-02-20T11:10:38.541 INFO:teuthology.orchestra.run.trial109.stdout:CentOS Stream 9 - BaseOS 2.1 MB/s | 8.9 MB 00:04 2026-02-20T11:10:41.855 INFO:teuthology.orchestra.run.trial109.stdout:CentOS Stream 9 - AppStream 9.6 MB/s | 27 MB 00:02 2026-02-20T11:10:43.969 INFO:teuthology.orchestra.run.trial043.stdout:CentOS Stream 9 - CRB 2.5 MB/s | 7.8 MB 00:03 2026-02-20T11:10:44.980 INFO:teuthology.orchestra.run.trial043.stdout:CentOS Stream 9 - Extras packages 69 kB/s | 20 kB 00:00 2026-02-20T11:10:45.880 INFO:teuthology.orchestra.run.trial109.stdout:CentOS Stream 9 - CRB 4.7 MB/s | 7.8 MB 00:01 2026-02-20T11:10:46.547 INFO:teuthology.orchestra.run.trial043.stdout:Extra Packages for Enterprise Linux 13 MB/s | 20 MB 00:01 2026-02-20T11:10:46.810 INFO:teuthology.orchestra.run.trial109.stdout:CentOS Stream 9 - Extras packages 94 kB/s | 20 kB 00:00 2026-02-20T11:10:48.091 INFO:teuthology.orchestra.run.trial109.stdout:Extra Packages for Enterprise Linux 17 MB/s | 20 MB 00:01 2026-02-20T11:10:49.850 INFO:teuthology.orchestra.run.trial043.stdout:lab-extras 269 kB/s | 50 kB 00:00 2026-02-20T11:10:51.050 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:10:51.051 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-20T11:10:51.052 INFO:teuthology.orchestra.run.trial043.stdout: Package Arch Version Repository Size 2026-02-20T11:10:51.052 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-20T11:10:51.052 INFO:teuthology.orchestra.run.trial043.stdout:Installing: 2026-02-20T11:10:51.052 INFO:teuthology.orchestra.run.trial043.stdout: ceph-radosgw x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 11 M 2026-02-20T11:10:51.052 INFO:teuthology.orchestra.run.trial043.stdout:Upgrading: 2026-02-20T11:10:51.052 INFO:teuthology.orchestra.run.trial043.stdout: librados2 x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 3.3 M 2026-02-20T11:10:51.052 INFO:teuthology.orchestra.run.trial043.stdout: librbd1 x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 2.7 M 2026-02-20T11:10:51.052 INFO:teuthology.orchestra.run.trial043.stdout:Installing dependencies: 2026-02-20T11:10:51.053 INFO:teuthology.orchestra.run.trial043.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-20T11:10:51.053 INFO:teuthology.orchestra.run.trial043.stdout: ceph-base x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 5.1 M 2026-02-20T11:10:51.053 INFO:teuthology.orchestra.run.trial043.stdout: ceph-common x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 21 M 2026-02-20T11:10:51.053 INFO:teuthology.orchestra.run.trial043.stdout: ceph-selinux x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 25 k 2026-02-20T11:10:51.053 INFO:teuthology.orchestra.run.trial043.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-20T11:10:51.053 INFO:teuthology.orchestra.run.trial043.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-20T11:10:51.053 INFO:teuthology.orchestra.run.trial043.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-20T11:10:51.053 INFO:teuthology.orchestra.run.trial043.stdout: libcephfs2 x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 860 k 2026-02-20T11:10:51.053 INFO:teuthology.orchestra.run.trial043.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-20T11:10:51.053 INFO:teuthology.orchestra.run.trial043.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-20T11:10:51.054 INFO:teuthology.orchestra.run.trial043.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-20T11:10:51.054 INFO:teuthology.orchestra.run.trial043.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-20T11:10:51.054 INFO:teuthology.orchestra.run.trial043.stdout: libradosstriper1 x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 263 k 2026-02-20T11:10:51.054 INFO:teuthology.orchestra.run.trial043.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-20T11:10:51.054 INFO:teuthology.orchestra.run.trial043.stdout: librgw2 x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 5.6 M 2026-02-20T11:10:51.054 INFO:teuthology.orchestra.run.trial043.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-20T11:10:51.054 INFO:teuthology.orchestra.run.trial043.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-20T11:10:51.054 INFO:teuthology.orchestra.run.trial043.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-20T11:10:51.054 INFO:teuthology.orchestra.run.trial043.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-20T11:10:51.054 INFO:teuthology.orchestra.run.trial043.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-20T11:10:51.054 INFO:teuthology.orchestra.run.trial043.stdout: python3-ceph-argparse x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 45 k 2026-02-20T11:10:51.054 INFO:teuthology.orchestra.run.trial043.stdout: python3-ceph-common x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 185 k 2026-02-20T11:10:51.054 INFO:teuthology.orchestra.run.trial043.stdout: python3-cephfs x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 204 k 2026-02-20T11:10:51.055 INFO:teuthology.orchestra.run.trial043.stdout: python3-rados x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 324 k 2026-02-20T11:10:51.055 INFO:teuthology.orchestra.run.trial043.stdout: python3-rbd x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 305 k 2026-02-20T11:10:51.055 INFO:teuthology.orchestra.run.trial043.stdout: python3-rgw x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 100 k 2026-02-20T11:10:51.055 INFO:teuthology.orchestra.run.trial043.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-20T11:10:51.055 INFO:teuthology.orchestra.run.trial043.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-20T11:10:51.055 INFO:teuthology.orchestra.run.trial043.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-20T11:10:51.055 INFO:teuthology.orchestra.run.trial043.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-20T11:10:51.055 INFO:teuthology.orchestra.run.trial043.stdout:Installing weak dependencies: 2026-02-20T11:10:51.055 INFO:teuthology.orchestra.run.trial043.stdout: libcephfs-daemon x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 37 k 2026-02-20T11:10:51.055 INFO:teuthology.orchestra.run.trial043.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 26 k 2026-02-20T11:10:51.055 INFO:teuthology.orchestra.run.trial043.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-20T11:10:51.055 INFO:teuthology.orchestra.run.trial043.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-20T11:10:51.055 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:10:51.056 INFO:teuthology.orchestra.run.trial043.stdout:Transaction Summary 2026-02-20T11:10:51.056 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-20T11:10:51.056 INFO:teuthology.orchestra.run.trial043.stdout:Install 35 Packages 2026-02-20T11:10:51.056 INFO:teuthology.orchestra.run.trial043.stdout:Upgrade 2 Packages 2026-02-20T11:10:51.056 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:10:51.057 INFO:teuthology.orchestra.run.trial043.stdout:Total download size: 61 M 2026-02-20T11:10:51.057 INFO:teuthology.orchestra.run.trial043.stdout:Downloading Packages: 2026-02-20T11:10:51.228 INFO:teuthology.orchestra.run.trial109.stdout:lab-extras 2.5 MB/s | 50 kB 00:00 2026-02-20T11:10:52.422 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:10:52.423 INFO:teuthology.orchestra.run.trial109.stdout:================================================================================ 2026-02-20T11:10:52.424 INFO:teuthology.orchestra.run.trial109.stdout: Package Arch Version Repository Size 2026-02-20T11:10:52.424 INFO:teuthology.orchestra.run.trial109.stdout:================================================================================ 2026-02-20T11:10:52.424 INFO:teuthology.orchestra.run.trial109.stdout:Installing: 2026-02-20T11:10:52.424 INFO:teuthology.orchestra.run.trial109.stdout: ceph-radosgw x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 11 M 2026-02-20T11:10:52.424 INFO:teuthology.orchestra.run.trial109.stdout:Upgrading: 2026-02-20T11:10:52.424 INFO:teuthology.orchestra.run.trial109.stdout: librados2 x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 3.3 M 2026-02-20T11:10:52.424 INFO:teuthology.orchestra.run.trial109.stdout: librbd1 x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 2.7 M 2026-02-20T11:10:52.424 INFO:teuthology.orchestra.run.trial109.stdout:Installing dependencies: 2026-02-20T11:10:52.425 INFO:teuthology.orchestra.run.trial109.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-20T11:10:52.425 INFO:teuthology.orchestra.run.trial109.stdout: ceph-base x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 5.1 M 2026-02-20T11:10:52.425 INFO:teuthology.orchestra.run.trial109.stdout: ceph-common x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 21 M 2026-02-20T11:10:52.425 INFO:teuthology.orchestra.run.trial109.stdout: ceph-selinux x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 25 k 2026-02-20T11:10:52.425 INFO:teuthology.orchestra.run.trial109.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-20T11:10:52.425 INFO:teuthology.orchestra.run.trial109.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-20T11:10:52.425 INFO:teuthology.orchestra.run.trial109.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-20T11:10:52.425 INFO:teuthology.orchestra.run.trial109.stdout: libcephfs2 x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 860 k 2026-02-20T11:10:52.425 INFO:teuthology.orchestra.run.trial109.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-20T11:10:52.425 INFO:teuthology.orchestra.run.trial109.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-20T11:10:52.425 INFO:teuthology.orchestra.run.trial109.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-20T11:10:52.426 INFO:teuthology.orchestra.run.trial109.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-20T11:10:52.426 INFO:teuthology.orchestra.run.trial109.stdout: libradosstriper1 x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 263 k 2026-02-20T11:10:52.426 INFO:teuthology.orchestra.run.trial109.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-20T11:10:52.426 INFO:teuthology.orchestra.run.trial109.stdout: librgw2 x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 5.6 M 2026-02-20T11:10:52.426 INFO:teuthology.orchestra.run.trial109.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-20T11:10:52.426 INFO:teuthology.orchestra.run.trial109.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-20T11:10:52.426 INFO:teuthology.orchestra.run.trial109.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-20T11:10:52.426 INFO:teuthology.orchestra.run.trial109.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-20T11:10:52.426 INFO:teuthology.orchestra.run.trial109.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-20T11:10:52.426 INFO:teuthology.orchestra.run.trial109.stdout: python3-ceph-argparse x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 45 k 2026-02-20T11:10:52.427 INFO:teuthology.orchestra.run.trial109.stdout: python3-ceph-common x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 185 k 2026-02-20T11:10:52.427 INFO:teuthology.orchestra.run.trial109.stdout: python3-cephfs x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 204 k 2026-02-20T11:10:52.427 INFO:teuthology.orchestra.run.trial109.stdout: python3-rados x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 324 k 2026-02-20T11:10:52.427 INFO:teuthology.orchestra.run.trial109.stdout: python3-rbd x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 305 k 2026-02-20T11:10:52.427 INFO:teuthology.orchestra.run.trial109.stdout: python3-rgw x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 100 k 2026-02-20T11:10:52.427 INFO:teuthology.orchestra.run.trial109.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-20T11:10:52.427 INFO:teuthology.orchestra.run.trial109.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-20T11:10:52.427 INFO:teuthology.orchestra.run.trial109.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-20T11:10:52.427 INFO:teuthology.orchestra.run.trial109.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-20T11:10:52.427 INFO:teuthology.orchestra.run.trial109.stdout:Installing weak dependencies: 2026-02-20T11:10:52.428 INFO:teuthology.orchestra.run.trial109.stdout: libcephfs-daemon x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 37 k 2026-02-20T11:10:52.428 INFO:teuthology.orchestra.run.trial109.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 26 k 2026-02-20T11:10:52.428 INFO:teuthology.orchestra.run.trial109.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-20T11:10:52.428 INFO:teuthology.orchestra.run.trial109.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-20T11:10:52.428 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:10:52.428 INFO:teuthology.orchestra.run.trial109.stdout:Transaction Summary 2026-02-20T11:10:52.428 INFO:teuthology.orchestra.run.trial109.stdout:================================================================================ 2026-02-20T11:10:52.428 INFO:teuthology.orchestra.run.trial109.stdout:Install 35 Packages 2026-02-20T11:10:52.428 INFO:teuthology.orchestra.run.trial109.stdout:Upgrade 2 Packages 2026-02-20T11:10:52.428 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:10:52.429 INFO:teuthology.orchestra.run.trial109.stdout:Total download size: 61 M 2026-02-20T11:10:52.429 INFO:teuthology.orchestra.run.trial109.stdout:Downloading Packages: 2026-02-20T11:10:54.145 INFO:teuthology.orchestra.run.trial043.stdout:(1/37): ceph-base-20.3.0-5357.g0c3148ca.el9.x86 2.0 MB/s | 5.1 MB 00:02 2026-02-20T11:10:54.188 INFO:teuthology.orchestra.run.trial043.stdout:(2/37): ceph-selinux-20.3.0-5357.g0c3148ca.el9. 586 kB/s | 25 kB 00:00 2026-02-20T11:10:54.208 INFO:teuthology.orchestra.run.trial043.stdout:(3/37): ceph-radosgw-20.3.0-5357.g0c3148ca.el9. 4.3 MB/s | 11 MB 00:02 2026-02-20T11:10:54.230 INFO:teuthology.orchestra.run.trial043.stdout:(4/37): libcephfs-daemon-20.3.0-5357.g0c3148ca. 863 kB/s | 37 kB 00:00 2026-02-20T11:10:54.252 INFO:teuthology.orchestra.run.trial043.stdout:(5/37): libcephfs-proxy2-20.3.0-5357.g0c3148ca. 585 kB/s | 26 kB 00:00 2026-02-20T11:10:54.305 INFO:teuthology.orchestra.run.trial043.stdout:(6/37): libradosstriper1-20.3.0-5357.g0c3148ca. 4.9 MB/s | 263 kB 00:00 2026-02-20T11:10:54.386 INFO:teuthology.orchestra.run.trial043.stdout:(7/37): libcephfs2-20.3.0-5357.g0c3148ca.el9.x8 5.4 MB/s | 860 kB 00:00 2026-02-20T11:10:54.438 INFO:teuthology.orchestra.run.trial043.stdout:(8/37): python3-ceph-argparse-20.3.0-5357.g0c31 863 kB/s | 45 kB 00:00 2026-02-20T11:10:54.494 INFO:teuthology.orchestra.run.trial043.stdout:(9/37): python3-ceph-common-20.3.0-5357.g0c3148 3.2 MB/s | 185 kB 00:00 2026-02-20T11:10:54.551 INFO:teuthology.orchestra.run.trial043.stdout:(10/37): python3-cephfs-20.3.0-5357.g0c3148ca.e 3.5 MB/s | 204 kB 00:00 2026-02-20T11:10:54.610 INFO:teuthology.orchestra.run.trial043.stdout:(11/37): python3-rados-20.3.0-5357.g0c3148ca.el 5.4 MB/s | 324 kB 00:00 2026-02-20T11:10:54.669 INFO:teuthology.orchestra.run.trial043.stdout:(12/37): python3-rbd-20.3.0-5357.g0c3148ca.el9. 5.0 MB/s | 305 kB 00:00 2026-02-20T11:10:54.724 INFO:teuthology.orchestra.run.trial043.stdout:(13/37): python3-rgw-20.3.0-5357.g0c3148ca.el9. 1.8 MB/s | 100 kB 00:00 2026-02-20T11:10:54.953 INFO:teuthology.orchestra.run.trial109.stdout:(1/37): ceph-base-20.3.0-5357.g0c3148ca.el9.x86 2.2 MB/s | 5.1 MB 00:02 2026-02-20T11:10:54.961 INFO:teuthology.orchestra.run.trial043.stdout:(14/37): librgw2-20.3.0-5357.g0c3148ca.el9.x86_ 8.6 MB/s | 5.6 MB 00:00 2026-02-20T11:10:54.999 INFO:teuthology.orchestra.run.trial109.stdout:(2/37): ceph-selinux-20.3.0-5357.g0c3148ca.el9. 544 kB/s | 25 kB 00:00 2026-02-20T11:10:55.045 INFO:teuthology.orchestra.run.trial109.stdout:(3/37): libcephfs-daemon-20.3.0-5357.g0c3148ca. 815 kB/s | 37 kB 00:00 2026-02-20T11:10:55.092 INFO:teuthology.orchestra.run.trial109.stdout:(4/37): libcephfs-proxy2-20.3.0-5357.g0c3148ca. 545 kB/s | 26 kB 00:00 2026-02-20T11:10:55.365 INFO:teuthology.orchestra.run.trial043.stdout:(15/37): mailcap-2.1.49-5.el9.noarch.rpm 52 kB/s | 33 kB 00:00 2026-02-20T11:10:55.443 INFO:teuthology.orchestra.run.trial109.stdout:(5/37): libcephfs2-20.3.0-5357.g0c3148ca.el9.x8 2.4 MB/s | 860 kB 00:00 2026-02-20T11:10:55.570 INFO:teuthology.orchestra.run.trial109.stdout:(6/37): libradosstriper1-20.3.0-5357.g0c3148ca. 2.0 MB/s | 263 kB 00:00 2026-02-20T11:10:55.906 INFO:teuthology.orchestra.run.trial043.stdout:(16/37): nvme-cli-2.16-1.el9.x86_64.rpm 1.2 MB/s | 1.2 MB 00:00 2026-02-20T11:10:55.953 INFO:teuthology.orchestra.run.trial043.stdout:(17/37): boost-program-options-1.75.0-13.el9.x8 2.2 MB/s | 104 kB 00:00 2026-02-20T11:10:55.974 INFO:teuthology.orchestra.run.trial043.stdout:(18/37): libnbd-1.20.3-4.el9.x86_64.rpm 7.8 MB/s | 164 kB 00:00 2026-02-20T11:10:55.987 INFO:teuthology.orchestra.run.trial043.stdout:(19/37): libpmemobj-1.12.1-1.el9.x86_64.rpm 12 MB/s | 160 kB 00:00 2026-02-20T11:10:55.997 INFO:teuthology.orchestra.run.trial043.stdout:(20/37): librabbitmq-0.11.0-7.el9.x86_64.rpm 4.4 MB/s | 45 kB 00:00 2026-02-20T11:10:56.022 INFO:teuthology.orchestra.run.trial043.stdout:(21/37): librdkafka-1.6.1-102.el9.x86_64.rpm 25 MB/s | 662 kB 00:00 2026-02-20T11:10:56.023 INFO:teuthology.orchestra.run.trial043.stdout:(22/37): pciutils-3.7.0-7.el9.x86_64.rpm 141 kB/s | 93 kB 00:00 2026-02-20T11:10:56.035 INFO:teuthology.orchestra.run.trial043.stdout:(23/37): lttng-ust-2.12.0-6.el9.x86_64.rpm 24 MB/s | 292 kB 00:00 2026-02-20T11:10:56.047 INFO:teuthology.orchestra.run.trial043.stdout:(24/37): qatlib-service-25.08.0-2.el9.x86_64.rp 3.5 MB/s | 37 kB 00:00 2026-02-20T11:10:56.056 INFO:teuthology.orchestra.run.trial043.stdout:(25/37): qatzip-libs-1.3.1-1.el9.x86_64.rpm 6.3 MB/s | 66 kB 00:00 2026-02-20T11:10:56.076 INFO:teuthology.orchestra.run.trial043.stdout:(26/37): qatlib-25.08.0-2.el9.x86_64.rpm 4.5 MB/s | 240 kB 00:00 2026-02-20T11:10:56.282 INFO:teuthology.orchestra.run.trial043.stdout:(27/37): gperftools-libs-2.9.1-3.el9.x86_64.rpm 1.3 MB/s | 308 kB 00:00 2026-02-20T11:10:56.307 INFO:teuthology.orchestra.run.trial043.stdout:(28/37): libarrow-doc-9.0.0-15.el9.noarch.rpm 1.0 MB/s | 25 kB 00:00 2026-02-20T11:10:56.331 INFO:teuthology.orchestra.run.trial043.stdout:(29/37): liboath-2.6.12-1.el9.x86_64.rpm 2.0 MB/s | 49 kB 00:00 2026-02-20T11:10:56.356 INFO:teuthology.orchestra.run.trial043.stdout:(30/37): libunwind-1.6.2-1.el9.x86_64.rpm 2.7 MB/s | 67 kB 00:00 2026-02-20T11:10:56.407 INFO:teuthology.orchestra.run.trial043.stdout:(31/37): libarrow-9.0.0-15.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2026-02-20T11:10:56.408 INFO:teuthology.orchestra.run.trial043.stdout:(32/37): parquet-libs-9.0.0-15.el9.x86_64.rpm 16 MB/s | 838 kB 00:00 2026-02-20T11:10:56.436 INFO:teuthology.orchestra.run.trial043.stdout:(33/37): re2-20211101-20.el9.x86_64.rpm 6.4 MB/s | 191 kB 00:00 2026-02-20T11:10:56.486 INFO:teuthology.orchestra.run.trial043.stdout:(34/37): thrift-0.15.0-4.el9.x86_64.rpm 20 MB/s | 1.6 MB 00:00 2026-02-20T11:10:57.096 INFO:teuthology.orchestra.run.trial043.stdout:(35/37): librbd1-20.3.0-5357.g0c3148ca.el9.x86_ 4.4 MB/s | 2.7 MB 00:00 2026-02-20T11:10:57.290 INFO:teuthology.orchestra.run.trial043.stdout:(36/37): librados2-20.3.0-5357.g0c3148ca.el9.x8 3.8 MB/s | 3.3 MB 00:00 2026-02-20T11:10:57.574 INFO:teuthology.orchestra.run.trial109.stdout:(7/37): ceph-radosgw-20.3.0-5357.g0c3148ca.el9. 2.3 MB/s | 11 MB 00:04 2026-02-20T11:10:57.620 INFO:teuthology.orchestra.run.trial109.stdout:(8/37): python3-ceph-argparse-20.3.0-5357.g0c31 989 kB/s | 45 kB 00:00 2026-02-20T11:10:57.685 INFO:teuthology.orchestra.run.trial109.stdout:(9/37): librgw2-20.3.0-5357.g0c3148ca.el9.x86_6 2.7 MB/s | 5.6 MB 00:02 2026-02-20T11:10:57.710 INFO:teuthology.orchestra.run.trial109.stdout:(10/37): python3-ceph-common-20.3.0-5357.g0c314 2.0 MB/s | 185 kB 00:00 2026-02-20T11:10:57.775 INFO:teuthology.orchestra.run.trial109.stdout:(11/37): python3-cephfs-20.3.0-5357.g0c3148ca.e 2.2 MB/s | 204 kB 00:00 2026-02-20T11:10:57.849 INFO:teuthology.orchestra.run.trial109.stdout:(12/37): python3-rados-20.3.0-5357.g0c3148ca.el 2.3 MB/s | 324 kB 00:00 2026-02-20T11:10:57.869 INFO:teuthology.orchestra.run.trial109.stdout:(13/37): python3-rbd-20.3.0-5357.g0c3148ca.el9. 3.2 MB/s | 305 kB 00:00 2026-02-20T11:10:57.895 INFO:teuthology.orchestra.run.trial109.stdout:(14/37): python3-rgw-20.3.0-5357.g0c3148ca.el9. 2.1 MB/s | 100 kB 00:00 2026-02-20T11:10:58.483 INFO:teuthology.orchestra.run.trial109.stdout:(15/37): mailcap-2.1.49-5.el9.noarch.rpm 54 kB/s | 33 kB 00:00 2026-02-20T11:10:58.744 INFO:teuthology.orchestra.run.trial109.stdout:(16/37): pciutils-3.7.0-7.el9.x86_64.rpm 357 kB/s | 93 kB 00:00 2026-02-20T11:10:58.943 INFO:teuthology.orchestra.run.trial109.stdout:(17/37): nvme-cli-2.16-1.el9.x86_64.rpm 1.1 MB/s | 1.2 MB 00:01 2026-02-20T11:10:59.263 INFO:teuthology.orchestra.run.trial109.stdout:(18/37): libnbd-1.20.3-4.el9.x86_64.rpm 512 kB/s | 164 kB 00:00 2026-02-20T11:10:59.327 INFO:teuthology.orchestra.run.trial109.stdout:(19/37): boost-program-options-1.75.0-13.el9.x8 178 kB/s | 104 kB 00:00 2026-02-20T11:10:59.418 INFO:teuthology.orchestra.run.trial109.stdout:(20/37): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 160 kB 00:00 2026-02-20T11:10:59.454 INFO:teuthology.orchestra.run.trial109.stdout:(21/37): ceph-common-20.3.0-5357.g0c3148ca.el9. 3.1 MB/s | 21 MB 00:06 2026-02-20T11:10:59.455 INFO:teuthology.orchestra.run.trial109.stdout:(22/37): librabbitmq-0.11.0-7.el9.x86_64.rpm 354 kB/s | 45 kB 00:00 2026-02-20T11:10:59.701 INFO:teuthology.orchestra.run.trial109.stdout:(23/37): librdkafka-1.6.1-102.el9.x86_64.rpm 2.3 MB/s | 662 kB 00:00 2026-02-20T11:10:59.703 INFO:teuthology.orchestra.run.trial109.stdout:(24/37): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.1 MB/s | 292 kB 00:00 2026-02-20T11:10:59.710 INFO:teuthology.orchestra.run.trial109.stdout:(25/37): qatlib-25.08.0-2.el9.x86_64.rpm 936 kB/s | 240 kB 00:00 2026-02-20T11:10:59.791 INFO:teuthology.orchestra.run.trial109.stdout:(26/37): qatlib-service-25.08.0-2.el9.x86_64.rp 414 kB/s | 37 kB 00:00 2026-02-20T11:10:59.797 INFO:teuthology.orchestra.run.trial109.stdout:(27/37): qatzip-libs-1.3.1-1.el9.x86_64.rpm 704 kB/s | 66 kB 00:00 2026-02-20T11:10:59.872 INFO:teuthology.orchestra.run.trial109.stdout:(28/37): libarrow-doc-9.0.0-15.el9.noarch.rpm 332 kB/s | 25 kB 00:00 2026-02-20T11:10:59.923 INFO:teuthology.orchestra.run.trial109.stdout:(29/37): liboath-2.6.12-1.el9.x86_64.rpm 970 kB/s | 49 kB 00:00 2026-02-20T11:10:59.931 INFO:teuthology.orchestra.run.trial109.stdout:(30/37): gperftools-libs-2.9.1-3.el9.x86_64.rpm 1.4 MB/s | 308 kB 00:00 2026-02-20T11:10:59.951 INFO:teuthology.orchestra.run.trial109.stdout:(31/37): libunwind-1.6.2-1.el9.x86_64.rpm 2.4 MB/s | 67 kB 00:00 2026-02-20T11:10:59.984 INFO:teuthology.orchestra.run.trial109.stdout:(32/37): parquet-libs-9.0.0-15.el9.x86_64.rpm 15 MB/s | 838 kB 00:00 2026-02-20T11:11:00.003 INFO:teuthology.orchestra.run.trial109.stdout:(33/37): re2-20211101-20.el9.x86_64.rpm 3.6 MB/s | 191 kB 00:00 2026-02-20T11:11:00.021 INFO:teuthology.orchestra.run.trial109.stdout:(34/37): libarrow-9.0.0-15.el9.x86_64.rpm 19 MB/s | 4.4 MB 00:00 2026-02-20T11:11:00.066 INFO:teuthology.orchestra.run.trial109.stdout:(35/37): thrift-0.15.0-4.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2026-02-20T11:11:00.833 INFO:teuthology.orchestra.run.trial109.stdout:(36/37): librbd1-20.3.0-5357.g0c3148ca.el9.x86_ 3.3 MB/s | 2.7 MB 00:00 2026-02-20T11:11:01.441 INFO:teuthology.orchestra.run.trial109.stdout:(37/37): librados2-20.3.0-5357.g0c3148ca.el9.x8 2.3 MB/s | 3.3 MB 00:01 2026-02-20T11:11:01.444 INFO:teuthology.orchestra.run.trial109.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:11:01.444 INFO:teuthology.orchestra.run.trial109.stdout:Total 6.8 MB/s | 61 MB 00:09 2026-02-20T11:11:01.542 INFO:teuthology.orchestra.run.trial109.stdout:Running transaction check 2026-02-20T11:11:01.562 INFO:teuthology.orchestra.run.trial109.stdout:Transaction check succeeded. 2026-02-20T11:11:01.562 INFO:teuthology.orchestra.run.trial109.stdout:Running transaction test 2026-02-20T11:11:01.813 INFO:teuthology.orchestra.run.trial109.stdout:Transaction test succeeded. 2026-02-20T11:11:01.813 INFO:teuthology.orchestra.run.trial109.stdout:Running transaction 2026-02-20T11:11:01.939 INFO:teuthology.orchestra.run.trial043.stdout:(37/37): ceph-common-20.3.0-5357.g0c3148ca.el9. 2.0 MB/s | 21 MB 00:10 2026-02-20T11:11:01.941 INFO:teuthology.orchestra.run.trial043.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:11:01.941 INFO:teuthology.orchestra.run.trial043.stdout:Total 5.6 MB/s | 61 MB 00:10 2026-02-20T11:11:02.036 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction check 2026-02-20T11:11:02.055 INFO:teuthology.orchestra.run.trial043.stdout:Transaction check succeeded. 2026-02-20T11:11:02.055 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction test 2026-02-20T11:11:02.101 INFO:teuthology.orchestra.run.trial109.stdout: Preparing : 1/1 2026-02-20T11:11:02.243 INFO:teuthology.orchestra.run.trial109.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/39 2026-02-20T11:11:02.248 INFO:teuthology.orchestra.run.trial109.stdout: Upgrading : librados2-2:20.3.0-5357.g0c3148ca.el9.x86_64 2/39 2026-02-20T11:11:02.297 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: librados2-2:20.3.0-5357.g0c3148ca.el9.x86_64 2/39 2026-02-20T11:11:02.298 INFO:teuthology.orchestra.run.trial109.stdout: Installing : libcephfs2-2:20.3.0-5357.g0c3148ca.el9.x86_64 3/39 2026-02-20T11:11:02.306 INFO:teuthology.orchestra.run.trial043.stdout:Transaction test succeeded. 2026-02-20T11:11:02.306 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction 2026-02-20T11:11:02.311 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: libcephfs2-2:20.3.0-5357.g0c3148ca.el9.x86_64 3/39 2026-02-20T11:11:02.313 INFO:teuthology.orchestra.run.trial109.stdout: Installing : libcephfs-daemon-2:20.3.0-5357.g0c3148ca.el9.x86_6 4/39 2026-02-20T11:11:02.314 INFO:teuthology.orchestra.run.trial109.stdout: Installing : libcephfs-proxy2-2:20.3.0-5357.g0c3148ca.el9.x86_6 5/39 2026-02-20T11:11:02.341 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5357.g0c3148ca.el9.x86_6 5/39 2026-02-20T11:11:02.344 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-rados-2:20.3.0-5357.g0c3148ca.el9.x86_64 6/39 2026-02-20T11:11:02.352 INFO:teuthology.orchestra.run.trial109.stdout: Installing : liboath-2.6.12-1.el9.x86_64 7/39 2026-02-20T11:11:02.355 INFO:teuthology.orchestra.run.trial109.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/39 2026-02-20T11:11:02.358 INFO:teuthology.orchestra.run.trial109.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/39 2026-02-20T11:11:02.363 INFO:teuthology.orchestra.run.trial109.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/39 2026-02-20T11:11:02.374 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-ceph-argparse-2:20.3.0-5357.g0c3148ca.el9. 11/39 2026-02-20T11:11:02.387 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-cephfs-2:20.3.0-5357.g0c3148ca.el9.x86_64 12/39 2026-02-20T11:11:02.387 INFO:teuthology.orchestra.run.trial109.stdout: Installing : libradosstriper1-2:20.3.0-5357.g0c3148ca.el9.x86_6 13/39 2026-02-20T11:11:02.412 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5357.g0c3148ca.el9.x86_6 13/39 2026-02-20T11:11:02.416 INFO:teuthology.orchestra.run.trial109.stdout: Installing : thrift-0.15.0-4.el9.x86_64 14/39 2026-02-20T11:11:02.450 INFO:teuthology.orchestra.run.trial109.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/39 2026-02-20T11:11:02.454 INFO:teuthology.orchestra.run.trial109.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/39 2026-02-20T11:11:02.462 INFO:teuthology.orchestra.run.trial109.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/39 2026-02-20T11:11:02.464 INFO:teuthology.orchestra.run.trial109.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/39 2026-02-20T11:11:02.469 INFO:teuthology.orchestra.run.trial109.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 19/39 2026-02-20T11:11:02.583 INFO:teuthology.orchestra.run.trial109.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-20T11:11:02.584 INFO:teuthology.orchestra.run.trial109.stdout: Upgrading : librbd1-2:20.3.0-5357.g0c3148ca.el9.x86_64 21/39 2026-02-20T11:11:02.596 INFO:teuthology.orchestra.run.trial043.stdout: Preparing : 1/1 2026-02-20T11:11:02.610 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: librbd1-2:20.3.0-5357.g0c3148ca.el9.x86_64 21/39 2026-02-20T11:11:02.614 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-rbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 22/39 2026-02-20T11:11:02.621 INFO:teuthology.orchestra.run.trial109.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 23/39 2026-02-20T11:11:02.738 INFO:teuthology.orchestra.run.trial043.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/39 2026-02-20T11:11:02.742 INFO:teuthology.orchestra.run.trial043.stdout: Upgrading : librados2-2:20.3.0-5357.g0c3148ca.el9.x86_64 2/39 2026-02-20T11:11:02.788 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: librados2-2:20.3.0-5357.g0c3148ca.el9.x86_64 2/39 2026-02-20T11:11:02.789 INFO:teuthology.orchestra.run.trial043.stdout: Installing : libcephfs2-2:20.3.0-5357.g0c3148ca.el9.x86_64 3/39 2026-02-20T11:11:02.801 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: libcephfs2-2:20.3.0-5357.g0c3148ca.el9.x86_64 3/39 2026-02-20T11:11:02.804 INFO:teuthology.orchestra.run.trial043.stdout: Installing : libcephfs-daemon-2:20.3.0-5357.g0c3148ca.el9.x86_6 4/39 2026-02-20T11:11:02.805 INFO:teuthology.orchestra.run.trial043.stdout: Installing : libcephfs-proxy2-2:20.3.0-5357.g0c3148ca.el9.x86_6 5/39 2026-02-20T11:11:02.831 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5357.g0c3148ca.el9.x86_6 5/39 2026-02-20T11:11:02.833 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-rados-2:20.3.0-5357.g0c3148ca.el9.x86_64 6/39 2026-02-20T11:11:02.840 INFO:teuthology.orchestra.run.trial043.stdout: Installing : liboath-2.6.12-1.el9.x86_64 7/39 2026-02-20T11:11:02.843 INFO:teuthology.orchestra.run.trial043.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/39 2026-02-20T11:11:02.846 INFO:teuthology.orchestra.run.trial043.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/39 2026-02-20T11:11:02.850 INFO:teuthology.orchestra.run.trial043.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/39 2026-02-20T11:11:02.863 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-ceph-argparse-2:20.3.0-5357.g0c3148ca.el9. 11/39 2026-02-20T11:11:02.868 INFO:teuthology.orchestra.run.trial109.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 24/39 2026-02-20T11:11:02.869 INFO:teuthology.orchestra.run.trial109.stdout: Installing : librgw2-2:20.3.0-5357.g0c3148ca.el9.x86_64 25/39 2026-02-20T11:11:02.876 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-cephfs-2:20.3.0-5357.g0c3148ca.el9.x86_64 12/39 2026-02-20T11:11:02.876 INFO:teuthology.orchestra.run.trial043.stdout: Installing : libradosstriper1-2:20.3.0-5357.g0c3148ca.el9.x86_6 13/39 2026-02-20T11:11:02.887 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: librgw2-2:20.3.0-5357.g0c3148ca.el9.x86_64 25/39 2026-02-20T11:11:02.891 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-rgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 26/39 2026-02-20T11:11:02.896 INFO:teuthology.orchestra.run.trial109.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 27/39 2026-02-20T11:11:02.897 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5357.g0c3148ca.el9.x86_6 13/39 2026-02-20T11:11:02.898 INFO:teuthology.orchestra.run.trial109.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 28/39 2026-02-20T11:11:02.899 INFO:teuthology.orchestra.run.trial109.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-20T11:11:02.901 INFO:teuthology.orchestra.run.trial043.stdout: Installing : thrift-0.15.0-4.el9.x86_64 14/39 2026-02-20T11:11:02.918 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-20T11:11:02.932 INFO:teuthology.orchestra.run.trial043.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/39 2026-02-20T11:11:02.935 INFO:teuthology.orchestra.run.trial043.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/39 2026-02-20T11:11:02.942 INFO:teuthology.orchestra.run.trial043.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/39 2026-02-20T11:11:02.945 INFO:teuthology.orchestra.run.trial043.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/39 2026-02-20T11:11:02.950 INFO:teuthology.orchestra.run.trial043.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 19/39 2026-02-20T11:11:03.031 INFO:teuthology.orchestra.run.trial109.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 30/39 2026-02-20T11:11:03.034 INFO:teuthology.orchestra.run.trial109.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-20T11:11:03.063 INFO:teuthology.orchestra.run.trial043.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-20T11:11:03.064 INFO:teuthology.orchestra.run.trial043.stdout: Upgrading : librbd1-2:20.3.0-5357.g0c3148ca.el9.x86_64 21/39 2026-02-20T11:11:03.091 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: librbd1-2:20.3.0-5357.g0c3148ca.el9.x86_64 21/39 2026-02-20T11:11:03.095 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-rbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 22/39 2026-02-20T11:11:03.101 INFO:teuthology.orchestra.run.trial043.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 23/39 2026-02-20T11:11:03.334 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-20T11:11:03.334 INFO:teuthology.orchestra.run.trial109.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-20T11:11:03.334 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:11:03.350 INFO:teuthology.orchestra.run.trial043.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 24/39 2026-02-20T11:11:03.351 INFO:teuthology.orchestra.run.trial043.stdout: Installing : librgw2-2:20.3.0-5357.g0c3148ca.el9.x86_64 25/39 2026-02-20T11:11:03.366 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: librgw2-2:20.3.0-5357.g0c3148ca.el9.x86_64 25/39 2026-02-20T11:11:03.371 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-rgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 26/39 2026-02-20T11:11:03.375 INFO:teuthology.orchestra.run.trial043.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 27/39 2026-02-20T11:11:03.378 INFO:teuthology.orchestra.run.trial043.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 28/39 2026-02-20T11:11:03.378 INFO:teuthology.orchestra.run.trial109.stdout: Installing : mailcap-2.1.49-5.el9.noarch 32/39 2026-02-20T11:11:03.379 INFO:teuthology.orchestra.run.trial043.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-20T11:11:03.381 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-ceph-common-2:20.3.0-5357.g0c3148ca.el9.x8 33/39 2026-02-20T11:11:03.397 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-20T11:11:03.515 INFO:teuthology.orchestra.run.trial043.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 30/39 2026-02-20T11:11:03.517 INFO:teuthology.orchestra.run.trial043.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-20T11:11:03.813 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-20T11:11:03.814 INFO:teuthology.orchestra.run.trial043.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-20T11:11:03.814 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:11:03.858 INFO:teuthology.orchestra.run.trial043.stdout: Installing : mailcap-2.1.49-5.el9.noarch 32/39 2026-02-20T11:11:03.860 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-ceph-common-2:20.3.0-5357.g0c3148ca.el9.x8 33/39 2026-02-20T11:11:04.322 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: ceph-common-2:20.3.0-5357.g0c3148ca.el9.x86_64 34/39 2026-02-20T11:11:04.325 INFO:teuthology.orchestra.run.trial109.stdout: Installing : ceph-common-2:20.3.0-5357.g0c3148ca.el9.x86_64 34/39 2026-02-20T11:11:04.590 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: ceph-common-2:20.3.0-5357.g0c3148ca.el9.x86_64 34/39 2026-02-20T11:11:04.593 INFO:teuthology.orchestra.run.trial109.stdout: Installing : ceph-base-2:20.3.0-5357.g0c3148ca.el9.x86_64 35/39 2026-02-20T11:11:04.634 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: ceph-base-2:20.3.0-5357.g0c3148ca.el9.x86_64 35/39 2026-02-20T11:11:04.635 INFO:teuthology.orchestra.run.trial109.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-20T11:11:04.635 INFO:teuthology.orchestra.run.trial109.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-20T11:11:04.635 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:11:04.637 INFO:teuthology.orchestra.run.trial109.stdout: Installing : ceph-selinux-2:20.3.0-5357.g0c3148ca.el9.x86_64 36/39 2026-02-20T11:11:04.803 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-common-2:20.3.0-5357.g0c3148ca.el9.x86_64 34/39 2026-02-20T11:11:04.806 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-common-2:20.3.0-5357.g0c3148ca.el9.x86_64 34/39 2026-02-20T11:11:05.066 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-common-2:20.3.0-5357.g0c3148ca.el9.x86_64 34/39 2026-02-20T11:11:05.070 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-base-2:20.3.0-5357.g0c3148ca.el9.x86_64 35/39 2026-02-20T11:11:05.113 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-base-2:20.3.0-5357.g0c3148ca.el9.x86_64 35/39 2026-02-20T11:11:05.113 INFO:teuthology.orchestra.run.trial043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-20T11:11:05.113 INFO:teuthology.orchestra.run.trial043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-20T11:11:05.113 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:11:05.116 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-selinux-2:20.3.0-5357.g0c3148ca.el9.x86_64 36/39 2026-02-20T11:11:09.187 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5357.g0c3148ca.el9.x86_64 36/39 2026-02-20T11:11:09.187 INFO:teuthology.orchestra.run.trial109.stdout:skipping the directory /sys 2026-02-20T11:11:09.187 INFO:teuthology.orchestra.run.trial109.stdout:skipping the directory /proc 2026-02-20T11:11:09.187 INFO:teuthology.orchestra.run.trial109.stdout:skipping the directory /mnt 2026-02-20T11:11:09.188 INFO:teuthology.orchestra.run.trial109.stdout:skipping the directory /var/tmp 2026-02-20T11:11:09.188 INFO:teuthology.orchestra.run.trial109.stdout:skipping the directory /home 2026-02-20T11:11:09.188 INFO:teuthology.orchestra.run.trial109.stdout:skipping the directory /root 2026-02-20T11:11:09.188 INFO:teuthology.orchestra.run.trial109.stdout:skipping the directory /tmp 2026-02-20T11:11:09.188 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:11:09.587 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5357.g0c3148ca.el9.x86_64 36/39 2026-02-20T11:11:09.587 INFO:teuthology.orchestra.run.trial043.stdout:skipping the directory /sys 2026-02-20T11:11:09.587 INFO:teuthology.orchestra.run.trial043.stdout:skipping the directory /proc 2026-02-20T11:11:09.587 INFO:teuthology.orchestra.run.trial043.stdout:skipping the directory /mnt 2026-02-20T11:11:09.587 INFO:teuthology.orchestra.run.trial043.stdout:skipping the directory /var/tmp 2026-02-20T11:11:09.588 INFO:teuthology.orchestra.run.trial043.stdout:skipping the directory /home 2026-02-20T11:11:09.588 INFO:teuthology.orchestra.run.trial043.stdout:skipping the directory /root 2026-02-20T11:11:09.588 INFO:teuthology.orchestra.run.trial043.stdout:skipping the directory /tmp 2026-02-20T11:11:09.588 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:11:09.680 INFO:teuthology.orchestra.run.trial109.stdout: Installing : ceph-radosgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 37/39 2026-02-20T11:11:09.702 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 37/39 2026-02-20T11:11:09.703 INFO:teuthology.orchestra.run.trial109.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T11:11:09.703 INFO:teuthology.orchestra.run.trial109.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-20T11:11:09.703 INFO:teuthology.orchestra.run.trial109.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-20T11:11:09.703 INFO:teuthology.orchestra.run.trial109.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-20T11:11:09.703 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:11:09.704 INFO:teuthology.orchestra.run.trial109.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-20T11:11:09.715 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-20T11:11:09.715 INFO:teuthology.orchestra.run.trial109.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-20T11:11:09.769 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-20T11:11:09.770 INFO:teuthology.orchestra.run.trial109.stdout:Creating group 'qat' with GID 992. 2026-02-20T11:11:09.770 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:11:10.079 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : ceph-base-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/39 2026-02-20T11:11:10.080 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : ceph-common-2:20.3.0-5357.g0c3148ca.el9.x86_64 2/39 2026-02-20T11:11:10.080 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : ceph-radosgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 3/39 2026-02-20T11:11:10.080 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : ceph-selinux-2:20.3.0-5357.g0c3148ca.el9.x86_64 4/39 2026-02-20T11:11:10.080 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : libcephfs-daemon-2:20.3.0-5357.g0c3148ca.el9.x86_6 5/39 2026-02-20T11:11:10.080 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5357.g0c3148ca.el9.x86_6 6/39 2026-02-20T11:11:10.080 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : libcephfs2-2:20.3.0-5357.g0c3148ca.el9.x86_64 7/39 2026-02-20T11:11:10.081 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : libradosstriper1-2:20.3.0-5357.g0c3148ca.el9.x86_6 8/39 2026-02-20T11:11:10.081 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : librgw2-2:20.3.0-5357.g0c3148ca.el9.x86_64 9/39 2026-02-20T11:11:10.081 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5357.g0c3148ca.el9. 10/39 2026-02-20T11:11:10.081 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-ceph-common-2:20.3.0-5357.g0c3148ca.el9.x8 11/39 2026-02-20T11:11:10.081 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-cephfs-2:20.3.0-5357.g0c3148ca.el9.x86_64 12/39 2026-02-20T11:11:10.081 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-rados-2:20.3.0-5357.g0c3148ca.el9.x86_64 13/39 2026-02-20T11:11:10.082 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-rbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 14/39 2026-02-20T11:11:10.082 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-rgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 15/39 2026-02-20T11:11:10.082 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 16/39 2026-02-20T11:11:10.082 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 17/39 2026-02-20T11:11:10.082 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 18/39 2026-02-20T11:11:10.082 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 19/39 2026-02-20T11:11:10.083 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-20T11:11:10.083 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 21/39 2026-02-20T11:11:10.083 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 22/39 2026-02-20T11:11:10.083 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 23/39 2026-02-20T11:11:10.083 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/39 2026-02-20T11:11:10.083 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 25/39 2026-02-20T11:11:10.084 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 26/39 2026-02-20T11:11:10.084 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 27/39 2026-02-20T11:11:10.084 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 28/39 2026-02-20T11:11:10.084 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 29/39 2026-02-20T11:11:10.084 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 30/39 2026-02-20T11:11:10.084 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 31/39 2026-02-20T11:11:10.085 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 32/39 2026-02-20T11:11:10.085 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 33/39 2026-02-20T11:11:10.085 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : re2-1:20211101-20.el9.x86_64 34/39 2026-02-20T11:11:10.085 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 35/39 2026-02-20T11:11:10.085 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : librados2-2:20.3.0-5357.g0c3148ca.el9.x86_64 36/39 2026-02-20T11:11:10.085 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 37/39 2026-02-20T11:11:10.086 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : librbd1-2:20.3.0-5357.g0c3148ca.el9.x86_64 38/39 2026-02-20T11:11:10.087 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-radosgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 37/39 2026-02-20T11:11:10.105 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 37/39 2026-02-20T11:11:10.105 INFO:teuthology.orchestra.run.trial043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T11:11:10.105 INFO:teuthology.orchestra.run.trial043.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-20T11:11:10.105 INFO:teuthology.orchestra.run.trial043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-20T11:11:10.105 INFO:teuthology.orchestra.run.trial043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-20T11:11:10.105 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:11:10.107 INFO:teuthology.orchestra.run.trial043.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-20T11:11:10.119 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-20T11:11:10.119 INFO:teuthology.orchestra.run.trial043.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-20T11:11:10.140 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 39/39 2026-02-20T11:11:10.140 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:11:10.140 INFO:teuthology.orchestra.run.trial109.stdout:Upgraded: 2026-02-20T11:11:10.140 INFO:teuthology.orchestra.run.trial109.stdout: librados2-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:11:10.141 INFO:teuthology.orchestra.run.trial109.stdout: librbd1-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:11:10.141 INFO:teuthology.orchestra.run.trial109.stdout:Installed: 2026-02-20T11:11:10.141 INFO:teuthology.orchestra.run.trial109.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-20T11:11:10.141 INFO:teuthology.orchestra.run.trial109.stdout: ceph-base-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:11:10.141 INFO:teuthology.orchestra.run.trial109.stdout: ceph-common-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:11:10.141 INFO:teuthology.orchestra.run.trial109.stdout: ceph-radosgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:11:10.141 INFO:teuthology.orchestra.run.trial109.stdout: ceph-selinux-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:11:10.141 INFO:teuthology.orchestra.run.trial109.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-20T11:11:10.141 INFO:teuthology.orchestra.run.trial109.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-20T11:11:10.141 INFO:teuthology.orchestra.run.trial109.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-20T11:11:10.141 INFO:teuthology.orchestra.run.trial109.stdout: libcephfs-daemon-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:11:10.142 INFO:teuthology.orchestra.run.trial109.stdout: libcephfs-proxy2-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:11:10.142 INFO:teuthology.orchestra.run.trial109.stdout: libcephfs2-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:11:10.142 INFO:teuthology.orchestra.run.trial109.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-20T11:11:10.142 INFO:teuthology.orchestra.run.trial109.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-20T11:11:10.142 INFO:teuthology.orchestra.run.trial109.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-20T11:11:10.142 INFO:teuthology.orchestra.run.trial109.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-20T11:11:10.142 INFO:teuthology.orchestra.run.trial109.stdout: libradosstriper1-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:11:10.142 INFO:teuthology.orchestra.run.trial109.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-20T11:11:10.142 INFO:teuthology.orchestra.run.trial109.stdout: librgw2-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:11:10.142 INFO:teuthology.orchestra.run.trial109.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-20T11:11:10.142 INFO:teuthology.orchestra.run.trial109.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-20T11:11:10.142 INFO:teuthology.orchestra.run.trial109.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-20T11:11:10.143 INFO:teuthology.orchestra.run.trial109.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-20T11:11:10.143 INFO:teuthology.orchestra.run.trial109.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-20T11:11:10.143 INFO:teuthology.orchestra.run.trial109.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-20T11:11:10.143 INFO:teuthology.orchestra.run.trial109.stdout: python3-ceph-argparse-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:11:10.143 INFO:teuthology.orchestra.run.trial109.stdout: python3-ceph-common-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:11:10.143 INFO:teuthology.orchestra.run.trial109.stdout: python3-cephfs-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:11:10.143 INFO:teuthology.orchestra.run.trial109.stdout: python3-rados-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:11:10.143 INFO:teuthology.orchestra.run.trial109.stdout: python3-rbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:11:10.143 INFO:teuthology.orchestra.run.trial109.stdout: python3-rgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:11:10.143 INFO:teuthology.orchestra.run.trial109.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-20T11:11:10.143 INFO:teuthology.orchestra.run.trial109.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-20T11:11:10.143 INFO:teuthology.orchestra.run.trial109.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-20T11:11:10.143 INFO:teuthology.orchestra.run.trial109.stdout: re2-1:20211101-20.el9.x86_64 2026-02-20T11:11:10.144 INFO:teuthology.orchestra.run.trial109.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-20T11:11:10.144 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:11:10.144 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:11:10.175 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-20T11:11:10.175 INFO:teuthology.orchestra.run.trial043.stdout:Creating group 'qat' with GID 992. 2026-02-20T11:11:10.175 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:11:10.180 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install ceph-test 2026-02-20T11:11:10.480 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-base-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/39 2026-02-20T11:11:10.480 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-common-2:20.3.0-5357.g0c3148ca.el9.x86_64 2/39 2026-02-20T11:11:10.481 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-radosgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 3/39 2026-02-20T11:11:10.481 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-selinux-2:20.3.0-5357.g0c3148ca.el9.x86_64 4/39 2026-02-20T11:11:10.481 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : libcephfs-daemon-2:20.3.0-5357.g0c3148ca.el9.x86_6 5/39 2026-02-20T11:11:10.481 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5357.g0c3148ca.el9.x86_6 6/39 2026-02-20T11:11:10.481 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : libcephfs2-2:20.3.0-5357.g0c3148ca.el9.x86_64 7/39 2026-02-20T11:11:10.481 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : libradosstriper1-2:20.3.0-5357.g0c3148ca.el9.x86_6 8/39 2026-02-20T11:11:10.481 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : librgw2-2:20.3.0-5357.g0c3148ca.el9.x86_64 9/39 2026-02-20T11:11:10.481 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5357.g0c3148ca.el9. 10/39 2026-02-20T11:11:10.481 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-ceph-common-2:20.3.0-5357.g0c3148ca.el9.x8 11/39 2026-02-20T11:11:10.481 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-cephfs-2:20.3.0-5357.g0c3148ca.el9.x86_64 12/39 2026-02-20T11:11:10.481 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-rados-2:20.3.0-5357.g0c3148ca.el9.x86_64 13/39 2026-02-20T11:11:10.482 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-rbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 14/39 2026-02-20T11:11:10.482 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-rgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 15/39 2026-02-20T11:11:10.482 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 16/39 2026-02-20T11:11:10.482 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 17/39 2026-02-20T11:11:10.482 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 18/39 2026-02-20T11:11:10.482 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 19/39 2026-02-20T11:11:10.482 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-20T11:11:10.482 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 21/39 2026-02-20T11:11:10.482 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 22/39 2026-02-20T11:11:10.482 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 23/39 2026-02-20T11:11:10.482 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/39 2026-02-20T11:11:10.482 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 25/39 2026-02-20T11:11:10.483 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 26/39 2026-02-20T11:11:10.483 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 27/39 2026-02-20T11:11:10.483 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 28/39 2026-02-20T11:11:10.483 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 29/39 2026-02-20T11:11:10.483 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 30/39 2026-02-20T11:11:10.483 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 31/39 2026-02-20T11:11:10.483 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 32/39 2026-02-20T11:11:10.483 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 33/39 2026-02-20T11:11:10.483 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : re2-1:20211101-20.el9.x86_64 34/39 2026-02-20T11:11:10.483 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 35/39 2026-02-20T11:11:10.483 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : librados2-2:20.3.0-5357.g0c3148ca.el9.x86_64 36/39 2026-02-20T11:11:10.483 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 37/39 2026-02-20T11:11:10.483 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : librbd1-2:20.3.0-5357.g0c3148ca.el9.x86_64 38/39 2026-02-20T11:11:10.487 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:00:19 ago on Fri 20 Feb 2026 11:10:51 AM UTC. 2026-02-20T11:11:10.541 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 39/39 2026-02-20T11:11:10.542 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:11:10.542 INFO:teuthology.orchestra.run.trial043.stdout:Upgraded: 2026-02-20T11:11:10.542 INFO:teuthology.orchestra.run.trial043.stdout: librados2-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:11:10.542 INFO:teuthology.orchestra.run.trial043.stdout: librbd1-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:11:10.542 INFO:teuthology.orchestra.run.trial043.stdout:Installed: 2026-02-20T11:11:10.542 INFO:teuthology.orchestra.run.trial043.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-20T11:11:10.542 INFO:teuthology.orchestra.run.trial043.stdout: ceph-base-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:11:10.542 INFO:teuthology.orchestra.run.trial043.stdout: ceph-common-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:11:10.543 INFO:teuthology.orchestra.run.trial043.stdout: ceph-radosgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:11:10.543 INFO:teuthology.orchestra.run.trial043.stdout: ceph-selinux-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:11:10.543 INFO:teuthology.orchestra.run.trial043.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-20T11:11:10.543 INFO:teuthology.orchestra.run.trial043.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-20T11:11:10.543 INFO:teuthology.orchestra.run.trial043.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-20T11:11:10.543 INFO:teuthology.orchestra.run.trial043.stdout: libcephfs-daemon-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:11:10.543 INFO:teuthology.orchestra.run.trial043.stdout: libcephfs-proxy2-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:11:10.543 INFO:teuthology.orchestra.run.trial043.stdout: libcephfs2-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:11:10.543 INFO:teuthology.orchestra.run.trial043.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-20T11:11:10.543 INFO:teuthology.orchestra.run.trial043.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-20T11:11:10.543 INFO:teuthology.orchestra.run.trial043.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-20T11:11:10.544 INFO:teuthology.orchestra.run.trial043.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-20T11:11:10.544 INFO:teuthology.orchestra.run.trial043.stdout: libradosstriper1-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:11:10.544 INFO:teuthology.orchestra.run.trial043.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-20T11:11:10.544 INFO:teuthology.orchestra.run.trial043.stdout: librgw2-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:11:10.544 INFO:teuthology.orchestra.run.trial043.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-20T11:11:10.544 INFO:teuthology.orchestra.run.trial043.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-20T11:11:10.544 INFO:teuthology.orchestra.run.trial043.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-20T11:11:10.544 INFO:teuthology.orchestra.run.trial043.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-20T11:11:10.544 INFO:teuthology.orchestra.run.trial043.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-20T11:11:10.544 INFO:teuthology.orchestra.run.trial043.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-20T11:11:10.544 INFO:teuthology.orchestra.run.trial043.stdout: python3-ceph-argparse-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:11:10.544 INFO:teuthology.orchestra.run.trial043.stdout: python3-ceph-common-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:11:10.544 INFO:teuthology.orchestra.run.trial043.stdout: python3-cephfs-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:11:10.545 INFO:teuthology.orchestra.run.trial043.stdout: python3-rados-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:11:10.545 INFO:teuthology.orchestra.run.trial043.stdout: python3-rbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:11:10.545 INFO:teuthology.orchestra.run.trial043.stdout: python3-rgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:11:10.545 INFO:teuthology.orchestra.run.trial043.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-20T11:11:10.545 INFO:teuthology.orchestra.run.trial043.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-20T11:11:10.545 INFO:teuthology.orchestra.run.trial043.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-20T11:11:10.545 INFO:teuthology.orchestra.run.trial043.stdout: re2-1:20211101-20.el9.x86_64 2026-02-20T11:11:10.545 INFO:teuthology.orchestra.run.trial043.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-20T11:11:10.545 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:11:10.545 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:11:10.569 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:11:10.570 INFO:teuthology.orchestra.run.trial109.stdout:================================================================================ 2026-02-20T11:11:10.570 INFO:teuthology.orchestra.run.trial109.stdout: Package Arch Version Repository Size 2026-02-20T11:11:10.570 INFO:teuthology.orchestra.run.trial109.stdout:================================================================================ 2026-02-20T11:11:10.570 INFO:teuthology.orchestra.run.trial109.stdout:Installing: 2026-02-20T11:11:10.570 INFO:teuthology.orchestra.run.trial109.stdout: ceph-test x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 127 M 2026-02-20T11:11:10.570 INFO:teuthology.orchestra.run.trial109.stdout:Installing dependencies: 2026-02-20T11:11:10.571 INFO:teuthology.orchestra.run.trial109.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-20T11:11:10.571 INFO:teuthology.orchestra.run.trial109.stdout: libasan8 x86_64 15.2.1-7.el9 appstream 525 k 2026-02-20T11:11:10.571 INFO:teuthology.orchestra.run.trial109.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-20T11:11:10.571 INFO:teuthology.orchestra.run.trial109.stdout: libcephsqlite x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 153 k 2026-02-20T11:11:10.571 INFO:teuthology.orchestra.run.trial109.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-20T11:11:10.571 INFO:teuthology.orchestra.run.trial109.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-20T11:11:10.571 INFO:teuthology.orchestra.run.trial109.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-20T11:11:10.571 INFO:teuthology.orchestra.run.trial109.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-20T11:11:10.571 INFO:teuthology.orchestra.run.trial109.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-20T11:11:10.571 INFO:teuthology.orchestra.run.trial109.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-20T11:11:10.571 INFO:teuthology.orchestra.run.trial109.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-20T11:11:10.571 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:11:10.572 INFO:teuthology.orchestra.run.trial109.stdout:Transaction Summary 2026-02-20T11:11:10.572 INFO:teuthology.orchestra.run.trial109.stdout:================================================================================ 2026-02-20T11:11:10.572 INFO:teuthology.orchestra.run.trial109.stdout:Install 12 Packages 2026-02-20T11:11:10.572 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:11:10.573 INFO:teuthology.orchestra.run.trial109.stdout:Total download size: 132 M 2026-02-20T11:11:10.573 INFO:teuthology.orchestra.run.trial109.stdout:Installed size: 989 M 2026-02-20T11:11:10.573 INFO:teuthology.orchestra.run.trial109.stdout:Downloading Packages: 2026-02-20T11:11:10.584 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-test 2026-02-20T11:11:10.889 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:21 ago on Fri 20 Feb 2026 11:10:49 AM UTC. 2026-02-20T11:11:10.972 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:11:10.973 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-20T11:11:10.973 INFO:teuthology.orchestra.run.trial043.stdout: Package Arch Version Repository Size 2026-02-20T11:11:10.973 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-20T11:11:10.974 INFO:teuthology.orchestra.run.trial043.stdout:Installing: 2026-02-20T11:11:10.974 INFO:teuthology.orchestra.run.trial043.stdout: ceph-test x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 127 M 2026-02-20T11:11:10.974 INFO:teuthology.orchestra.run.trial043.stdout:Installing dependencies: 2026-02-20T11:11:10.974 INFO:teuthology.orchestra.run.trial043.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-20T11:11:10.974 INFO:teuthology.orchestra.run.trial043.stdout: libasan8 x86_64 15.2.1-7.el9 appstream 525 k 2026-02-20T11:11:10.974 INFO:teuthology.orchestra.run.trial043.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-20T11:11:10.974 INFO:teuthology.orchestra.run.trial043.stdout: libcephsqlite x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 153 k 2026-02-20T11:11:10.974 INFO:teuthology.orchestra.run.trial043.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-20T11:11:10.974 INFO:teuthology.orchestra.run.trial043.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-20T11:11:10.974 INFO:teuthology.orchestra.run.trial043.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-20T11:11:10.974 INFO:teuthology.orchestra.run.trial043.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-20T11:11:10.974 INFO:teuthology.orchestra.run.trial043.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-20T11:11:10.975 INFO:teuthology.orchestra.run.trial043.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-20T11:11:10.975 INFO:teuthology.orchestra.run.trial043.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-20T11:11:10.975 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:11:10.975 INFO:teuthology.orchestra.run.trial043.stdout:Transaction Summary 2026-02-20T11:11:10.975 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-20T11:11:10.975 INFO:teuthology.orchestra.run.trial043.stdout:Install 12 Packages 2026-02-20T11:11:10.975 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:11:10.975 INFO:teuthology.orchestra.run.trial043.stdout:Total download size: 132 M 2026-02-20T11:11:10.975 INFO:teuthology.orchestra.run.trial043.stdout:Installed size: 989 M 2026-02-20T11:11:10.975 INFO:teuthology.orchestra.run.trial043.stdout:Downloading Packages: 2026-02-20T11:11:11.435 INFO:teuthology.orchestra.run.trial043.stdout:(1/12): hwloc-libs-2.4.1-6.el9.x86_64.rpm 11 MB/s | 2.1 MB 00:00 2026-02-20T11:11:11.571 INFO:teuthology.orchestra.run.trial043.stdout:(2/12): libatomic-11.5.0-14.el9.x86_64.rpm 168 kB/s | 23 kB 00:00 2026-02-20T11:11:11.594 INFO:teuthology.orchestra.run.trial043.stdout:(3/12): lksctp-tools-1.0.19-2.el9.x86_64.rpm 4.0 MB/s | 94 kB 00:00 2026-02-20T11:11:11.691 INFO:teuthology.orchestra.run.trial043.stdout:(4/12): libcephsqlite-20.3.0-5357.g0c3148ca.el9 341 kB/s | 153 kB 00:00 2026-02-20T11:11:12.252 INFO:teuthology.orchestra.run.trial109.stdout:(1/12): hwloc-libs-2.4.1-6.el9.x86_64.rpm 2.3 MB/s | 2.1 MB 00:00 2026-02-20T11:11:12.329 INFO:teuthology.orchestra.run.trial109.stdout:(2/12): libcephsqlite-20.3.0-5357.g0c3148ca.el9 156 kB/s | 153 kB 00:00 2026-02-20T11:11:12.333 INFO:teuthology.orchestra.run.trial109.stdout:(3/12): libatomic-11.5.0-14.el9.x86_64.rpm 278 kB/s | 23 kB 00:00 2026-02-20T11:11:12.738 INFO:teuthology.orchestra.run.trial109.stdout:(4/12): lksctp-tools-1.0.19-2.el9.x86_64.rpm 230 kB/s | 94 kB 00:00 2026-02-20T11:11:12.862 INFO:teuthology.orchestra.run.trial043.stdout:(5/12): libasan8-15.2.1-7.el9.x86_64.rpm 414 kB/s | 525 kB 00:01 2026-02-20T11:11:13.016 INFO:teuthology.orchestra.run.trial109.stdout:(5/12): libasan8-15.2.1-7.el9.x86_64.rpm 769 kB/s | 525 kB 00:00 2026-02-20T11:11:13.022 INFO:teuthology.orchestra.run.trial109.stdout:(6/12): libubsan-11.5.0-14.el9.x86_64.rpm 619 kB/s | 176 kB 00:00 2026-02-20T11:11:13.037 INFO:teuthology.orchestra.run.trial043.stdout:(6/12): libxslt-1.1.34-12.el9.x86_64.rpm 1.3 MB/s | 233 kB 00:00 2026-02-20T11:11:13.103 INFO:teuthology.orchestra.run.trial043.stdout:(7/12): libubsan-11.5.0-14.el9.x86_64.rpm 125 kB/s | 176 kB 00:01 2026-02-20T11:11:13.138 INFO:teuthology.orchestra.run.trial109.stdout:(7/12): libxslt-1.1.34-12.el9.x86_64.rpm 1.9 MB/s | 233 kB 00:00 2026-02-20T11:11:13.241 INFO:teuthology.orchestra.run.trial109.stdout:(8/12): protobuf-3.14.0-17.el9.x86_64.rpm 4.6 MB/s | 1.0 MB 00:00 2026-02-20T11:11:13.266 INFO:teuthology.orchestra.run.trial109.stdout:(9/12): socat-1.7.4.1-8.el9.x86_64.rpm 2.3 MB/s | 303 kB 00:00 2026-02-20T11:11:13.303 INFO:teuthology.orchestra.run.trial043.stdout:(8/12): protobuf-3.14.0-17.el9.x86_64.rpm 3.8 MB/s | 1.0 MB 00:00 2026-02-20T11:11:13.344 INFO:teuthology.orchestra.run.trial109.stdout:(10/12): xmlstarlet-1.6.1-20.el9.x86_64.rpm 617 kB/s | 64 kB 00:00 2026-02-20T11:11:13.480 INFO:teuthology.orchestra.run.trial043.stdout:(9/12): xmlstarlet-1.6.1-20.el9.x86_64.rpm 359 kB/s | 64 kB 00:00 2026-02-20T11:11:13.675 INFO:teuthology.orchestra.run.trial109.stdout:(11/12): yaml-cpp-0.6.3-5.el9.x86_64.rpm 298 kB/s | 122 kB 00:00 2026-02-20T11:11:13.731 INFO:teuthology.orchestra.run.trial043.stdout:(10/12): yaml-cpp-0.6.3-5.el9.x86_64.rpm 486 kB/s | 122 kB 00:00 2026-02-20T11:11:14.050 INFO:teuthology.orchestra.run.trial043.stdout:(11/12): socat-1.7.4.1-8.el9.x86_64.rpm 320 kB/s | 303 kB 00:00 2026-02-20T11:11:35.120 INFO:teuthology.orchestra.run.trial043.stdout:(12/12): ceph-test-20.3.0-5357.g0c3148ca.el9.x8 5.3 MB/s | 127 MB 00:23 2026-02-20T11:11:35.122 INFO:teuthology.orchestra.run.trial043.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:11:35.123 INFO:teuthology.orchestra.run.trial043.stdout:Total 5.5 MB/s | 132 MB 00:24 2026-02-20T11:11:35.203 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction check 2026-02-20T11:11:35.214 INFO:teuthology.orchestra.run.trial043.stdout:Transaction check succeeded. 2026-02-20T11:11:35.214 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction test 2026-02-20T11:11:35.645 INFO:teuthology.orchestra.run.trial043.stdout:Transaction test succeeded. 2026-02-20T11:11:35.645 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction 2026-02-20T11:11:36.102 INFO:teuthology.orchestra.run.trial043.stdout: Preparing : 1/1 2026-02-20T11:11:36.110 INFO:teuthology.orchestra.run.trial043.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/12 2026-02-20T11:11:36.119 INFO:teuthology.orchestra.run.trial043.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/12 2026-02-20T11:11:36.124 INFO:teuthology.orchestra.run.trial043.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/12 2026-02-20T11:11:36.128 INFO:teuthology.orchestra.run.trial043.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/12 2026-02-20T11:11:36.132 INFO:teuthology.orchestra.run.trial043.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/12 2026-02-20T11:11:36.137 INFO:teuthology.orchestra.run.trial043.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 6/12 2026-02-20T11:11:36.138 INFO:teuthology.orchestra.run.trial043.stdout: Installing : libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-20T11:11:36.154 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-20T11:11:36.156 INFO:teuthology.orchestra.run.trial043.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 8/12 2026-02-20T11:11:36.166 INFO:teuthology.orchestra.run.trial043.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 9/12 2026-02-20T11:11:36.175 INFO:teuthology.orchestra.run.trial043.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 10/12 2026-02-20T11:11:36.176 INFO:teuthology.orchestra.run.trial043.stdout: Installing : libcephsqlite-2:20.3.0-5357.g0c3148ca.el9.x86_64 11/12 2026-02-20T11:11:36.612 INFO:teuthology.orchestra.run.trial109.stdout:(12/12): ceph-test-20.3.0-5357.g0c3148ca.el9.x8 5.0 MB/s | 127 MB 00:25 2026-02-20T11:11:36.613 INFO:teuthology.orchestra.run.trial109.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:11:36.614 INFO:teuthology.orchestra.run.trial109.stdout:Total 5.1 MB/s | 132 MB 00:26 2026-02-20T11:11:36.688 INFO:teuthology.orchestra.run.trial109.stdout:Running transaction check 2026-02-20T11:11:36.699 INFO:teuthology.orchestra.run.trial109.stdout:Transaction check succeeded. 2026-02-20T11:11:36.699 INFO:teuthology.orchestra.run.trial109.stdout:Running transaction test 2026-02-20T11:11:37.128 INFO:teuthology.orchestra.run.trial109.stdout:Transaction test succeeded. 2026-02-20T11:11:37.129 INFO:teuthology.orchestra.run.trial109.stdout:Running transaction 2026-02-20T11:11:37.584 INFO:teuthology.orchestra.run.trial109.stdout: Preparing : 1/1 2026-02-20T11:11:37.593 INFO:teuthology.orchestra.run.trial109.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/12 2026-02-20T11:11:37.602 INFO:teuthology.orchestra.run.trial109.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/12 2026-02-20T11:11:37.607 INFO:teuthology.orchestra.run.trial109.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/12 2026-02-20T11:11:37.611 INFO:teuthology.orchestra.run.trial109.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/12 2026-02-20T11:11:37.613 INFO:teuthology.orchestra.run.trial109.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/12 2026-02-20T11:11:37.618 INFO:teuthology.orchestra.run.trial109.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 6/12 2026-02-20T11:11:37.619 INFO:teuthology.orchestra.run.trial109.stdout: Installing : libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-20T11:11:37.637 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-20T11:11:37.640 INFO:teuthology.orchestra.run.trial109.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 8/12 2026-02-20T11:11:37.650 INFO:teuthology.orchestra.run.trial109.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 9/12 2026-02-20T11:11:37.658 INFO:teuthology.orchestra.run.trial109.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 10/12 2026-02-20T11:11:37.659 INFO:teuthology.orchestra.run.trial109.stdout: Installing : libcephsqlite-2:20.3.0-5357.g0c3148ca.el9.x86_64 11/12 2026-02-20T11:11:43.125 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5357.g0c3148ca.el9.x86_64 11/12 2026-02-20T11:11:43.143 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-test-2:20.3.0-5357.g0c3148ca.el9.x86_64 12/12 2026-02-20T11:11:43.913 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-test-2:20.3.0-5357.g0c3148ca.el9.x86_64 12/12 2026-02-20T11:11:43.913 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-test-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/12 2026-02-20T11:11:43.913 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : libcephsqlite-2:20.3.0-5357.g0c3148ca.el9.x86_64 2/12 2026-02-20T11:11:43.913 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/12 2026-02-20T11:11:43.913 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/12 2026-02-20T11:11:43.913 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/12 2026-02-20T11:11:43.914 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : libasan8-15.2.1-7.el9.x86_64 6/12 2026-02-20T11:11:43.914 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 7/12 2026-02-20T11:11:43.914 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 8/12 2026-02-20T11:11:43.914 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 9/12 2026-02-20T11:11:43.914 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 10/12 2026-02-20T11:11:43.914 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 11/12 2026-02-20T11:11:43.967 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 12/12 2026-02-20T11:11:43.967 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:11:43.967 INFO:teuthology.orchestra.run.trial043.stdout:Installed: 2026-02-20T11:11:43.967 INFO:teuthology.orchestra.run.trial043.stdout: ceph-test-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:11:43.967 INFO:teuthology.orchestra.run.trial043.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-20T11:11:43.967 INFO:teuthology.orchestra.run.trial043.stdout: libasan8-15.2.1-7.el9.x86_64 2026-02-20T11:11:43.967 INFO:teuthology.orchestra.run.trial043.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-20T11:11:43.967 INFO:teuthology.orchestra.run.trial043.stdout: libcephsqlite-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:11:43.967 INFO:teuthology.orchestra.run.trial043.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-20T11:11:43.967 INFO:teuthology.orchestra.run.trial043.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-20T11:11:43.967 INFO:teuthology.orchestra.run.trial043.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-20T11:11:43.967 INFO:teuthology.orchestra.run.trial043.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-20T11:11:43.967 INFO:teuthology.orchestra.run.trial043.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-20T11:11:43.967 INFO:teuthology.orchestra.run.trial043.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-20T11:11:43.967 INFO:teuthology.orchestra.run.trial043.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-20T11:11:43.967 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:11:43.967 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:11:44.010 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph 2026-02-20T11:11:44.315 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:55 ago on Fri 20 Feb 2026 11:10:49 AM UTC. 2026-02-20T11:11:44.402 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:11:44.405 INFO:teuthology.orchestra.run.trial043.stdout:======================================================================================= 2026-02-20T11:11:44.405 INFO:teuthology.orchestra.run.trial043.stdout: Package Arch Version Repository Size 2026-02-20T11:11:44.405 INFO:teuthology.orchestra.run.trial043.stdout:======================================================================================= 2026-02-20T11:11:44.405 INFO:teuthology.orchestra.run.trial043.stdout:Installing: 2026-02-20T11:11:44.405 INFO:teuthology.orchestra.run.trial043.stdout: ceph x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 6.5 k 2026-02-20T11:11:44.405 INFO:teuthology.orchestra.run.trial043.stdout:Installing dependencies: 2026-02-20T11:11:44.405 INFO:teuthology.orchestra.run.trial043.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-20T11:11:44.405 INFO:teuthology.orchestra.run.trial043.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 57 k 2026-02-20T11:11:44.405 INFO:teuthology.orchestra.run.trial043.stdout: ceph-mds x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 2.0 M 2026-02-20T11:11:44.405 INFO:teuthology.orchestra.run.trial043.stdout: ceph-mgr x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 937 k 2026-02-20T11:11:44.406 INFO:teuthology.orchestra.run.trial043.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 297 k 2026-02-20T11:11:44.406 INFO:teuthology.orchestra.run.trial043.stdout: ceph-mon x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 4.9 M 2026-02-20T11:11:44.406 INFO:teuthology.orchestra.run.trial043.stdout: ceph-osd x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 163 k 2026-02-20T11:11:44.406 INFO:teuthology.orchestra.run.trial043.stdout: ceph-osd-classic x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 15 M 2026-02-20T11:11:44.406 INFO:teuthology.orchestra.run.trial043.stdout: ceph-osd-crimson x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 88 M 2026-02-20T11:11:44.406 INFO:teuthology.orchestra.run.trial043.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 17 k 2026-02-20T11:11:44.406 INFO:teuthology.orchestra.run.trial043.stdout: cephadm noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 1.0 M 2026-02-20T11:11:44.406 INFO:teuthology.orchestra.run.trial043.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-20T11:11:44.406 INFO:teuthology.orchestra.run.trial043.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-20T11:11:44.406 INFO:teuthology.orchestra.run.trial043.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-20T11:11:44.406 INFO:teuthology.orchestra.run.trial043.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-20T11:11:44.406 INFO:teuthology.orchestra.run.trial043.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-20T11:11:44.407 INFO:teuthology.orchestra.run.trial043.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-20T11:11:44.407 INFO:teuthology.orchestra.run.trial043.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-20T11:11:44.407 INFO:teuthology.orchestra.run.trial043.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-20T11:11:44.407 INFO:teuthology.orchestra.run.trial043.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-20T11:11:44.407 INFO:teuthology.orchestra.run.trial043.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-20T11:11:44.407 INFO:teuthology.orchestra.run.trial043.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-20T11:11:44.407 INFO:teuthology.orchestra.run.trial043.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-20T11:11:44.407 INFO:teuthology.orchestra.run.trial043.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-20T11:11:44.407 INFO:teuthology.orchestra.run.trial043.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-20T11:11:44.407 INFO:teuthology.orchestra.run.trial043.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-20T11:11:44.407 INFO:teuthology.orchestra.run.trial043.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-20T11:11:44.407 INFO:teuthology.orchestra.run.trial043.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-20T11:11:44.407 INFO:teuthology.orchestra.run.trial043.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-20T11:11:44.408 INFO:teuthology.orchestra.run.trial043.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-20T11:11:44.408 INFO:teuthology.orchestra.run.trial043.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-20T11:11:44.408 INFO:teuthology.orchestra.run.trial043.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-20T11:11:44.408 INFO:teuthology.orchestra.run.trial043.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-20T11:11:44.408 INFO:teuthology.orchestra.run.trial043.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-20T11:11:44.408 INFO:teuthology.orchestra.run.trial043.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-20T11:11:44.408 INFO:teuthology.orchestra.run.trial043.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-20T11:11:44.408 INFO:teuthology.orchestra.run.trial043.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-20T11:11:44.408 INFO:teuthology.orchestra.run.trial043.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-20T11:11:44.408 INFO:teuthology.orchestra.run.trial043.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-20T11:11:44.408 INFO:teuthology.orchestra.run.trial043.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-20T11:11:44.408 INFO:teuthology.orchestra.run.trial043.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-20T11:11:44.408 INFO:teuthology.orchestra.run.trial043.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-20T11:11:44.409 INFO:teuthology.orchestra.run.trial043.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-20T11:11:44.409 INFO:teuthology.orchestra.run.trial043.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-20T11:11:44.409 INFO:teuthology.orchestra.run.trial043.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-20T11:11:44.409 INFO:teuthology.orchestra.run.trial043.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-20T11:11:44.409 INFO:teuthology.orchestra.run.trial043.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-20T11:11:44.409 INFO:teuthology.orchestra.run.trial043.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-20T11:11:44.409 INFO:teuthology.orchestra.run.trial043.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-20T11:11:44.409 INFO:teuthology.orchestra.run.trial043.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-20T11:11:44.409 INFO:teuthology.orchestra.run.trial043.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-20T11:11:44.409 INFO:teuthology.orchestra.run.trial043.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-20T11:11:44.409 INFO:teuthology.orchestra.run.trial043.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-20T11:11:44.409 INFO:teuthology.orchestra.run.trial043.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-20T11:11:44.409 INFO:teuthology.orchestra.run.trial043.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-20T11:11:44.410 INFO:teuthology.orchestra.run.trial043.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-20T11:11:44.410 INFO:teuthology.orchestra.run.trial043.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-20T11:11:44.410 INFO:teuthology.orchestra.run.trial043.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-20T11:11:44.410 INFO:teuthology.orchestra.run.trial043.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-20T11:11:44.410 INFO:teuthology.orchestra.run.trial043.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-20T11:11:44.410 INFO:teuthology.orchestra.run.trial043.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-20T11:11:44.410 INFO:teuthology.orchestra.run.trial043.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-20T11:11:44.410 INFO:teuthology.orchestra.run.trial043.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-20T11:11:44.410 INFO:teuthology.orchestra.run.trial043.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-20T11:11:44.410 INFO:teuthology.orchestra.run.trial043.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-20T11:11:44.410 INFO:teuthology.orchestra.run.trial043.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2026-02-20T11:11:44.410 INFO:teuthology.orchestra.run.trial043.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2026-02-20T11:11:44.410 INFO:teuthology.orchestra.run.trial043.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-20T11:11:44.411 INFO:teuthology.orchestra.run.trial043.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-20T11:11:44.411 INFO:teuthology.orchestra.run.trial043.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-20T11:11:44.411 INFO:teuthology.orchestra.run.trial043.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-20T11:11:44.411 INFO:teuthology.orchestra.run.trial043.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-20T11:11:44.411 INFO:teuthology.orchestra.run.trial043.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-20T11:11:44.411 INFO:teuthology.orchestra.run.trial043.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-20T11:11:44.411 INFO:teuthology.orchestra.run.trial043.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-20T11:11:44.411 INFO:teuthology.orchestra.run.trial043.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-20T11:11:44.411 INFO:teuthology.orchestra.run.trial043.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-20T11:11:44.411 INFO:teuthology.orchestra.run.trial043.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-20T11:11:44.411 INFO:teuthology.orchestra.run.trial043.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-20T11:11:44.411 INFO:teuthology.orchestra.run.trial043.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-20T11:11:44.411 INFO:teuthology.orchestra.run.trial043.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-20T11:11:44.412 INFO:teuthology.orchestra.run.trial043.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-20T11:11:44.412 INFO:teuthology.orchestra.run.trial043.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-20T11:11:44.412 INFO:teuthology.orchestra.run.trial043.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-20T11:11:44.412 INFO:teuthology.orchestra.run.trial043.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-20T11:11:44.412 INFO:teuthology.orchestra.run.trial043.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-20T11:11:44.412 INFO:teuthology.orchestra.run.trial043.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-20T11:11:44.412 INFO:teuthology.orchestra.run.trial043.stdout:Installing weak dependencies: 2026-02-20T11:11:44.412 INFO:teuthology.orchestra.run.trial043.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 184 k 2026-02-20T11:11:44.412 INFO:teuthology.orchestra.run.trial043.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 51 M 2026-02-20T11:11:44.412 INFO:teuthology.orchestra.run.trial043.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 7.4 M 2026-02-20T11:11:44.412 INFO:teuthology.orchestra.run.trial043.stdout: ceph-mgr-k8sevents noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 22 k 2026-02-20T11:11:44.412 INFO:teuthology.orchestra.run.trial043.stdout: ceph-mgr-rook noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 50 k 2026-02-20T11:11:44.412 INFO:teuthology.orchestra.run.trial043.stdout: ceph-volume noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 301 k 2026-02-20T11:11:44.413 INFO:teuthology.orchestra.run.trial043.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-20T11:11:44.413 INFO:teuthology.orchestra.run.trial043.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-20T11:11:44.413 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:11:44.413 INFO:teuthology.orchestra.run.trial043.stdout:Transaction Summary 2026-02-20T11:11:44.413 INFO:teuthology.orchestra.run.trial043.stdout:======================================================================================= 2026-02-20T11:11:44.413 INFO:teuthology.orchestra.run.trial043.stdout:Install 96 Packages 2026-02-20T11:11:44.413 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:11:44.414 INFO:teuthology.orchestra.run.trial043.stdout:Total download size: 226 M 2026-02-20T11:11:44.414 INFO:teuthology.orchestra.run.trial043.stdout:Installed size: 1.5 G 2026-02-20T11:11:44.414 INFO:teuthology.orchestra.run.trial043.stdout:Downloading Packages: 2026-02-20T11:11:44.590 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5357.g0c3148ca.el9.x86_64 11/12 2026-02-20T11:11:44.608 INFO:teuthology.orchestra.run.trial109.stdout: Installing : ceph-test-2:20.3.0-5357.g0c3148ca.el9.x86_64 12/12 2026-02-20T11:11:45.169 INFO:teuthology.orchestra.run.trial043.stdout:(1/96): ceph-20.3.0-5357.g0c3148ca.el9.x86_64.r 124 kB/s | 6.5 kB 00:00 2026-02-20T11:11:45.257 INFO:teuthology.orchestra.run.trial043.stdout:(2/96): ceph-mgr-20.3.0-5357.g0c3148ca.el9.x86_ 6.5 MB/s | 937 kB 00:00 2026-02-20T11:11:45.276 INFO:teuthology.orchestra.run.trial043.stdout:(3/96): ceph-mds-20.3.0-5357.g0c3148ca.el9.x86_ 13 MB/s | 2.0 MB 00:00 2026-02-20T11:11:45.279 INFO:teuthology.orchestra.run.trial043.stdout:(4/96): ceph-osd-20.3.0-5357.g0c3148ca.el9.x86_ 7.1 MB/s | 163 kB 00:00 2026-02-20T11:11:45.407 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: ceph-test-2:20.3.0-5357.g0c3148ca.el9.x86_64 12/12 2026-02-20T11:11:45.407 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : ceph-test-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/12 2026-02-20T11:11:45.407 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : libcephsqlite-2:20.3.0-5357.g0c3148ca.el9.x86_64 2/12 2026-02-20T11:11:45.407 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/12 2026-02-20T11:11:45.407 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/12 2026-02-20T11:11:45.407 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/12 2026-02-20T11:11:45.408 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : libasan8-15.2.1-7.el9.x86_64 6/12 2026-02-20T11:11:45.408 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 7/12 2026-02-20T11:11:45.408 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 8/12 2026-02-20T11:11:45.408 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 9/12 2026-02-20T11:11:45.408 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 10/12 2026-02-20T11:11:45.408 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 11/12 2026-02-20T11:11:45.455 INFO:teuthology.orchestra.run.trial043.stdout:(5/96): ceph-mon-20.3.0-5357.g0c3148ca.el9.x86_ 17 MB/s | 4.9 MB 00:00 2026-02-20T11:11:45.460 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 12/12 2026-02-20T11:11:45.460 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:11:45.460 INFO:teuthology.orchestra.run.trial109.stdout:Installed: 2026-02-20T11:11:45.460 INFO:teuthology.orchestra.run.trial109.stdout: ceph-test-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:11:45.460 INFO:teuthology.orchestra.run.trial109.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-20T11:11:45.461 INFO:teuthology.orchestra.run.trial109.stdout: libasan8-15.2.1-7.el9.x86_64 2026-02-20T11:11:45.461 INFO:teuthology.orchestra.run.trial109.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-20T11:11:45.461 INFO:teuthology.orchestra.run.trial109.stdout: libcephsqlite-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:11:45.461 INFO:teuthology.orchestra.run.trial109.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-20T11:11:45.461 INFO:teuthology.orchestra.run.trial109.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-20T11:11:45.461 INFO:teuthology.orchestra.run.trial109.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-20T11:11:45.461 INFO:teuthology.orchestra.run.trial109.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-20T11:11:45.461 INFO:teuthology.orchestra.run.trial109.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-20T11:11:45.461 INFO:teuthology.orchestra.run.trial109.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-20T11:11:45.461 INFO:teuthology.orchestra.run.trial109.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-20T11:11:45.461 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:11:45.462 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:11:45.499 INFO:teuthology.orchestra.run.trial043.stdout:(6/96): ceph-grafana-dashboards-20.3.0-5357.g0c 1.3 MB/s | 57 kB 00:00 2026-02-20T11:11:45.502 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install ceph 2026-02-20T11:11:45.563 INFO:teuthology.orchestra.run.trial043.stdout:(7/96): ceph-mgr-cephadm-20.3.0-5357.g0c3148ca. 2.9 MB/s | 184 kB 00:00 2026-02-20T11:11:45.805 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:00:54 ago on Fri 20 Feb 2026 11:10:51 AM UTC. 2026-02-20T11:11:45.848 INFO:teuthology.orchestra.run.trial043.stdout:(8/96): ceph-osd-classic-20.3.0-5357.g0c3148ca. 26 MB/s | 15 MB 00:00 2026-02-20T11:11:45.892 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:11:45.894 INFO:teuthology.orchestra.run.trial109.stdout:======================================================================================= 2026-02-20T11:11:45.894 INFO:teuthology.orchestra.run.trial109.stdout: Package Arch Version Repository Size 2026-02-20T11:11:45.895 INFO:teuthology.orchestra.run.trial109.stdout:======================================================================================= 2026-02-20T11:11:45.895 INFO:teuthology.orchestra.run.trial109.stdout:Installing: 2026-02-20T11:11:45.895 INFO:teuthology.orchestra.run.trial109.stdout: ceph x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 6.5 k 2026-02-20T11:11:45.895 INFO:teuthology.orchestra.run.trial109.stdout:Installing dependencies: 2026-02-20T11:11:45.895 INFO:teuthology.orchestra.run.trial109.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-20T11:11:45.895 INFO:teuthology.orchestra.run.trial109.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 57 k 2026-02-20T11:11:45.895 INFO:teuthology.orchestra.run.trial109.stdout: ceph-mds x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 2.0 M 2026-02-20T11:11:45.895 INFO:teuthology.orchestra.run.trial109.stdout: ceph-mgr x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 937 k 2026-02-20T11:11:45.895 INFO:teuthology.orchestra.run.trial109.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 297 k 2026-02-20T11:11:45.895 INFO:teuthology.orchestra.run.trial109.stdout: ceph-mon x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 4.9 M 2026-02-20T11:11:45.896 INFO:teuthology.orchestra.run.trial109.stdout: ceph-osd x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 163 k 2026-02-20T11:11:45.896 INFO:teuthology.orchestra.run.trial109.stdout: ceph-osd-classic x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 15 M 2026-02-20T11:11:45.896 INFO:teuthology.orchestra.run.trial109.stdout: ceph-osd-crimson x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 88 M 2026-02-20T11:11:45.896 INFO:teuthology.orchestra.run.trial109.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 17 k 2026-02-20T11:11:45.896 INFO:teuthology.orchestra.run.trial109.stdout: cephadm noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 1.0 M 2026-02-20T11:11:45.896 INFO:teuthology.orchestra.run.trial109.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-20T11:11:45.896 INFO:teuthology.orchestra.run.trial109.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-20T11:11:45.896 INFO:teuthology.orchestra.run.trial109.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-20T11:11:45.896 INFO:teuthology.orchestra.run.trial109.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-20T11:11:45.896 INFO:teuthology.orchestra.run.trial109.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-20T11:11:45.896 INFO:teuthology.orchestra.run.trial109.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-20T11:11:45.896 INFO:teuthology.orchestra.run.trial109.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-20T11:11:45.896 INFO:teuthology.orchestra.run.trial109.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-20T11:11:45.897 INFO:teuthology.orchestra.run.trial109.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-20T11:11:45.897 INFO:teuthology.orchestra.run.trial109.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-20T11:11:45.897 INFO:teuthology.orchestra.run.trial109.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-20T11:11:45.897 INFO:teuthology.orchestra.run.trial109.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-20T11:11:45.897 INFO:teuthology.orchestra.run.trial109.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-20T11:11:45.897 INFO:teuthology.orchestra.run.trial109.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-20T11:11:45.897 INFO:teuthology.orchestra.run.trial109.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-20T11:11:45.897 INFO:teuthology.orchestra.run.trial109.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-20T11:11:45.897 INFO:teuthology.orchestra.run.trial109.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-20T11:11:45.897 INFO:teuthology.orchestra.run.trial109.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-20T11:11:45.897 INFO:teuthology.orchestra.run.trial109.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-20T11:11:45.897 INFO:teuthology.orchestra.run.trial109.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-20T11:11:45.898 INFO:teuthology.orchestra.run.trial109.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-20T11:11:45.898 INFO:teuthology.orchestra.run.trial109.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-20T11:11:45.898 INFO:teuthology.orchestra.run.trial109.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-20T11:11:45.898 INFO:teuthology.orchestra.run.trial109.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-20T11:11:45.898 INFO:teuthology.orchestra.run.trial109.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-20T11:11:45.898 INFO:teuthology.orchestra.run.trial109.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-20T11:11:45.898 INFO:teuthology.orchestra.run.trial109.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-20T11:11:45.898 INFO:teuthology.orchestra.run.trial109.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-20T11:11:45.898 INFO:teuthology.orchestra.run.trial109.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-20T11:11:45.898 INFO:teuthology.orchestra.run.trial109.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-20T11:11:45.898 INFO:teuthology.orchestra.run.trial109.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-20T11:11:45.898 INFO:teuthology.orchestra.run.trial109.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-20T11:11:45.898 INFO:teuthology.orchestra.run.trial109.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-20T11:11:45.899 INFO:teuthology.orchestra.run.trial109.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-20T11:11:45.899 INFO:teuthology.orchestra.run.trial109.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-20T11:11:45.899 INFO:teuthology.orchestra.run.trial109.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-20T11:11:45.899 INFO:teuthology.orchestra.run.trial109.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-20T11:11:45.899 INFO:teuthology.orchestra.run.trial109.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-20T11:11:45.899 INFO:teuthology.orchestra.run.trial109.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-20T11:11:45.899 INFO:teuthology.orchestra.run.trial109.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-20T11:11:45.899 INFO:teuthology.orchestra.run.trial109.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-20T11:11:45.899 INFO:teuthology.orchestra.run.trial109.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-20T11:11:45.899 INFO:teuthology.orchestra.run.trial109.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-20T11:11:45.899 INFO:teuthology.orchestra.run.trial109.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-20T11:11:45.899 INFO:teuthology.orchestra.run.trial109.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-20T11:11:45.899 INFO:teuthology.orchestra.run.trial109.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-20T11:11:45.900 INFO:teuthology.orchestra.run.trial109.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-20T11:11:45.900 INFO:teuthology.orchestra.run.trial109.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-20T11:11:45.900 INFO:teuthology.orchestra.run.trial109.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-20T11:11:45.900 INFO:teuthology.orchestra.run.trial109.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-20T11:11:45.900 INFO:teuthology.orchestra.run.trial109.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-20T11:11:45.900 INFO:teuthology.orchestra.run.trial109.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-20T11:11:45.900 INFO:teuthology.orchestra.run.trial109.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-20T11:11:45.900 INFO:teuthology.orchestra.run.trial109.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-20T11:11:45.900 INFO:teuthology.orchestra.run.trial109.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2026-02-20T11:11:45.900 INFO:teuthology.orchestra.run.trial109.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2026-02-20T11:11:45.900 INFO:teuthology.orchestra.run.trial109.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-20T11:11:45.900 INFO:teuthology.orchestra.run.trial109.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-20T11:11:45.900 INFO:teuthology.orchestra.run.trial109.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-20T11:11:45.901 INFO:teuthology.orchestra.run.trial109.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-20T11:11:45.901 INFO:teuthology.orchestra.run.trial109.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-20T11:11:45.901 INFO:teuthology.orchestra.run.trial109.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-20T11:11:45.901 INFO:teuthology.orchestra.run.trial109.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-20T11:11:45.901 INFO:teuthology.orchestra.run.trial109.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-20T11:11:45.901 INFO:teuthology.orchestra.run.trial109.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-20T11:11:45.901 INFO:teuthology.orchestra.run.trial109.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-20T11:11:45.901 INFO:teuthology.orchestra.run.trial109.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-20T11:11:45.901 INFO:teuthology.orchestra.run.trial109.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-20T11:11:45.901 INFO:teuthology.orchestra.run.trial109.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-20T11:11:45.901 INFO:teuthology.orchestra.run.trial109.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-20T11:11:45.902 INFO:teuthology.orchestra.run.trial109.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-20T11:11:45.902 INFO:teuthology.orchestra.run.trial109.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-20T11:11:45.902 INFO:teuthology.orchestra.run.trial109.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-20T11:11:45.902 INFO:teuthology.orchestra.run.trial109.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-20T11:11:45.902 INFO:teuthology.orchestra.run.trial109.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-20T11:11:45.902 INFO:teuthology.orchestra.run.trial109.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-20T11:11:45.902 INFO:teuthology.orchestra.run.trial109.stdout:Installing weak dependencies: 2026-02-20T11:11:45.902 INFO:teuthology.orchestra.run.trial109.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 184 k 2026-02-20T11:11:45.902 INFO:teuthology.orchestra.run.trial109.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 51 M 2026-02-20T11:11:45.902 INFO:teuthology.orchestra.run.trial109.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 7.4 M 2026-02-20T11:11:45.902 INFO:teuthology.orchestra.run.trial109.stdout: ceph-mgr-k8sevents noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 22 k 2026-02-20T11:11:45.902 INFO:teuthology.orchestra.run.trial109.stdout: ceph-mgr-rook noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 50 k 2026-02-20T11:11:45.903 INFO:teuthology.orchestra.run.trial109.stdout: ceph-volume noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 301 k 2026-02-20T11:11:45.903 INFO:teuthology.orchestra.run.trial109.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-20T11:11:45.903 INFO:teuthology.orchestra.run.trial109.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-20T11:11:45.903 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:11:45.903 INFO:teuthology.orchestra.run.trial109.stdout:Transaction Summary 2026-02-20T11:11:45.903 INFO:teuthology.orchestra.run.trial109.stdout:======================================================================================= 2026-02-20T11:11:45.903 INFO:teuthology.orchestra.run.trial109.stdout:Install 96 Packages 2026-02-20T11:11:45.903 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:11:45.904 INFO:teuthology.orchestra.run.trial109.stdout:Total download size: 226 M 2026-02-20T11:11:45.904 INFO:teuthology.orchestra.run.trial109.stdout:Installed size: 1.5 G 2026-02-20T11:11:45.904 INFO:teuthology.orchestra.run.trial109.stdout:Downloading Packages: 2026-02-20T11:11:46.168 INFO:teuthology.orchestra.run.trial043.stdout:(9/96): ceph-mgr-diskprediction-local-20.3.0-53 23 MB/s | 7.4 MB 00:00 2026-02-20T11:11:46.199 INFO:teuthology.orchestra.run.trial043.stdout:(10/96): ceph-mgr-k8sevents-20.3.0-5357.g0c3148 714 kB/s | 22 kB 00:00 2026-02-20T11:11:46.237 INFO:teuthology.orchestra.run.trial043.stdout:(11/96): ceph-mgr-modules-core-20.3.0-5357.g0c3 7.7 MB/s | 297 kB 00:00 2026-02-20T11:11:46.270 INFO:teuthology.orchestra.run.trial043.stdout:(12/96): ceph-mgr-rook-20.3.0-5357.g0c3148ca.el 1.5 MB/s | 50 kB 00:00 2026-02-20T11:11:46.302 INFO:teuthology.orchestra.run.trial043.stdout:(13/96): ceph-prometheus-alerts-20.3.0-5357.g0c 541 kB/s | 17 kB 00:00 2026-02-20T11:11:46.339 INFO:teuthology.orchestra.run.trial043.stdout:(14/96): ceph-volume-20.3.0-5357.g0c3148ca.el9. 7.9 MB/s | 301 kB 00:00 2026-02-20T11:11:46.391 INFO:teuthology.orchestra.run.trial043.stdout:(15/96): cephadm-20.3.0-5357.g0c3148ca.el9.noar 20 MB/s | 1.0 MB 00:00 2026-02-20T11:11:46.581 INFO:teuthology.orchestra.run.trial043.stdout:(16/96): cryptsetup-2.8.1-3.el9.x86_64.rpm 1.8 MB/s | 351 kB 00:00 2026-02-20T11:11:46.602 INFO:teuthology.orchestra.run.trial043.stdout:(17/96): ledmon-libs-1.1.0-3.el9.x86_64.rpm 1.8 MB/s | 40 kB 00:00 2026-02-20T11:11:46.624 INFO:teuthology.orchestra.run.trial043.stdout:(18/96): libconfig-1.7.2-9.el9.x86_64.rpm 3.2 MB/s | 72 kB 00:00 2026-02-20T11:11:46.668 INFO:teuthology.orchestra.run.trial043.stdout:(19/96): libgfortran-11.5.0-14.el9.x86_64.rpm 18 MB/s | 794 kB 00:00 2026-02-20T11:11:46.692 INFO:teuthology.orchestra.run.trial043.stdout:(20/96): libquadmath-11.5.0-14.el9.x86_64.rpm 7.4 MB/s | 184 kB 00:00 2026-02-20T11:11:46.702 INFO:teuthology.orchestra.run.trial109.stdout:(1/96): ceph-20.3.0-5357.g0c3148ca.el9.x86_64.r 45 kB/s | 6.5 kB 00:00 2026-02-20T11:11:46.716 INFO:teuthology.orchestra.run.trial043.stdout:(21/96): python3-cffi-1.14.5-5.el9.x86_64.rpm 10 MB/s | 253 kB 00:00 2026-02-20T11:11:46.763 INFO:teuthology.orchestra.run.trial043.stdout:(22/96): python3-cryptography-36.0.1-5.el9.x86_ 27 MB/s | 1.2 MB 00:00 2026-02-20T11:11:46.786 INFO:teuthology.orchestra.run.trial043.stdout:(23/96): python3-ply-3.11-14.el9.noarch.rpm 4.5 MB/s | 106 kB 00:00 2026-02-20T11:11:46.809 INFO:teuthology.orchestra.run.trial043.stdout:(24/96): python3-pycparser-2.20-6.el9.noarch.rp 5.6 MB/s | 135 kB 00:00 2026-02-20T11:11:46.833 INFO:teuthology.orchestra.run.trial043.stdout:(25/96): python3-pyparsing-2.4.7-9.el9.noarch.r 6.2 MB/s | 150 kB 00:00 2026-02-20T11:11:46.858 INFO:teuthology.orchestra.run.trial043.stdout:(26/96): python3-requests-2.25.1-10.el9.noarch. 5.1 MB/s | 126 kB 00:00 2026-02-20T11:11:46.882 INFO:teuthology.orchestra.run.trial043.stdout:(27/96): python3-urllib3-1.26.5-7.el9.noarch.rp 8.9 MB/s | 218 kB 00:00 2026-02-20T11:11:46.906 INFO:teuthology.orchestra.run.trial043.stdout:(28/96): unzip-6.0-59.el9.x86_64.rpm 7.3 MB/s | 182 kB 00:00 2026-02-20T11:11:46.929 INFO:teuthology.orchestra.run.trial043.stdout:(29/96): zip-3.0-35.el9.x86_64.rpm 11 MB/s | 266 kB 00:00 2026-02-20T11:11:46.934 INFO:teuthology.orchestra.run.trial109.stdout:(2/96): ceph-mgr-20.3.0-5357.g0c3148ca.el9.x86_ 2.4 MB/s | 937 kB 00:00 2026-02-20T11:11:46.977 INFO:teuthology.orchestra.run.trial109.stdout:(3/96): ceph-osd-20.3.0-5357.g0c3148ca.el9.x86_ 3.7 MB/s | 163 kB 00:00 2026-02-20T11:11:47.050 INFO:teuthology.orchestra.run.trial043.stdout:(30/96): flexiblas-3.0.4-9.el9.x86_64.rpm 246 kB/s | 30 kB 00:00 2026-02-20T11:11:47.123 INFO:teuthology.orchestra.run.trial043.stdout:(31/96): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 41 MB/s | 3.0 MB 00:00 2026-02-20T11:11:47.133 INFO:teuthology.orchestra.run.trial043.stdout:(32/96): flexiblas-openblas-openmp-3.0.4-9.el9. 1.5 MB/s | 15 kB 00:00 2026-02-20T11:11:47.144 INFO:teuthology.orchestra.run.trial043.stdout:(33/96): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 21 MB/s | 246 kB 00:00 2026-02-20T11:11:47.156 INFO:teuthology.orchestra.run.trial043.stdout:(34/96): lua-5.4.4-4.el9.x86_64.rpm 16 MB/s | 188 kB 00:00 2026-02-20T11:11:47.166 INFO:teuthology.orchestra.run.trial043.stdout:(35/96): openblas-0.3.29-1.el9.x86_64.rpm 4.1 MB/s | 42 kB 00:00 2026-02-20T11:11:47.197 INFO:teuthology.orchestra.run.trial043.stdout:(36/96): openblas-openmp-0.3.29-1.el9.x86_64.rp 175 MB/s | 5.3 MB 00:00 2026-02-20T11:11:47.242 INFO:teuthology.orchestra.run.trial043.stdout:(37/96): python3-babel-2.9.1-2.el9.noarch.rpm 132 MB/s | 6.0 MB 00:00 2026-02-20T11:11:47.256 INFO:teuthology.orchestra.run.trial043.stdout:(38/96): python3-devel-3.9.25-3.el9.x86_64.rpm 18 MB/s | 244 kB 00:00 2026-02-20T11:11:47.269 INFO:teuthology.orchestra.run.trial043.stdout:(39/96): python3-jinja2-2.11.3-8.el9.noarch.rpm 19 MB/s | 249 kB 00:00 2026-02-20T11:11:47.279 INFO:teuthology.orchestra.run.trial043.stdout:(40/96): python3-jmespath-1.0.1-1.el9.noarch.rp 4.3 MB/s | 48 kB 00:00 2026-02-20T11:11:47.284 INFO:teuthology.orchestra.run.trial109.stdout:(4/96): ceph-mds-20.3.0-5357.g0c3148ca.el9.x86_ 2.8 MB/s | 2.0 MB 00:00 2026-02-20T11:11:47.291 INFO:teuthology.orchestra.run.trial043.stdout:(41/96): python3-libstoragemgmt-1.10.1-1.el9.x8 15 MB/s | 177 kB 00:00 2026-02-20T11:11:47.307 INFO:teuthology.orchestra.run.trial043.stdout:(42/96): python3-lxml-4.6.5-3.el9.x86_64.rpm 78 MB/s | 1.2 MB 00:00 2026-02-20T11:11:47.318 INFO:teuthology.orchestra.run.trial043.stdout:(43/96): python3-markupsafe-1.1.1-12.el9.x86_64 3.1 MB/s | 35 kB 00:00 2026-02-20T11:11:47.358 INFO:teuthology.orchestra.run.trial043.stdout:(44/96): python3-numpy-1.23.5-2.el9.x86_64.rpm 153 MB/s | 6.1 MB 00:00 2026-02-20T11:11:47.372 INFO:teuthology.orchestra.run.trial043.stdout:(45/96): python3-numpy-f2py-1.23.5-2.el9.x86_64 31 MB/s | 442 kB 00:00 2026-02-20T11:11:47.384 INFO:teuthology.orchestra.run.trial043.stdout:(46/96): python3-packaging-20.9-5.el9.noarch.rp 6.6 MB/s | 77 kB 00:00 2026-02-20T11:11:47.396 INFO:teuthology.orchestra.run.trial043.stdout:(47/96): python3-protobuf-3.14.0-17.el9.noarch. 22 MB/s | 267 kB 00:00 2026-02-20T11:11:47.408 INFO:teuthology.orchestra.run.trial043.stdout:(48/96): python3-pyasn1-0.4.8-6.el9.noarch.rpm 12 MB/s | 159 kB 00:00 2026-02-20T11:11:47.421 INFO:teuthology.orchestra.run.trial043.stdout:(49/96): python3-pyasn1-modules-0.4.8-6.el9.noa 21 MB/s | 279 kB 00:00 2026-02-20T11:11:47.432 INFO:teuthology.orchestra.run.trial043.stdout:(50/96): python3-requests-oauthlib-1.3.0-12.el9 4.8 MB/s | 54 kB 00:00 2026-02-20T11:11:47.536 INFO:teuthology.orchestra.run.trial043.stdout:(51/96): python3-scipy-1.9.3-2.el9.x86_64.rpm 187 MB/s | 19 MB 00:00 2026-02-20T11:11:47.547 INFO:teuthology.orchestra.run.trial043.stdout:(52/96): python3-toml-0.10.2-6.el9.noarch.rpm 3.7 MB/s | 42 kB 00:00 2026-02-20T11:11:47.558 INFO:teuthology.orchestra.run.trial043.stdout:(53/96): xmlsec1-1.2.29-13.el9.x86_64.rpm 17 MB/s | 189 kB 00:00 2026-02-20T11:11:47.569 INFO:teuthology.orchestra.run.trial043.stdout:(54/96): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 8.2 MB/s | 90 kB 00:00 2026-02-20T11:11:47.693 INFO:teuthology.orchestra.run.trial109.stdout:(5/96): ceph-mon-20.3.0-5357.g0c3148ca.el9.x86_ 4.9 MB/s | 4.9 MB 00:00 2026-02-20T11:11:47.737 INFO:teuthology.orchestra.run.trial109.stdout:(6/96): ceph-grafana-dashboards-20.3.0-5357.g0c 1.3 MB/s | 57 kB 00:00 2026-02-20T11:11:47.782 INFO:teuthology.orchestra.run.trial109.stdout:(7/96): ceph-mgr-cephadm-20.3.0-5357.g0c3148ca. 3.9 MB/s | 184 kB 00:00 2026-02-20T11:11:48.197 INFO:teuthology.orchestra.run.trial043.stdout:(55/96): lua-devel-5.4.4-4.el9.x86_64.rpm 35 kB/s | 22 kB 00:00 2026-02-20T11:11:48.247 INFO:teuthology.orchestra.run.trial109.stdout:(8/96): ceph-osd-classic-20.3.0-5357.g0c3148ca. 12 MB/s | 15 MB 00:01 2026-02-20T11:11:48.605 INFO:teuthology.orchestra.run.trial043.stdout:(56/96): ceph-mgr-dashboard-20.3.0-5357.g0c3148 17 MB/s | 51 MB 00:03 2026-02-20T11:11:48.694 INFO:teuthology.orchestra.run.trial109.stdout:(9/96): ceph-mgr-diskprediction-local-20.3.0-53 17 MB/s | 7.4 MB 00:00 2026-02-20T11:11:48.739 INFO:teuthology.orchestra.run.trial109.stdout:(10/96): ceph-mgr-k8sevents-20.3.0-5357.g0c3148 497 kB/s | 22 kB 00:00 2026-02-20T11:11:48.782 INFO:teuthology.orchestra.run.trial043.stdout:(57/96): abseil-cpp-20211102.0-4.el9.x86_64.rpm 3.0 MB/s | 551 kB 00:00 2026-02-20T11:11:48.791 INFO:teuthology.orchestra.run.trial109.stdout:(11/96): ceph-mgr-modules-core-20.3.0-5357.g0c3 5.5 MB/s | 297 kB 00:00 2026-02-20T11:11:48.807 INFO:teuthology.orchestra.run.trial043.stdout:(58/96): grpc-data-1.46.7-10.el9.noarch.rpm 803 kB/s | 19 kB 00:00 2026-02-20T11:11:48.834 INFO:teuthology.orchestra.run.trial043.stdout:(59/96): luarocks-3.9.2-5.el9.noarch.rpm 5.4 MB/s | 151 kB 00:00 2026-02-20T11:11:48.839 INFO:teuthology.orchestra.run.trial109.stdout:(12/96): ceph-mgr-rook-20.3.0-5357.g0c3148ca.el 1.0 MB/s | 50 kB 00:00 2026-02-20T11:11:48.875 INFO:teuthology.orchestra.run.trial043.stdout:(60/96): python3-asyncssh-2.13.2-5.el9.noarch.r 13 MB/s | 548 kB 00:00 2026-02-20T11:11:48.888 INFO:teuthology.orchestra.run.trial109.stdout:(13/96): ceph-prometheus-alerts-20.3.0-5357.g0c 357 kB/s | 17 kB 00:00 2026-02-20T11:11:48.900 INFO:teuthology.orchestra.run.trial043.stdout:(61/96): python3-autocommand-2.2.2-8.el9.noarch 1.2 MB/s | 29 kB 00:00 2026-02-20T11:11:48.925 INFO:teuthology.orchestra.run.trial043.stdout:(62/96): python3-backports-tarfile-1.2.0-1.el9. 2.4 MB/s | 60 kB 00:00 2026-02-20T11:11:48.944 INFO:teuthology.orchestra.run.trial109.stdout:(14/96): ceph-volume-20.3.0-5357.g0c3148ca.el9. 5.2 MB/s | 301 kB 00:00 2026-02-20T11:11:48.950 INFO:teuthology.orchestra.run.trial043.stdout:(63/96): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.7 MB/s | 43 kB 00:00 2026-02-20T11:11:48.975 INFO:teuthology.orchestra.run.trial043.stdout:(64/96): python3-cachetools-4.2.4-1.el9.noarch. 1.3 MB/s | 32 kB 00:00 2026-02-20T11:11:48.999 INFO:teuthology.orchestra.run.trial043.stdout:(65/96): python3-certifi-2023.05.07-4.el9.noarc 575 kB/s | 14 kB 00:00 2026-02-20T11:11:49.017 INFO:teuthology.orchestra.run.trial109.stdout:(15/96): cephadm-20.3.0-5357.g0c3148ca.el9.noar 14 MB/s | 1.0 MB 00:00 2026-02-20T11:11:49.027 INFO:teuthology.orchestra.run.trial043.stdout:(66/96): python3-cheroot-10.0.1-4.el9.noarch.rp 6.1 MB/s | 173 kB 00:00 2026-02-20T11:11:49.060 INFO:teuthology.orchestra.run.trial043.stdout:(67/96): python3-cherrypy-18.6.1-2.el9.noarch.r 11 MB/s | 358 kB 00:00 2026-02-20T11:11:49.090 INFO:teuthology.orchestra.run.trial043.stdout:(68/96): python3-google-auth-2.45.0-1.el9.noarc 8.4 MB/s | 254 kB 00:00 2026-02-20T11:11:49.124 INFO:teuthology.orchestra.run.trial043.stdout:(69/96): protobuf-compiler-3.14.0-17.el9.x86_64 929 kB/s | 862 kB 00:00 2026-02-20T11:11:49.164 INFO:teuthology.orchestra.run.trial043.stdout:(70/96): python3-grpcio-1.46.7-10.el9.x86_64.rp 28 MB/s | 2.0 MB 00:00 2026-02-20T11:11:49.190 INFO:teuthology.orchestra.run.trial043.stdout:(71/96): python3-influxdb-5.3.1-1.el9.noarch.rp 5.2 MB/s | 139 kB 00:00 2026-02-20T11:11:49.215 INFO:teuthology.orchestra.run.trial043.stdout:(72/96): python3-isodate-0.6.1-3.el9.noarch.rpm 2.2 MB/s | 56 kB 00:00 2026-02-20T11:11:49.240 INFO:teuthology.orchestra.run.trial043.stdout:(73/96): python3-jaraco-8.2.1-3.el9.noarch.rpm 441 kB/s | 11 kB 00:00 2026-02-20T11:11:49.254 INFO:teuthology.orchestra.run.trial043.stdout:(74/96): python3-grpcio-tools-1.46.7-10.el9.x86 1.1 MB/s | 144 kB 00:00 2026-02-20T11:11:49.264 INFO:teuthology.orchestra.run.trial043.stdout:(75/96): python3-jaraco-classes-3.2.1-5.el9.noa 725 kB/s | 18 kB 00:00 2026-02-20T11:11:49.280 INFO:teuthology.orchestra.run.trial043.stdout:(76/96): python3-jaraco-collections-3.0.0-8.el9 913 kB/s | 23 kB 00:00 2026-02-20T11:11:49.289 INFO:teuthology.orchestra.run.trial043.stdout:(77/96): python3-jaraco-context-6.0.1-3.el9.noa 800 kB/s | 20 kB 00:00 2026-02-20T11:11:49.306 INFO:teuthology.orchestra.run.trial043.stdout:(78/96): python3-jaraco-functools-3.5.0-2.el9.n 765 kB/s | 19 kB 00:00 2026-02-20T11:11:49.314 INFO:teuthology.orchestra.run.trial043.stdout:(79/96): python3-jaraco-text-4.0.0-2.el9.noarch 1.0 MB/s | 26 kB 00:00 2026-02-20T11:11:49.339 INFO:teuthology.orchestra.run.trial043.stdout:(80/96): python3-more-itertools-8.12.0-2.el9.no 3.0 MB/s | 79 kB 00:00 2026-02-20T11:11:49.364 INFO:teuthology.orchestra.run.trial043.stdout:(81/96): python3-msgpack-1.0.3-2.el9.x86_64.rpm 3.3 MB/s | 86 kB 00:00 2026-02-20T11:11:49.390 INFO:teuthology.orchestra.run.trial043.stdout:(82/96): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2026-02-20T11:11:49.401 INFO:teuthology.orchestra.run.trial043.stdout:(83/96): python3-kubernetes-26.1.0-3.el9.noarch 11 MB/s | 1.0 MB 00:00 2026-02-20T11:11:49.414 INFO:teuthology.orchestra.run.trial043.stdout:(84/96): python3-portend-3.1.0-2.el9.noarch.rpm 674 kB/s | 16 kB 00:00 2026-02-20T11:11:49.426 INFO:teuthology.orchestra.run.trial043.stdout:(85/96): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.5 MB/s | 90 kB 00:00 2026-02-20T11:11:49.438 INFO:teuthology.orchestra.run.trial043.stdout:(86/96): python3-repoze-lru-0.7-16.el9.noarch.r 1.2 MB/s | 31 kB 00:00 2026-02-20T11:11:49.452 INFO:teuthology.orchestra.run.trial043.stdout:(87/96): python3-routes-2.5.1-5.el9.noarch.rpm 7.1 MB/s | 188 kB 00:00 2026-02-20T11:11:49.464 INFO:teuthology.orchestra.run.trial043.stdout:(88/96): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2026-02-20T11:11:49.478 INFO:teuthology.orchestra.run.trial043.stdout:(89/96): python3-saml-1.16.0-1.el9.noarch.rpm 4.7 MB/s | 125 kB 00:00 2026-02-20T11:11:49.489 INFO:teuthology.orchestra.run.trial043.stdout:(90/96): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2026-02-20T11:11:49.503 INFO:teuthology.orchestra.run.trial043.stdout:(91/96): python3-typing-extensions-4.15.0-1.el9 3.4 MB/s | 86 kB 00:00 2026-02-20T11:11:49.514 INFO:teuthology.orchestra.run.trial043.stdout:(92/96): python3-websocket-client-1.2.3-2.el9.n 3.4 MB/s | 90 kB 00:00 2026-02-20T11:11:49.526 INFO:teuthology.orchestra.run.trial109.stdout:(16/96): cryptsetup-2.8.1-3.el9.x86_64.rpm 688 kB/s | 351 kB 00:00 2026-02-20T11:11:49.528 INFO:teuthology.orchestra.run.trial043.stdout:(93/96): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 1.9 MB/s | 48 kB 00:00 2026-02-20T11:11:49.539 INFO:teuthology.orchestra.run.trial043.stdout:(94/96): python3-xmltodict-0.12.0-15.el9.noarch 897 kB/s | 22 kB 00:00 2026-02-20T11:11:49.553 INFO:teuthology.orchestra.run.trial043.stdout:(95/96): python3-zc-lockfile-2.0-10.el9.noarch. 816 kB/s | 20 kB 00:00 2026-02-20T11:11:49.637 INFO:teuthology.orchestra.run.trial109.stdout:(17/96): ledmon-libs-1.1.0-3.el9.x86_64.rpm 368 kB/s | 40 kB 00:00 2026-02-20T11:11:49.745 INFO:teuthology.orchestra.run.trial109.stdout:(18/96): libconfig-1.7.2-9.el9.x86_64.rpm 668 kB/s | 72 kB 00:00 2026-02-20T11:11:49.975 INFO:teuthology.orchestra.run.trial109.stdout:(19/96): libgfortran-11.5.0-14.el9.x86_64.rpm 3.4 MB/s | 794 kB 00:00 2026-02-20T11:11:50.082 INFO:teuthology.orchestra.run.trial109.stdout:(20/96): libquadmath-11.5.0-14.el9.x86_64.rpm 1.7 MB/s | 184 kB 00:00 2026-02-20T11:11:50.194 INFO:teuthology.orchestra.run.trial109.stdout:(21/96): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.2 MB/s | 253 kB 00:00 2026-02-20T11:11:50.365 INFO:teuthology.orchestra.run.trial109.stdout:(22/96): python3-cryptography-36.0.1-5.el9.x86_ 7.3 MB/s | 1.2 MB 00:00 2026-02-20T11:11:50.392 INFO:teuthology.orchestra.run.trial043.stdout:(96/96): ceph-osd-crimson-20.3.0-5357.g0c3148ca 17 MB/s | 88 MB 00:05 2026-02-20T11:11:50.394 INFO:teuthology.orchestra.run.trial043.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:11:50.394 INFO:teuthology.orchestra.run.trial043.stdout:Total 38 MB/s | 226 MB 00:05 2026-02-20T11:11:50.466 INFO:teuthology.orchestra.run.trial109.stdout:(23/96): python3-ply-3.11-14.el9.noarch.rpm 1.0 MB/s | 106 kB 00:00 2026-02-20T11:11:50.567 INFO:teuthology.orchestra.run.trial109.stdout:(24/96): python3-pycparser-2.20-6.el9.noarch.rp 1.3 MB/s | 135 kB 00:00 2026-02-20T11:11:50.663 INFO:teuthology.orchestra.run.trial109.stdout:(25/96): python3-pyparsing-2.4.7-9.el9.noarch.r 1.5 MB/s | 150 kB 00:00 2026-02-20T11:11:50.767 INFO:teuthology.orchestra.run.trial109.stdout:(26/96): python3-requests-2.25.1-10.el9.noarch. 1.2 MB/s | 126 kB 00:00 2026-02-20T11:11:50.802 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction check 2026-02-20T11:11:50.840 INFO:teuthology.orchestra.run.trial043.stdout:Transaction check succeeded. 2026-02-20T11:11:50.841 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction test 2026-02-20T11:11:50.869 INFO:teuthology.orchestra.run.trial109.stdout:(27/96): python3-urllib3-1.26.5-7.el9.noarch.rp 2.1 MB/s | 218 kB 00:00 2026-02-20T11:11:50.968 INFO:teuthology.orchestra.run.trial109.stdout:(28/96): unzip-6.0-59.el9.x86_64.rpm 1.8 MB/s | 182 kB 00:00 2026-02-20T11:11:51.181 INFO:teuthology.orchestra.run.trial109.stdout:(29/96): ceph-osd-crimson-20.3.0-5357.g0c3148ca 23 MB/s | 88 MB 00:03 2026-02-20T11:11:51.182 INFO:teuthology.orchestra.run.trial109.stdout:(30/96): zip-3.0-35.el9.x86_64.rpm 1.2 MB/s | 266 kB 00:00 2026-02-20T11:11:51.316 INFO:teuthology.orchestra.run.trial109.stdout:(31/96): ceph-mgr-dashboard-20.3.0-5357.g0c3148 14 MB/s | 51 MB 00:03 2026-02-20T11:11:51.634 INFO:teuthology.orchestra.run.trial043.stdout:Transaction test succeeded. 2026-02-20T11:11:51.634 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction 2026-02-20T11:11:51.868 INFO:teuthology.orchestra.run.trial109.stdout:(32/96): flexiblas-openblas-openmp-3.0.4-9.el9. 27 kB/s | 15 kB 00:00 2026-02-20T11:11:51.875 INFO:teuthology.orchestra.run.trial109.stdout:(33/96): flexiblas-3.0.4-9.el9.x86_64.rpm 43 kB/s | 30 kB 00:00 2026-02-20T11:11:52.155 INFO:teuthology.orchestra.run.trial109.stdout:(34/96): lua-5.4.4-4.el9.x86_64.rpm 672 kB/s | 188 kB 00:00 2026-02-20T11:11:52.262 INFO:teuthology.orchestra.run.trial109.stdout:(35/96): openblas-0.3.29-1.el9.x86_64.rpm 393 kB/s | 42 kB 00:00 2026-02-20T11:11:52.318 INFO:teuthology.orchestra.run.trial109.stdout:(36/96): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 548 kB/s | 246 kB 00:00 2026-02-20T11:11:52.440 INFO:teuthology.orchestra.run.trial109.stdout:(37/96): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 2.4 MB/s | 3.0 MB 00:01 2026-02-20T11:11:52.471 INFO:teuthology.orchestra.run.trial043.stdout: Preparing : 1/1 2026-02-20T11:11:52.473 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/96 2026-02-20T11:11:52.477 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/96 2026-02-20T11:11:52.498 INFO:teuthology.orchestra.run.trial043.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 3/96 2026-02-20T11:11:52.532 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 4/96 2026-02-20T11:11:52.538 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 5/96 2026-02-20T11:11:52.540 INFO:teuthology.orchestra.run.trial043.stdout: Installing : lua-5.4.4-4.el9.x86_64 6/96 2026-02-20T11:11:52.545 INFO:teuthology.orchestra.run.trial043.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 7/96 2026-02-20T11:11:52.549 INFO:teuthology.orchestra.run.trial109.stdout:(38/96): python3-devel-3.9.25-3.el9.x86_64.rpm 2.2 MB/s | 244 kB 00:00 2026-02-20T11:11:52.568 INFO:teuthology.orchestra.run.trial043.stdout: Installing : unzip-6.0-59.el9.x86_64 8/96 2026-02-20T11:11:52.581 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 9/96 2026-02-20T11:11:52.584 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 10/96 2026-02-20T11:11:52.590 INFO:teuthology.orchestra.run.trial043.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 11/96 2026-02-20T11:11:52.591 INFO:teuthology.orchestra.run.trial043.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 12/96 2026-02-20T11:11:52.603 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 13/96 2026-02-20T11:11:52.609 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 14/96 2026-02-20T11:11:52.633 INFO:teuthology.orchestra.run.trial043.stdout: Installing : zip-3.0-35.el9.x86_64 15/96 2026-02-20T11:11:52.636 INFO:teuthology.orchestra.run.trial043.stdout: Installing : luarocks-3.9.2-5.el9.noarch 16/96 2026-02-20T11:11:52.658 INFO:teuthology.orchestra.run.trial109.stdout:(39/96): python3-jinja2-2.11.3-8.el9.noarch.rpm 2.2 MB/s | 249 kB 00:00 2026-02-20T11:11:52.686 INFO:teuthology.orchestra.run.trial043.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 17/96 2026-02-20T11:11:52.699 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 18/96 2026-02-20T11:11:52.701 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-rsa-4.9-2.el9.noarch 19/96 2026-02-20T11:11:52.706 INFO:teuthology.orchestra.run.trial043.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 20/96 2026-02-20T11:11:52.711 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 21/96 2026-02-20T11:11:52.716 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 22/96 2026-02-20T11:11:52.719 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 23/96 2026-02-20T11:11:52.734 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 24/96 2026-02-20T11:11:52.738 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 25/96 2026-02-20T11:11:52.743 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 26/96 2026-02-20T11:11:52.754 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 27/96 2026-02-20T11:11:52.763 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 28/96 2026-02-20T11:11:52.767 INFO:teuthology.orchestra.run.trial109.stdout:(40/96): python3-jmespath-1.0.1-1.el9.noarch.rp 440 kB/s | 48 kB 00:00 2026-02-20T11:11:52.770 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 29/96 2026-02-20T11:11:52.789 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 30/96 2026-02-20T11:11:52.800 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 31/96 2026-02-20T11:11:52.816 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 32/96 2026-02-20T11:11:52.820 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 33/96 2026-02-20T11:11:52.828 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 34/96 2026-02-20T11:11:52.867 INFO:teuthology.orchestra.run.trial109.stdout:(41/96): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 6.0 MB 00:00 2026-02-20T11:11:52.867 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 35/96 2026-02-20T11:11:52.874 INFO:teuthology.orchestra.run.trial109.stdout:(42/96): python3-libstoragemgmt-1.10.1-1.el9.x8 1.6 MB/s | 177 kB 00:00 2026-02-20T11:11:52.900 INFO:teuthology.orchestra.run.trial109.stdout:(43/96): openblas-openmp-0.3.29-1.el9.x86_64.rp 8.3 MB/s | 5.3 MB 00:00 2026-02-20T11:11:52.980 INFO:teuthology.orchestra.run.trial109.stdout:(44/96): python3-lxml-4.6.5-3.el9.x86_64.rpm 11 MB/s | 1.2 MB 00:00 2026-02-20T11:11:52.981 INFO:teuthology.orchestra.run.trial109.stdout:(45/96): python3-markupsafe-1.1.1-12.el9.x86_64 325 kB/s | 35 kB 00:00 2026-02-20T11:11:53.089 INFO:teuthology.orchestra.run.trial109.stdout:(46/96): python3-packaging-20.9-5.el9.noarch.rp 714 kB/s | 77 kB 00:00 2026-02-20T11:11:53.090 INFO:teuthology.orchestra.run.trial109.stdout:(47/96): python3-numpy-f2py-1.23.5-2.el9.x86_64 3.9 MB/s | 442 kB 00:00 2026-02-20T11:11:53.192 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 36/96 2026-02-20T11:11:53.197 INFO:teuthology.orchestra.run.trial109.stdout:(48/96): python3-protobuf-3.14.0-17.el9.noarch. 2.4 MB/s | 267 kB 00:00 2026-02-20T11:11:53.198 INFO:teuthology.orchestra.run.trial109.stdout:(49/96): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2026-02-20T11:11:53.202 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 37/96 2026-02-20T11:11:53.206 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 38/96 2026-02-20T11:11:53.212 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 39/96 2026-02-20T11:11:53.213 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 40/96 2026-02-20T11:11:53.241 INFO:teuthology.orchestra.run.trial043.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 41/96 2026-02-20T11:11:53.283 INFO:teuthology.orchestra.run.trial109.stdout:(50/96): python3-numpy-1.23.5-2.el9.x86_64.rpm 16 MB/s | 6.1 MB 00:00 2026-02-20T11:11:53.285 INFO:teuthology.orchestra.run.trial043.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 42/96 2026-02-20T11:11:53.295 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 43/96 2026-02-20T11:11:53.300 INFO:teuthology.orchestra.run.trial043.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 44/96 2026-02-20T11:11:53.304 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 45/96 2026-02-20T11:11:53.305 INFO:teuthology.orchestra.run.trial109.stdout:(51/96): python3-requests-oauthlib-1.3.0-12.el9 502 kB/s | 54 kB 00:00 2026-02-20T11:11:53.307 INFO:teuthology.orchestra.run.trial109.stdout:(52/96): python3-pyasn1-modules-0.4.8-6.el9.noa 2.5 MB/s | 279 kB 00:00 2026-02-20T11:11:53.309 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 46/96 2026-02-20T11:11:53.313 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 47/96 2026-02-20T11:11:53.320 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 48/96 2026-02-20T11:11:53.324 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 49/96 2026-02-20T11:11:53.351 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 50/96 2026-02-20T11:11:53.361 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 51/96 2026-02-20T11:11:53.367 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 52/96 2026-02-20T11:11:53.374 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 53/96 2026-02-20T11:11:53.409 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 54/96 2026-02-20T11:11:53.413 INFO:teuthology.orchestra.run.trial109.stdout:(53/96): python3-toml-0.10.2-6.el9.noarch.rpm 387 kB/s | 42 kB 00:00 2026-02-20T11:11:53.414 INFO:teuthology.orchestra.run.trial109.stdout:(54/96): xmlsec1-1.2.29-13.el9.x86_64.rpm 1.7 MB/s | 189 kB 00:00 2026-02-20T11:11:53.521 INFO:teuthology.orchestra.run.trial109.stdout:(55/96): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 841 kB/s | 90 kB 00:00 2026-02-20T11:11:53.565 INFO:teuthology.orchestra.run.trial109.stdout:(56/96): lua-devel-5.4.4-4.el9.x86_64.rpm 148 kB/s | 22 kB 00:00 2026-02-20T11:11:53.635 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 55/96 2026-02-20T11:11:53.659 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 56/96 2026-02-20T11:11:53.663 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 57/96 2026-02-20T11:11:53.678 INFO:teuthology.orchestra.run.trial109.stdout:(57/96): protobuf-compiler-3.14.0-17.el9.x86_64 5.3 MB/s | 862 kB 00:00 2026-02-20T11:11:53.723 INFO:teuthology.orchestra.run.trial043.stdout: Installing : openblas-0.3.29-1.el9.x86_64 58/96 2026-02-20T11:11:53.725 INFO:teuthology.orchestra.run.trial043.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 59/96 2026-02-20T11:11:53.746 INFO:teuthology.orchestra.run.trial043.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 60/96 2026-02-20T11:11:53.766 INFO:teuthology.orchestra.run.trial109.stdout:(58/96): grpc-data-1.46.7-10.el9.noarch.rpm 222 kB/s | 19 kB 00:00 2026-02-20T11:11:53.822 INFO:teuthology.orchestra.run.trial109.stdout:(59/96): abseil-cpp-20211102.0-4.el9.x86_64.rpm 2.1 MB/s | 551 kB 00:00 2026-02-20T11:11:53.853 INFO:teuthology.orchestra.run.trial109.stdout:(60/96): python3-scipy-1.9.3-2.el9.x86_64.rpm 34 MB/s | 19 MB 00:00 2026-02-20T11:11:53.854 INFO:teuthology.orchestra.run.trial109.stdout:(61/96): luarocks-3.9.2-5.el9.noarch.rpm 1.7 MB/s | 151 kB 00:00 2026-02-20T11:11:53.863 INFO:teuthology.orchestra.run.trial109.stdout:(62/96): python3-asyncssh-2.13.2-5.el9.noarch.r 13 MB/s | 548 kB 00:00 2026-02-20T11:11:53.879 INFO:teuthology.orchestra.run.trial109.stdout:(63/96): python3-backports-tarfile-1.2.0-1.el9. 2.4 MB/s | 60 kB 00:00 2026-02-20T11:11:53.883 INFO:teuthology.orchestra.run.trial109.stdout:(64/96): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 2.1 MB/s | 43 kB 00:00 2026-02-20T11:11:53.902 INFO:teuthology.orchestra.run.trial109.stdout:(65/96): python3-cachetools-4.2.4-1.el9.noarch. 1.4 MB/s | 32 kB 00:00 2026-02-20T11:11:53.904 INFO:teuthology.orchestra.run.trial109.stdout:(66/96): python3-certifi-2023.05.07-4.el9.noarc 692 kB/s | 14 kB 00:00 2026-02-20T11:11:53.929 INFO:teuthology.orchestra.run.trial109.stdout:(67/96): python3-cherrypy-18.6.1-2.el9.noarch.r 14 MB/s | 358 kB 00:00 2026-02-20T11:11:53.936 INFO:teuthology.orchestra.run.trial109.stdout:(68/96): python3-cheroot-10.0.1-4.el9.noarch.rp 4.9 MB/s | 173 kB 00:00 2026-02-20T11:11:53.947 INFO:teuthology.orchestra.run.trial109.stdout:(69/96): python3-autocommand-2.2.2-8.el9.noarch 314 kB/s | 29 kB 00:00 2026-02-20T11:11:53.953 INFO:teuthology.orchestra.run.trial109.stdout:(70/96): python3-google-auth-2.45.0-1.el9.noarc 11 MB/s | 254 kB 00:00 2026-02-20T11:11:53.979 INFO:teuthology.orchestra.run.trial109.stdout:(71/96): python3-influxdb-5.3.1-1.el9.noarch.rp 5.1 MB/s | 139 kB 00:00 2026-02-20T11:11:54.017 INFO:teuthology.orchestra.run.trial109.stdout:(72/96): python3-grpcio-tools-1.46.7-10.el9.x86 2.0 MB/s | 144 kB 00:00 2026-02-20T11:11:54.023 INFO:teuthology.orchestra.run.trial109.stdout:(73/96): python3-isodate-0.6.1-3.el9.noarch.rpm 1.3 MB/s | 56 kB 00:00 2026-02-20T11:11:54.034 INFO:teuthology.orchestra.run.trial109.stdout:(74/96): python3-grpcio-1.46.7-10.el9.x86_64.rp 21 MB/s | 2.0 MB 00:00 2026-02-20T11:11:54.038 INFO:teuthology.orchestra.run.trial109.stdout:(75/96): python3-jaraco-8.2.1-3.el9.noarch.rpm 511 kB/s | 11 kB 00:00 2026-02-20T11:11:54.043 INFO:teuthology.orchestra.run.trial109.stdout:(76/96): python3-jaraco-classes-3.2.1-5.el9.noa 859 kB/s | 18 kB 00:00 2026-02-20T11:11:54.056 INFO:teuthology.orchestra.run.trial109.stdout:(77/96): python3-jaraco-collections-3.0.0-8.el9 1.0 MB/s | 23 kB 00:00 2026-02-20T11:11:54.059 INFO:teuthology.orchestra.run.trial109.stdout:(78/96): python3-jaraco-context-6.0.1-3.el9.noa 906 kB/s | 20 kB 00:00 2026-02-20T11:11:54.067 INFO:teuthology.orchestra.run.trial109.stdout:(79/96): python3-jaraco-functools-3.5.0-2.el9.n 832 kB/s | 19 kB 00:00 2026-02-20T11:11:54.073 INFO:teuthology.orchestra.run.trial043.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 61/96 2026-02-20T11:11:54.077 INFO:teuthology.orchestra.run.trial109.stdout:(80/96): python3-jaraco-text-4.0.0-2.el9.noarch 1.2 MB/s | 26 kB 00:00 2026-02-20T11:11:54.090 INFO:teuthology.orchestra.run.trial109.stdout:(81/96): python3-more-itertools-8.12.0-2.el9.no 3.3 MB/s | 79 kB 00:00 2026-02-20T11:11:54.101 INFO:teuthology.orchestra.run.trial109.stdout:(82/96): python3-msgpack-1.0.3-2.el9.x86_64.rpm 3.6 MB/s | 86 kB 00:00 2026-02-20T11:11:54.111 INFO:teuthology.orchestra.run.trial109.stdout:(83/96): python3-natsort-7.1.1-5.el9.noarch.rpm 2.7 MB/s | 58 kB 00:00 2026-02-20T11:11:54.124 INFO:teuthology.orchestra.run.trial109.stdout:(84/96): python3-portend-3.1.0-2.el9.noarch.rpm 716 kB/s | 16 kB 00:00 2026-02-20T11:11:54.133 INFO:teuthology.orchestra.run.trial109.stdout:(85/96): python3-pyOpenSSL-21.0.0-1.el9.noarch. 4.0 MB/s | 90 kB 00:00 2026-02-20T11:11:54.140 INFO:teuthology.orchestra.run.trial109.stdout:(86/96): python3-kubernetes-26.1.0-3.el9.noarch 13 MB/s | 1.0 MB 00:00 2026-02-20T11:11:54.144 INFO:teuthology.orchestra.run.trial109.stdout:(87/96): python3-repoze-lru-0.7-16.el9.noarch.r 1.4 MB/s | 31 kB 00:00 2026-02-20T11:11:54.146 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 62/96 2026-02-20T11:11:54.155 INFO:teuthology.orchestra.run.trial109.stdout:(88/96): python3-routes-2.5.1-5.el9.noarch.rpm 8.3 MB/s | 188 kB 00:00 2026-02-20T11:11:54.161 INFO:teuthology.orchestra.run.trial109.stdout:(89/96): python3-rsa-4.9-2.el9.noarch.rpm 2.8 MB/s | 59 kB 00:00 2026-02-20T11:11:54.169 INFO:teuthology.orchestra.run.trial109.stdout:(90/96): python3-saml-1.16.0-1.el9.noarch.rpm 5.1 MB/s | 125 kB 00:00 2026-02-20T11:11:54.176 INFO:teuthology.orchestra.run.trial109.stdout:(91/96): python3-tempora-5.0.0-2.el9.noarch.rpm 1.7 MB/s | 36 kB 00:00 2026-02-20T11:11:54.182 INFO:teuthology.orchestra.run.trial109.stdout:(92/96): python3-typing-extensions-4.15.0-1.el9 3.9 MB/s | 86 kB 00:00 2026-02-20T11:11:54.190 INFO:teuthology.orchestra.run.trial109.stdout:(93/96): python3-websocket-client-1.2.3-2.el9.n 4.2 MB/s | 90 kB 00:00 2026-02-20T11:11:54.197 INFO:teuthology.orchestra.run.trial109.stdout:(94/96): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 2.3 MB/s | 48 kB 00:00 2026-02-20T11:11:54.208 INFO:teuthology.orchestra.run.trial109.stdout:(95/96): python3-xmltodict-0.12.0-15.el9.noarch 882 kB/s | 22 kB 00:00 2026-02-20T11:11:54.210 INFO:teuthology.orchestra.run.trial109.stdout:(96/96): python3-zc-lockfile-2.0-10.el9.noarch. 975 kB/s | 20 kB 00:00 2026-02-20T11:11:54.213 INFO:teuthology.orchestra.run.trial109.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:11:54.213 INFO:teuthology.orchestra.run.trial109.stdout:Total 27 MB/s | 226 MB 00:08 2026-02-20T11:11:54.635 INFO:teuthology.orchestra.run.trial109.stdout:Running transaction check 2026-02-20T11:11:54.673 INFO:teuthology.orchestra.run.trial109.stdout:Transaction check succeeded. 2026-02-20T11:11:54.673 INFO:teuthology.orchestra.run.trial109.stdout:Running transaction test 2026-02-20T11:11:54.819 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 63/96 2026-02-20T11:11:54.835 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 64/96 2026-02-20T11:11:54.845 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 65/96 2026-02-20T11:11:54.852 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-packaging-20.9-5.el9.noarch 66/96 2026-02-20T11:11:54.865 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-ply-3.11-14.el9.noarch 67/96 2026-02-20T11:11:54.881 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 68/96 2026-02-20T11:11:54.957 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 69/96 2026-02-20T11:11:54.966 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 70/96 2026-02-20T11:11:54.989 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 71/96 2026-02-20T11:11:55.019 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 72/96 2026-02-20T11:11:55.070 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 73/96 2026-02-20T11:11:55.076 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 74/96 2026-02-20T11:11:55.079 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 75/96 2026-02-20T11:11:55.080 INFO:teuthology.orchestra.run.trial043.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 76/96 2026-02-20T11:11:55.098 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-20T11:11:55.099 INFO:teuthology.orchestra.run.trial043.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-20T11:11:55.099 INFO:teuthology.orchestra.run.trial043.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-20T11:11:55.099 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:11:55.106 INFO:teuthology.orchestra.run.trial043.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-20T11:11:55.132 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-20T11:11:55.132 INFO:teuthology.orchestra.run.trial043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-20T11:11:55.132 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:11:55.165 INFO:teuthology.orchestra.run.trial043.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 78/96 2026-02-20T11:11:55.472 INFO:teuthology.orchestra.run.trial109.stdout:Transaction test succeeded. 2026-02-20T11:11:55.472 INFO:teuthology.orchestra.run.trial109.stdout:Running transaction 2026-02-20T11:11:55.754 INFO:teuthology.orchestra.run.trial043.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 79/96 2026-02-20T11:11:56.307 INFO:teuthology.orchestra.run.trial109.stdout: Preparing : 1/1 2026-02-20T11:11:56.309 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/96 2026-02-20T11:11:56.312 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/96 2026-02-20T11:11:56.334 INFO:teuthology.orchestra.run.trial109.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 3/96 2026-02-20T11:11:56.369 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 4/96 2026-02-20T11:11:56.375 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 5/96 2026-02-20T11:11:56.377 INFO:teuthology.orchestra.run.trial109.stdout: Installing : lua-5.4.4-4.el9.x86_64 6/96 2026-02-20T11:11:56.382 INFO:teuthology.orchestra.run.trial109.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 7/96 2026-02-20T11:11:56.405 INFO:teuthology.orchestra.run.trial109.stdout: Installing : unzip-6.0-59.el9.x86_64 8/96 2026-02-20T11:11:56.418 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 9/96 2026-02-20T11:11:56.421 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 10/96 2026-02-20T11:11:56.427 INFO:teuthology.orchestra.run.trial109.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 11/96 2026-02-20T11:11:56.429 INFO:teuthology.orchestra.run.trial109.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 12/96 2026-02-20T11:11:56.440 INFO:teuthology.orchestra.run.trial109.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 13/96 2026-02-20T11:11:56.446 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 14/96 2026-02-20T11:11:56.470 INFO:teuthology.orchestra.run.trial109.stdout: Installing : zip-3.0-35.el9.x86_64 15/96 2026-02-20T11:11:56.473 INFO:teuthology.orchestra.run.trial109.stdout: Installing : luarocks-3.9.2-5.el9.noarch 16/96 2026-02-20T11:11:56.524 INFO:teuthology.orchestra.run.trial109.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 17/96 2026-02-20T11:11:56.537 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 18/96 2026-02-20T11:11:56.539 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-rsa-4.9-2.el9.noarch 19/96 2026-02-20T11:11:56.543 INFO:teuthology.orchestra.run.trial109.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 20/96 2026-02-20T11:11:56.549 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 21/96 2026-02-20T11:11:56.553 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 22/96 2026-02-20T11:11:56.556 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 23/96 2026-02-20T11:11:56.571 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 24/96 2026-02-20T11:11:56.576 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 25/96 2026-02-20T11:11:56.580 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 26/96 2026-02-20T11:11:56.592 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 27/96 2026-02-20T11:11:56.601 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 28/96 2026-02-20T11:11:56.607 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 29/96 2026-02-20T11:11:56.627 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 30/96 2026-02-20T11:11:56.638 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 31/96 2026-02-20T11:11:56.654 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 32/96 2026-02-20T11:11:56.658 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 33/96 2026-02-20T11:11:56.666 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 34/96 2026-02-20T11:11:56.706 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 35/96 2026-02-20T11:11:57.033 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 36/96 2026-02-20T11:11:57.042 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 37/96 2026-02-20T11:11:57.046 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 38/96 2026-02-20T11:11:57.052 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 39/96 2026-02-20T11:11:57.054 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 40/96 2026-02-20T11:11:57.082 INFO:teuthology.orchestra.run.trial109.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 41/96 2026-02-20T11:11:57.125 INFO:teuthology.orchestra.run.trial109.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 42/96 2026-02-20T11:11:57.133 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 43/96 2026-02-20T11:11:57.139 INFO:teuthology.orchestra.run.trial109.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 44/96 2026-02-20T11:11:57.142 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 45/96 2026-02-20T11:11:57.149 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 46/96 2026-02-20T11:11:57.152 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 47/96 2026-02-20T11:11:57.159 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 48/96 2026-02-20T11:11:57.162 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 49/96 2026-02-20T11:11:57.190 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 50/96 2026-02-20T11:11:57.200 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 51/96 2026-02-20T11:11:57.206 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 52/96 2026-02-20T11:11:57.213 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 53/96 2026-02-20T11:11:57.247 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 54/96 2026-02-20T11:11:57.472 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 55/96 2026-02-20T11:11:57.495 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 56/96 2026-02-20T11:11:57.499 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 57/96 2026-02-20T11:11:57.557 INFO:teuthology.orchestra.run.trial109.stdout: Installing : openblas-0.3.29-1.el9.x86_64 58/96 2026-02-20T11:11:57.559 INFO:teuthology.orchestra.run.trial109.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 59/96 2026-02-20T11:11:57.580 INFO:teuthology.orchestra.run.trial109.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 60/96 2026-02-20T11:11:57.907 INFO:teuthology.orchestra.run.trial109.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 61/96 2026-02-20T11:11:57.980 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 62/96 2026-02-20T11:11:58.654 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 63/96 2026-02-20T11:11:58.670 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 64/96 2026-02-20T11:11:58.680 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 65/96 2026-02-20T11:11:58.687 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-packaging-20.9-5.el9.noarch 66/96 2026-02-20T11:11:58.700 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-ply-3.11-14.el9.noarch 67/96 2026-02-20T11:11:58.716 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 68/96 2026-02-20T11:11:58.792 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 69/96 2026-02-20T11:11:58.801 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 70/96 2026-02-20T11:11:58.825 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 71/96 2026-02-20T11:11:58.855 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 72/96 2026-02-20T11:11:58.905 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 73/96 2026-02-20T11:11:58.911 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 74/96 2026-02-20T11:11:58.915 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 75/96 2026-02-20T11:11:58.916 INFO:teuthology.orchestra.run.trial109.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 76/96 2026-02-20T11:11:58.933 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-20T11:11:58.934 INFO:teuthology.orchestra.run.trial109.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-20T11:11:58.934 INFO:teuthology.orchestra.run.trial109.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-20T11:11:58.934 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:11:58.941 INFO:teuthology.orchestra.run.trial109.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-20T11:11:58.966 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-20T11:11:58.967 INFO:teuthology.orchestra.run.trial109.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-20T11:11:58.967 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:11:59.002 INFO:teuthology.orchestra.run.trial109.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 78/96 2026-02-20T11:11:59.588 INFO:teuthology.orchestra.run.trial109.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 79/96 2026-02-20T11:12:01.169 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-osd-classic-2:20.3.0-5357.g0c3148ca.el9.x86_6 80/96 2026-02-20T11:12:01.233 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-osd-crimson-2:20.3.0-5357.g0c3148ca.el9.x86_6 81/96 2026-02-20T11:12:01.235 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-volume-2:20.3.0-5357.g0c3148ca.el9.noarch 82/96 2026-02-20T11:12:01.249 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-volume-2:20.3.0-5357.g0c3148ca.el9.noarch 82/96 2026-02-20T11:12:01.249 INFO:teuthology.orchestra.run.trial043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T11:12:01.249 INFO:teuthology.orchestra.run.trial043.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-20T11:12:01.249 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:01.259 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-osd-2:20.3.0-5357.g0c3148ca.el9.x86_64 83/96 2026-02-20T11:12:01.284 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-osd-2:20.3.0-5357.g0c3148ca.el9.x86_64 83/96 2026-02-20T11:12:01.284 INFO:teuthology.orchestra.run.trial043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T11:12:01.284 INFO:teuthology.orchestra.run.trial043.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-20T11:12:01.284 INFO:teuthology.orchestra.run.trial043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-20T11:12:01.284 INFO:teuthology.orchestra.run.trial043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-20T11:12:01.284 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:01.343 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: cephadm-2:20.3.0-5357.g0c3148ca.el9.noarch 84/96 2026-02-20T11:12:01.345 INFO:teuthology.orchestra.run.trial043.stdout: Installing : cephadm-2:20.3.0-5357.g0c3148ca.el9.noarch 84/96 2026-02-20T11:12:01.354 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5357.g0c3148ca.el9 85/96 2026-02-20T11:12:01.378 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5357.g0c3148ca.el 86/96 2026-02-20T11:12:01.380 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5357.g0c3148ca.el9.noarc 87/96 2026-02-20T11:12:04.279 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5357.g0c3148ca.el9.noarc 87/96 2026-02-20T11:12:04.286 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5357.g0c3148ca.el9.noa 88/96 2026-02-20T11:12:04.740 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5357.g0c3148ca.el9.noa 88/96 2026-02-20T11:12:04.741 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5357.g0c314 89/96 2026-02-20T11:12:04.753 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5357.g0c314 89/96 2026-02-20T11:12:04.753 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-mgr-k8sevents-2:20.3.0-5357.g0c3148ca.el9.noa 90/96 2026-02-20T11:12:04.814 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.3.0-5357.g0c3148ca.el9.noa 90/96 2026-02-20T11:12:04.854 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5357.g0c3148ca.el9. 91/96 2026-02-20T11:12:04.855 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-mgr-2:20.3.0-5357.g0c3148ca.el9.x86_64 92/96 2026-02-20T11:12:04.877 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5357.g0c3148ca.el9.x86_64 92/96 2026-02-20T11:12:04.877 INFO:teuthology.orchestra.run.trial043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T11:12:04.877 INFO:teuthology.orchestra.run.trial043.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-20T11:12:04.877 INFO:teuthology.orchestra.run.trial043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-20T11:12:04.877 INFO:teuthology.orchestra.run.trial043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-20T11:12:04.877 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:04.889 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-mgr-rook-2:20.3.0-5357.g0c3148ca.el9.noarch 93/96 2026-02-20T11:12:04.997 INFO:teuthology.orchestra.run.trial109.stdout: Installing : ceph-osd-classic-2:20.3.0-5357.g0c3148ca.el9.x86_6 80/96 2026-02-20T11:12:05.061 INFO:teuthology.orchestra.run.trial109.stdout: Installing : ceph-osd-crimson-2:20.3.0-5357.g0c3148ca.el9.x86_6 81/96 2026-02-20T11:12:05.063 INFO:teuthology.orchestra.run.trial109.stdout: Installing : ceph-volume-2:20.3.0-5357.g0c3148ca.el9.noarch 82/96 2026-02-20T11:12:05.077 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: ceph-volume-2:20.3.0-5357.g0c3148ca.el9.noarch 82/96 2026-02-20T11:12:05.078 INFO:teuthology.orchestra.run.trial109.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T11:12:05.078 INFO:teuthology.orchestra.run.trial109.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-20T11:12:05.078 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:05.088 INFO:teuthology.orchestra.run.trial109.stdout: Installing : ceph-osd-2:20.3.0-5357.g0c3148ca.el9.x86_64 83/96 2026-02-20T11:12:05.105 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5357.g0c3148ca.el9.noarch 93/96 2026-02-20T11:12:05.107 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-mon-2:20.3.0-5357.g0c3148ca.el9.x86_64 94/96 2026-02-20T11:12:05.112 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: ceph-osd-2:20.3.0-5357.g0c3148ca.el9.x86_64 83/96 2026-02-20T11:12:05.112 INFO:teuthology.orchestra.run.trial109.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T11:12:05.112 INFO:teuthology.orchestra.run.trial109.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-20T11:12:05.112 INFO:teuthology.orchestra.run.trial109.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-20T11:12:05.112 INFO:teuthology.orchestra.run.trial109.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-20T11:12:05.112 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:05.129 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-mon-2:20.3.0-5357.g0c3148ca.el9.x86_64 94/96 2026-02-20T11:12:05.129 INFO:teuthology.orchestra.run.trial043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T11:12:05.129 INFO:teuthology.orchestra.run.trial043.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-20T11:12:05.130 INFO:teuthology.orchestra.run.trial043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-20T11:12:05.130 INFO:teuthology.orchestra.run.trial043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-20T11:12:05.130 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:05.170 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: cephadm-2:20.3.0-5357.g0c3148ca.el9.noarch 84/96 2026-02-20T11:12:05.172 INFO:teuthology.orchestra.run.trial109.stdout: Installing : cephadm-2:20.3.0-5357.g0c3148ca.el9.noarch 84/96 2026-02-20T11:12:05.181 INFO:teuthology.orchestra.run.trial109.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5357.g0c3148ca.el9 85/96 2026-02-20T11:12:05.205 INFO:teuthology.orchestra.run.trial109.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5357.g0c3148ca.el 86/96 2026-02-20T11:12:05.207 INFO:teuthology.orchestra.run.trial109.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5357.g0c3148ca.el9.noarc 87/96 2026-02-20T11:12:05.214 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-mds-2:20.3.0-5357.g0c3148ca.el9.x86_64 95/96 2026-02-20T11:12:05.235 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-mds-2:20.3.0-5357.g0c3148ca.el9.x86_64 95/96 2026-02-20T11:12:05.235 INFO:teuthology.orchestra.run.trial043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T11:12:05.235 INFO:teuthology.orchestra.run.trial043.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-20T11:12:05.235 INFO:teuthology.orchestra.run.trial043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-20T11:12:05.235 INFO:teuthology.orchestra.run.trial043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-20T11:12:05.235 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:05.236 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-2:20.3.0-5357.g0c3148ca.el9.x86_64 96/96 2026-02-20T11:12:05.242 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5357.g0c3148ca.el9.x86_6 96/96 2026-02-20T11:12:05.267 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5357.g0c3148ca.el9.x86_6 96/96 2026-02-20T11:12:06.666 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-2:20.3.0-5357.g0c3148ca.el9.x86_64 96/96 2026-02-20T11:12:06.666 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/96 2026-02-20T11:12:06.666 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-mds-2:20.3.0-5357.g0c3148ca.el9.x86_64 2/96 2026-02-20T11:12:06.667 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-mgr-2:20.3.0-5357.g0c3148ca.el9.x86_64 3/96 2026-02-20T11:12:06.667 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-mon-2:20.3.0-5357.g0c3148ca.el9.x86_64 4/96 2026-02-20T11:12:06.667 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-osd-2:20.3.0-5357.g0c3148ca.el9.x86_64 5/96 2026-02-20T11:12:06.667 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-osd-classic-2:20.3.0-5357.g0c3148ca.el9.x86_6 6/96 2026-02-20T11:12:06.667 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5357.g0c3148ca.el9.x86_6 7/96 2026-02-20T11:12:06.667 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5357.g0c3148ca.el 8/96 2026-02-20T11:12:06.667 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5357.g0c3148ca.el9.noarc 9/96 2026-02-20T11:12:06.667 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5357.g0c3148ca.el9.noa 10/96 2026-02-20T11:12:06.667 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5357.g0c314 11/96 2026-02-20T11:12:06.667 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-mgr-k8sevents-2:20.3.0-5357.g0c3148ca.el9.noa 12/96 2026-02-20T11:12:06.667 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5357.g0c3148ca.el9. 13/96 2026-02-20T11:12:06.667 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5357.g0c3148ca.el9.noarch 14/96 2026-02-20T11:12:06.668 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5357.g0c3148ca.el9 15/96 2026-02-20T11:12:06.668 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-volume-2:20.3.0-5357.g0c3148ca.el9.noarch 16/96 2026-02-20T11:12:06.668 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : cephadm-2:20.3.0-5357.g0c3148ca.el9.noarch 17/96 2026-02-20T11:12:06.668 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 18/96 2026-02-20T11:12:06.668 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 19/96 2026-02-20T11:12:06.668 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 20/96 2026-02-20T11:12:06.668 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 21/96 2026-02-20T11:12:06.668 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 22/96 2026-02-20T11:12:06.668 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 23/96 2026-02-20T11:12:06.668 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 24/96 2026-02-20T11:12:06.668 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-ply-3.11-14.el9.noarch 25/96 2026-02-20T11:12:06.669 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 26/96 2026-02-20T11:12:06.669 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 27/96 2026-02-20T11:12:06.669 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 28/96 2026-02-20T11:12:06.669 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 29/96 2026-02-20T11:12:06.669 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : unzip-6.0-59.el9.x86_64 30/96 2026-02-20T11:12:06.669 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : zip-3.0-35.el9.x86_64 31/96 2026-02-20T11:12:06.669 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 32/96 2026-02-20T11:12:06.669 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 33/96 2026-02-20T11:12:06.669 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 34/96 2026-02-20T11:12:06.669 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 35/96 2026-02-20T11:12:06.669 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : lua-5.4.4-4.el9.x86_64 36/96 2026-02-20T11:12:06.669 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 37/96 2026-02-20T11:12:06.669 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 38/96 2026-02-20T11:12:06.669 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 39/96 2026-02-20T11:12:06.670 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 40/96 2026-02-20T11:12:06.670 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 41/96 2026-02-20T11:12:06.670 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 42/96 2026-02-20T11:12:06.670 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 43/96 2026-02-20T11:12:06.670 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 44/96 2026-02-20T11:12:06.670 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 45/96 2026-02-20T11:12:06.670 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 46/96 2026-02-20T11:12:06.670 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 47/96 2026-02-20T11:12:06.670 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 48/96 2026-02-20T11:12:06.670 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 49/96 2026-02-20T11:12:06.670 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 50/96 2026-02-20T11:12:06.670 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 51/96 2026-02-20T11:12:06.671 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 52/96 2026-02-20T11:12:06.671 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 53/96 2026-02-20T11:12:06.671 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 54/96 2026-02-20T11:12:06.671 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 55/96 2026-02-20T11:12:06.671 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 56/96 2026-02-20T11:12:06.671 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 57/96 2026-02-20T11:12:06.671 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 58/96 2026-02-20T11:12:06.671 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 59/96 2026-02-20T11:12:06.671 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 60/96 2026-02-20T11:12:06.671 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 61/96 2026-02-20T11:12:06.671 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 62/96 2026-02-20T11:12:06.671 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 63/96 2026-02-20T11:12:06.671 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 64/96 2026-02-20T11:12:06.672 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 65/96 2026-02-20T11:12:06.672 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 66/96 2026-02-20T11:12:06.672 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 67/96 2026-02-20T11:12:06.672 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 68/96 2026-02-20T11:12:06.672 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 69/96 2026-02-20T11:12:06.672 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 70/96 2026-02-20T11:12:06.672 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 71/96 2026-02-20T11:12:06.672 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 72/96 2026-02-20T11:12:06.672 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 73/96 2026-02-20T11:12:06.672 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 74/96 2026-02-20T11:12:06.672 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 75/96 2026-02-20T11:12:06.672 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 76/96 2026-02-20T11:12:06.672 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 77/96 2026-02-20T11:12:06.673 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 78/96 2026-02-20T11:12:06.673 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 79/96 2026-02-20T11:12:06.673 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 80/96 2026-02-20T11:12:06.673 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 81/96 2026-02-20T11:12:06.673 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/96 2026-02-20T11:12:06.673 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/96 2026-02-20T11:12:06.673 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/96 2026-02-20T11:12:06.673 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 85/96 2026-02-20T11:12:06.673 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/96 2026-02-20T11:12:06.673 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 87/96 2026-02-20T11:12:06.673 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 88/96 2026-02-20T11:12:06.673 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 89/96 2026-02-20T11:12:06.673 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 90/96 2026-02-20T11:12:06.673 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 91/96 2026-02-20T11:12:06.674 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 92/96 2026-02-20T11:12:06.674 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 93/96 2026-02-20T11:12:06.674 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 94/96 2026-02-20T11:12:06.674 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 95/96 2026-02-20T11:12:06.730 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 96/96 2026-02-20T11:12:06.731 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:06.731 INFO:teuthology.orchestra.run.trial043.stdout:Installed: 2026-02-20T11:12:06.731 INFO:teuthology.orchestra.run.trial043.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-20T11:12:06.731 INFO:teuthology.orchestra.run.trial043.stdout: ceph-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:12:06.731 INFO:teuthology.orchestra.run.trial043.stdout: ceph-grafana-dashboards-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:12:06.731 INFO:teuthology.orchestra.run.trial043.stdout: ceph-mds-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:12:06.731 INFO:teuthology.orchestra.run.trial043.stdout: ceph-mgr-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:12:06.731 INFO:teuthology.orchestra.run.trial043.stdout: ceph-mgr-cephadm-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:12:06.731 INFO:teuthology.orchestra.run.trial043.stdout: ceph-mgr-dashboard-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:12:06.732 INFO:teuthology.orchestra.run.trial043.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:12:06.732 INFO:teuthology.orchestra.run.trial043.stdout: ceph-mgr-k8sevents-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:12:06.732 INFO:teuthology.orchestra.run.trial043.stdout: ceph-mgr-modules-core-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:12:06.732 INFO:teuthology.orchestra.run.trial043.stdout: ceph-mgr-rook-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:12:06.732 INFO:teuthology.orchestra.run.trial043.stdout: ceph-mon-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:12:06.732 INFO:teuthology.orchestra.run.trial043.stdout: ceph-osd-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:12:06.732 INFO:teuthology.orchestra.run.trial043.stdout: ceph-osd-classic-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:12:06.732 INFO:teuthology.orchestra.run.trial043.stdout: ceph-osd-crimson-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:12:06.732 INFO:teuthology.orchestra.run.trial043.stdout: ceph-prometheus-alerts-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:12:06.732 INFO:teuthology.orchestra.run.trial043.stdout: ceph-volume-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:12:06.732 INFO:teuthology.orchestra.run.trial043.stdout: cephadm-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:12:06.732 INFO:teuthology.orchestra.run.trial043.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-20T11:12:06.733 INFO:teuthology.orchestra.run.trial043.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-20T11:12:06.733 INFO:teuthology.orchestra.run.trial043.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-20T11:12:06.733 INFO:teuthology.orchestra.run.trial043.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-20T11:12:06.733 INFO:teuthology.orchestra.run.trial043.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-20T11:12:06.733 INFO:teuthology.orchestra.run.trial043.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-20T11:12:06.733 INFO:teuthology.orchestra.run.trial043.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-20T11:12:06.733 INFO:teuthology.orchestra.run.trial043.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-20T11:12:06.733 INFO:teuthology.orchestra.run.trial043.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-20T11:12:06.733 INFO:teuthology.orchestra.run.trial043.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-20T11:12:06.733 INFO:teuthology.orchestra.run.trial043.stdout: lua-5.4.4-4.el9.x86_64 2026-02-20T11:12:06.733 INFO:teuthology.orchestra.run.trial043.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-20T11:12:06.733 INFO:teuthology.orchestra.run.trial043.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-20T11:12:06.734 INFO:teuthology.orchestra.run.trial043.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-20T11:12:06.734 INFO:teuthology.orchestra.run.trial043.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-20T11:12:06.734 INFO:teuthology.orchestra.run.trial043.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-20T11:12:06.734 INFO:teuthology.orchestra.run.trial043.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-20T11:12:06.734 INFO:teuthology.orchestra.run.trial043.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-20T11:12:06.734 INFO:teuthology.orchestra.run.trial043.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-20T11:12:06.734 INFO:teuthology.orchestra.run.trial043.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-20T11:12:06.734 INFO:teuthology.orchestra.run.trial043.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-20T11:12:06.734 INFO:teuthology.orchestra.run.trial043.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-20T11:12:06.735 INFO:teuthology.orchestra.run.trial043.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-20T11:12:06.735 INFO:teuthology.orchestra.run.trial043.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-20T11:12:06.735 INFO:teuthology.orchestra.run.trial043.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-20T11:12:06.735 INFO:teuthology.orchestra.run.trial043.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-20T11:12:06.735 INFO:teuthology.orchestra.run.trial043.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-20T11:12:06.735 INFO:teuthology.orchestra.run.trial043.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-20T11:12:06.735 INFO:teuthology.orchestra.run.trial043.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-20T11:12:06.735 INFO:teuthology.orchestra.run.trial043.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-20T11:12:06.735 INFO:teuthology.orchestra.run.trial043.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-20T11:12:06.735 INFO:teuthology.orchestra.run.trial043.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-20T11:12:06.735 INFO:teuthology.orchestra.run.trial043.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-20T11:12:06.735 INFO:teuthology.orchestra.run.trial043.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-20T11:12:06.735 INFO:teuthology.orchestra.run.trial043.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-20T11:12:06.736 INFO:teuthology.orchestra.run.trial043.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-20T11:12:06.736 INFO:teuthology.orchestra.run.trial043.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-20T11:12:06.736 INFO:teuthology.orchestra.run.trial043.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-20T11:12:06.736 INFO:teuthology.orchestra.run.trial043.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-20T11:12:06.736 INFO:teuthology.orchestra.run.trial043.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-20T11:12:06.736 INFO:teuthology.orchestra.run.trial043.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-20T11:12:06.736 INFO:teuthology.orchestra.run.trial043.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-20T11:12:06.736 INFO:teuthology.orchestra.run.trial043.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-20T11:12:06.736 INFO:teuthology.orchestra.run.trial043.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-20T11:12:06.736 INFO:teuthology.orchestra.run.trial043.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-20T11:12:06.736 INFO:teuthology.orchestra.run.trial043.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-20T11:12:06.736 INFO:teuthology.orchestra.run.trial043.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-20T11:12:06.736 INFO:teuthology.orchestra.run.trial043.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-20T11:12:06.737 INFO:teuthology.orchestra.run.trial043.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-20T11:12:06.737 INFO:teuthology.orchestra.run.trial043.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-20T11:12:06.737 INFO:teuthology.orchestra.run.trial043.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-20T11:12:06.737 INFO:teuthology.orchestra.run.trial043.stdout: python3-ply-3.11-14.el9.noarch 2026-02-20T11:12:06.737 INFO:teuthology.orchestra.run.trial043.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-20T11:12:06.737 INFO:teuthology.orchestra.run.trial043.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-20T11:12:06.737 INFO:teuthology.orchestra.run.trial043.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-20T11:12:06.737 INFO:teuthology.orchestra.run.trial043.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2026-02-20T11:12:06.737 INFO:teuthology.orchestra.run.trial043.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2026-02-20T11:12:06.737 INFO:teuthology.orchestra.run.trial043.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-20T11:12:06.737 INFO:teuthology.orchestra.run.trial043.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-20T11:12:06.737 INFO:teuthology.orchestra.run.trial043.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-20T11:12:06.738 INFO:teuthology.orchestra.run.trial043.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-20T11:12:06.738 INFO:teuthology.orchestra.run.trial043.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-20T11:12:06.738 INFO:teuthology.orchestra.run.trial043.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-20T11:12:06.738 INFO:teuthology.orchestra.run.trial043.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-20T11:12:06.738 INFO:teuthology.orchestra.run.trial043.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-20T11:12:06.738 INFO:teuthology.orchestra.run.trial043.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-20T11:12:06.738 INFO:teuthology.orchestra.run.trial043.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-20T11:12:06.738 INFO:teuthology.orchestra.run.trial043.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-20T11:12:06.738 INFO:teuthology.orchestra.run.trial043.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-20T11:12:06.738 INFO:teuthology.orchestra.run.trial043.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-20T11:12:06.738 INFO:teuthology.orchestra.run.trial043.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-20T11:12:06.738 INFO:teuthology.orchestra.run.trial043.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-20T11:12:06.738 INFO:teuthology.orchestra.run.trial043.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-20T11:12:06.739 INFO:teuthology.orchestra.run.trial043.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-20T11:12:06.739 INFO:teuthology.orchestra.run.trial043.stdout: unzip-6.0-59.el9.x86_64 2026-02-20T11:12:06.739 INFO:teuthology.orchestra.run.trial043.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-20T11:12:06.739 INFO:teuthology.orchestra.run.trial043.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-20T11:12:06.739 INFO:teuthology.orchestra.run.trial043.stdout: zip-3.0-35.el9.x86_64 2026-02-20T11:12:06.739 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:06.739 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:12:06.799 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-base 2026-02-20T11:12:07.107 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:01:18 ago on Fri 20 Feb 2026 11:10:49 AM UTC. 2026-02-20T11:12:07.172 INFO:teuthology.orchestra.run.trial043.stdout:Package ceph-base-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:12:07.191 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:12:07.191 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:12:07.191 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:12:07.212 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install cephadm 2026-02-20T11:12:07.516 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:01:18 ago on Fri 20 Feb 2026 11:10:49 AM UTC. 2026-02-20T11:12:07.580 INFO:teuthology.orchestra.run.trial043.stdout:Package cephadm-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:12:07.598 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:12:07.599 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:12:07.599 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:12:07.620 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-immutable-object-cache 2026-02-20T11:12:07.925 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:01:18 ago on Fri 20 Feb 2026 11:10:49 AM UTC. 2026-02-20T11:12:08.008 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:12:08.008 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-20T11:12:08.009 INFO:teuthology.orchestra.run.trial043.stdout: Package Arch Version Repo Size 2026-02-20T11:12:08.009 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-20T11:12:08.009 INFO:teuthology.orchestra.run.trial043.stdout:Installing: 2026-02-20T11:12:08.009 INFO:teuthology.orchestra.run.trial043.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 149 k 2026-02-20T11:12:08.009 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:08.009 INFO:teuthology.orchestra.run.trial043.stdout:Transaction Summary 2026-02-20T11:12:08.009 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-20T11:12:08.009 INFO:teuthology.orchestra.run.trial043.stdout:Install 1 Package 2026-02-20T11:12:08.009 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:08.009 INFO:teuthology.orchestra.run.trial043.stdout:Total download size: 149 k 2026-02-20T11:12:08.009 INFO:teuthology.orchestra.run.trial043.stdout:Installed size: 455 k 2026-02-20T11:12:08.010 INFO:teuthology.orchestra.run.trial043.stdout:Downloading Packages: 2026-02-20T11:12:08.103 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5357.g0c3148ca.el9.noarc 87/96 2026-02-20T11:12:08.110 INFO:teuthology.orchestra.run.trial109.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5357.g0c3148ca.el9.noa 88/96 2026-02-20T11:12:08.325 INFO:teuthology.orchestra.run.trial043.stdout:ceph-immutable-object-cache-20.3.0-5357.g0c3148 473 kB/s | 149 kB 00:00 2026-02-20T11:12:08.326 INFO:teuthology.orchestra.run.trial043.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:12:08.326 INFO:teuthology.orchestra.run.trial043.stdout:Total 471 kB/s | 149 kB 00:00 2026-02-20T11:12:08.326 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction check 2026-02-20T11:12:08.330 INFO:teuthology.orchestra.run.trial043.stdout:Transaction check succeeded. 2026-02-20T11:12:08.330 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction test 2026-02-20T11:12:08.362 INFO:teuthology.orchestra.run.trial043.stdout:Transaction test succeeded. 2026-02-20T11:12:08.362 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction 2026-02-20T11:12:08.417 INFO:teuthology.orchestra.run.trial043.stdout: Preparing : 1/1 2026-02-20T11:12:08.418 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5357.g0c3148ca. 1/1 2026-02-20T11:12:08.440 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5357.g0c3148ca. 1/1 2026-02-20T11:12:08.440 INFO:teuthology.orchestra.run.trial043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T11:12:08.440 INFO:teuthology.orchestra.run.trial043.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-20T11:12:08.440 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:08.563 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5357.g0c3148ca.el9.noa 88/96 2026-02-20T11:12:08.564 INFO:teuthology.orchestra.run.trial109.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5357.g0c314 89/96 2026-02-20T11:12:08.577 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5357.g0c314 89/96 2026-02-20T11:12:08.578 INFO:teuthology.orchestra.run.trial109.stdout: Installing : ceph-mgr-k8sevents-2:20.3.0-5357.g0c3148ca.el9.noa 90/96 2026-02-20T11:12:08.639 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.3.0-5357.g0c3148ca.el9.noa 90/96 2026-02-20T11:12:08.679 INFO:teuthology.orchestra.run.trial109.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5357.g0c3148ca.el9. 91/96 2026-02-20T11:12:08.681 INFO:teuthology.orchestra.run.trial109.stdout: Installing : ceph-mgr-2:20.3.0-5357.g0c3148ca.el9.x86_64 92/96 2026-02-20T11:12:08.702 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5357.g0c3148ca.el9.x86_64 92/96 2026-02-20T11:12:08.702 INFO:teuthology.orchestra.run.trial109.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T11:12:08.702 INFO:teuthology.orchestra.run.trial109.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-20T11:12:08.702 INFO:teuthology.orchestra.run.trial109.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-20T11:12:08.702 INFO:teuthology.orchestra.run.trial109.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-20T11:12:08.702 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:08.714 INFO:teuthology.orchestra.run.trial109.stdout: Installing : ceph-mgr-rook-2:20.3.0-5357.g0c3148ca.el9.noarch 93/96 2026-02-20T11:12:08.722 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5357.g0c3148ca. 1/1 2026-02-20T11:12:08.722 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:08.722 INFO:teuthology.orchestra.run.trial043.stdout:Installed: 2026-02-20T11:12:08.722 INFO:teuthology.orchestra.run.trial043.stdout: ceph-immutable-object-cache-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:12:08.722 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:08.722 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:12:08.748 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-mgr 2026-02-20T11:12:08.931 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5357.g0c3148ca.el9.noarch 93/96 2026-02-20T11:12:08.932 INFO:teuthology.orchestra.run.trial109.stdout: Installing : ceph-mon-2:20.3.0-5357.g0c3148ca.el9.x86_64 94/96 2026-02-20T11:12:08.954 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: ceph-mon-2:20.3.0-5357.g0c3148ca.el9.x86_64 94/96 2026-02-20T11:12:08.955 INFO:teuthology.orchestra.run.trial109.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T11:12:08.955 INFO:teuthology.orchestra.run.trial109.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-20T11:12:08.955 INFO:teuthology.orchestra.run.trial109.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-20T11:12:08.955 INFO:teuthology.orchestra.run.trial109.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-20T11:12:08.955 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:09.037 INFO:teuthology.orchestra.run.trial109.stdout: Installing : ceph-mds-2:20.3.0-5357.g0c3148ca.el9.x86_64 95/96 2026-02-20T11:12:09.055 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:01:20 ago on Fri 20 Feb 2026 11:10:49 AM UTC. 2026-02-20T11:12:09.057 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: ceph-mds-2:20.3.0-5357.g0c3148ca.el9.x86_64 95/96 2026-02-20T11:12:09.058 INFO:teuthology.orchestra.run.trial109.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T11:12:09.058 INFO:teuthology.orchestra.run.trial109.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-20T11:12:09.058 INFO:teuthology.orchestra.run.trial109.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-20T11:12:09.058 INFO:teuthology.orchestra.run.trial109.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-20T11:12:09.058 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:09.059 INFO:teuthology.orchestra.run.trial109.stdout: Installing : ceph-2:20.3.0-5357.g0c3148ca.el9.x86_64 96/96 2026-02-20T11:12:09.066 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5357.g0c3148ca.el9.x86_6 96/96 2026-02-20T11:12:09.092 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5357.g0c3148ca.el9.x86_6 96/96 2026-02-20T11:12:09.119 INFO:teuthology.orchestra.run.trial043.stdout:Package ceph-mgr-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:12:09.138 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:12:09.139 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:12:09.139 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:12:09.160 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-mgr-dashboard 2026-02-20T11:12:09.470 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:01:20 ago on Fri 20 Feb 2026 11:10:49 AM UTC. 2026-02-20T11:12:09.534 INFO:teuthology.orchestra.run.trial043.stdout:Package ceph-mgr-dashboard-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:12:09.553 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:12:09.554 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:12:09.554 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:12:09.575 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-20T11:12:09.884 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:01:20 ago on Fri 20 Feb 2026 11:10:49 AM UTC. 2026-02-20T11:12:09.949 INFO:teuthology.orchestra.run.trial043.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:12:09.967 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:12:09.968 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:12:09.968 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:12:09.989 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-mgr-rook 2026-02-20T11:12:10.297 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:01:21 ago on Fri 20 Feb 2026 11:10:49 AM UTC. 2026-02-20T11:12:10.362 INFO:teuthology.orchestra.run.trial043.stdout:Package ceph-mgr-rook-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:12:10.380 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:12:10.381 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:12:10.381 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:12:10.401 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-mgr-cephadm 2026-02-20T11:12:10.465 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: ceph-2:20.3.0-5357.g0c3148ca.el9.x86_64 96/96 2026-02-20T11:12:10.465 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : ceph-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/96 2026-02-20T11:12:10.465 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : ceph-mds-2:20.3.0-5357.g0c3148ca.el9.x86_64 2/96 2026-02-20T11:12:10.465 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : ceph-mgr-2:20.3.0-5357.g0c3148ca.el9.x86_64 3/96 2026-02-20T11:12:10.465 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : ceph-mon-2:20.3.0-5357.g0c3148ca.el9.x86_64 4/96 2026-02-20T11:12:10.465 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : ceph-osd-2:20.3.0-5357.g0c3148ca.el9.x86_64 5/96 2026-02-20T11:12:10.465 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : ceph-osd-classic-2:20.3.0-5357.g0c3148ca.el9.x86_6 6/96 2026-02-20T11:12:10.465 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5357.g0c3148ca.el9.x86_6 7/96 2026-02-20T11:12:10.465 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5357.g0c3148ca.el 8/96 2026-02-20T11:12:10.465 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5357.g0c3148ca.el9.noarc 9/96 2026-02-20T11:12:10.465 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5357.g0c3148ca.el9.noa 10/96 2026-02-20T11:12:10.465 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5357.g0c314 11/96 2026-02-20T11:12:10.465 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : ceph-mgr-k8sevents-2:20.3.0-5357.g0c3148ca.el9.noa 12/96 2026-02-20T11:12:10.465 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5357.g0c3148ca.el9. 13/96 2026-02-20T11:12:10.465 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5357.g0c3148ca.el9.noarch 14/96 2026-02-20T11:12:10.465 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5357.g0c3148ca.el9 15/96 2026-02-20T11:12:10.465 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : ceph-volume-2:20.3.0-5357.g0c3148ca.el9.noarch 16/96 2026-02-20T11:12:10.465 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : cephadm-2:20.3.0-5357.g0c3148ca.el9.noarch 17/96 2026-02-20T11:12:10.465 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 18/96 2026-02-20T11:12:10.465 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 19/96 2026-02-20T11:12:10.465 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 20/96 2026-02-20T11:12:10.466 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 21/96 2026-02-20T11:12:10.466 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 22/96 2026-02-20T11:12:10.466 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 23/96 2026-02-20T11:12:10.466 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 24/96 2026-02-20T11:12:10.466 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-ply-3.11-14.el9.noarch 25/96 2026-02-20T11:12:10.466 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 26/96 2026-02-20T11:12:10.466 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 27/96 2026-02-20T11:12:10.466 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 28/96 2026-02-20T11:12:10.466 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 29/96 2026-02-20T11:12:10.466 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : unzip-6.0-59.el9.x86_64 30/96 2026-02-20T11:12:10.466 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : zip-3.0-35.el9.x86_64 31/96 2026-02-20T11:12:10.466 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 32/96 2026-02-20T11:12:10.466 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 33/96 2026-02-20T11:12:10.466 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 34/96 2026-02-20T11:12:10.466 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 35/96 2026-02-20T11:12:10.466 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : lua-5.4.4-4.el9.x86_64 36/96 2026-02-20T11:12:10.466 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 37/96 2026-02-20T11:12:10.466 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 38/96 2026-02-20T11:12:10.466 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 39/96 2026-02-20T11:12:10.466 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 40/96 2026-02-20T11:12:10.466 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 41/96 2026-02-20T11:12:10.466 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 42/96 2026-02-20T11:12:10.466 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 43/96 2026-02-20T11:12:10.466 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 44/96 2026-02-20T11:12:10.466 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 45/96 2026-02-20T11:12:10.466 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 46/96 2026-02-20T11:12:10.467 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 47/96 2026-02-20T11:12:10.467 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 48/96 2026-02-20T11:12:10.467 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 49/96 2026-02-20T11:12:10.467 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 50/96 2026-02-20T11:12:10.467 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 51/96 2026-02-20T11:12:10.467 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 52/96 2026-02-20T11:12:10.467 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 53/96 2026-02-20T11:12:10.467 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 54/96 2026-02-20T11:12:10.467 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 55/96 2026-02-20T11:12:10.467 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 56/96 2026-02-20T11:12:10.467 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 57/96 2026-02-20T11:12:10.467 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 58/96 2026-02-20T11:12:10.467 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 59/96 2026-02-20T11:12:10.467 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 60/96 2026-02-20T11:12:10.467 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 61/96 2026-02-20T11:12:10.467 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 62/96 2026-02-20T11:12:10.467 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 63/96 2026-02-20T11:12:10.467 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 64/96 2026-02-20T11:12:10.467 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 65/96 2026-02-20T11:12:10.467 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 66/96 2026-02-20T11:12:10.467 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 67/96 2026-02-20T11:12:10.467 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 68/96 2026-02-20T11:12:10.467 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 69/96 2026-02-20T11:12:10.467 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 70/96 2026-02-20T11:12:10.467 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 71/96 2026-02-20T11:12:10.467 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 72/96 2026-02-20T11:12:10.468 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 73/96 2026-02-20T11:12:10.468 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 74/96 2026-02-20T11:12:10.468 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 75/96 2026-02-20T11:12:10.468 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 76/96 2026-02-20T11:12:10.468 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 77/96 2026-02-20T11:12:10.468 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 78/96 2026-02-20T11:12:10.468 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 79/96 2026-02-20T11:12:10.468 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 80/96 2026-02-20T11:12:10.468 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 81/96 2026-02-20T11:12:10.468 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/96 2026-02-20T11:12:10.468 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/96 2026-02-20T11:12:10.468 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/96 2026-02-20T11:12:10.468 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 85/96 2026-02-20T11:12:10.468 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/96 2026-02-20T11:12:10.468 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 87/96 2026-02-20T11:12:10.468 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 88/96 2026-02-20T11:12:10.468 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 89/96 2026-02-20T11:12:10.468 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 90/96 2026-02-20T11:12:10.468 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 91/96 2026-02-20T11:12:10.468 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 92/96 2026-02-20T11:12:10.468 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 93/96 2026-02-20T11:12:10.468 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 94/96 2026-02-20T11:12:10.468 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 95/96 2026-02-20T11:12:10.531 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 96/96 2026-02-20T11:12:10.531 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:10.531 INFO:teuthology.orchestra.run.trial109.stdout:Installed: 2026-02-20T11:12:10.531 INFO:teuthology.orchestra.run.trial109.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-20T11:12:10.531 INFO:teuthology.orchestra.run.trial109.stdout: ceph-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:12:10.531 INFO:teuthology.orchestra.run.trial109.stdout: ceph-grafana-dashboards-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:12:10.531 INFO:teuthology.orchestra.run.trial109.stdout: ceph-mds-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:12:10.531 INFO:teuthology.orchestra.run.trial109.stdout: ceph-mgr-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:12:10.531 INFO:teuthology.orchestra.run.trial109.stdout: ceph-mgr-cephadm-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:12:10.531 INFO:teuthology.orchestra.run.trial109.stdout: ceph-mgr-dashboard-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:12:10.531 INFO:teuthology.orchestra.run.trial109.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:12:10.531 INFO:teuthology.orchestra.run.trial109.stdout: ceph-mgr-k8sevents-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:12:10.531 INFO:teuthology.orchestra.run.trial109.stdout: ceph-mgr-modules-core-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:12:10.532 INFO:teuthology.orchestra.run.trial109.stdout: ceph-mgr-rook-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:12:10.532 INFO:teuthology.orchestra.run.trial109.stdout: ceph-mon-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:12:10.532 INFO:teuthology.orchestra.run.trial109.stdout: ceph-osd-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:12:10.532 INFO:teuthology.orchestra.run.trial109.stdout: ceph-osd-classic-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:12:10.532 INFO:teuthology.orchestra.run.trial109.stdout: ceph-osd-crimson-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:12:10.532 INFO:teuthology.orchestra.run.trial109.stdout: ceph-prometheus-alerts-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:12:10.532 INFO:teuthology.orchestra.run.trial109.stdout: ceph-volume-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:12:10.532 INFO:teuthology.orchestra.run.trial109.stdout: cephadm-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:12:10.532 INFO:teuthology.orchestra.run.trial109.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-20T11:12:10.532 INFO:teuthology.orchestra.run.trial109.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-20T11:12:10.532 INFO:teuthology.orchestra.run.trial109.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-20T11:12:10.532 INFO:teuthology.orchestra.run.trial109.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-20T11:12:10.532 INFO:teuthology.orchestra.run.trial109.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-20T11:12:10.532 INFO:teuthology.orchestra.run.trial109.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-20T11:12:10.532 INFO:teuthology.orchestra.run.trial109.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-20T11:12:10.532 INFO:teuthology.orchestra.run.trial109.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-20T11:12:10.532 INFO:teuthology.orchestra.run.trial109.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-20T11:12:10.532 INFO:teuthology.orchestra.run.trial109.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-20T11:12:10.532 INFO:teuthology.orchestra.run.trial109.stdout: lua-5.4.4-4.el9.x86_64 2026-02-20T11:12:10.532 INFO:teuthology.orchestra.run.trial109.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-20T11:12:10.532 INFO:teuthology.orchestra.run.trial109.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-20T11:12:10.532 INFO:teuthology.orchestra.run.trial109.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-20T11:12:10.532 INFO:teuthology.orchestra.run.trial109.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-20T11:12:10.532 INFO:teuthology.orchestra.run.trial109.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-20T11:12:10.532 INFO:teuthology.orchestra.run.trial109.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-20T11:12:10.532 INFO:teuthology.orchestra.run.trial109.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-20T11:12:10.532 INFO:teuthology.orchestra.run.trial109.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-20T11:12:10.532 INFO:teuthology.orchestra.run.trial109.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-20T11:12:10.532 INFO:teuthology.orchestra.run.trial109.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-20T11:12:10.532 INFO:teuthology.orchestra.run.trial109.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-20T11:12:10.533 INFO:teuthology.orchestra.run.trial109.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-20T11:12:10.533 INFO:teuthology.orchestra.run.trial109.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-20T11:12:10.533 INFO:teuthology.orchestra.run.trial109.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-20T11:12:10.533 INFO:teuthology.orchestra.run.trial109.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-20T11:12:10.533 INFO:teuthology.orchestra.run.trial109.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-20T11:12:10.533 INFO:teuthology.orchestra.run.trial109.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-20T11:12:10.533 INFO:teuthology.orchestra.run.trial109.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-20T11:12:10.533 INFO:teuthology.orchestra.run.trial109.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-20T11:12:10.533 INFO:teuthology.orchestra.run.trial109.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-20T11:12:10.533 INFO:teuthology.orchestra.run.trial109.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-20T11:12:10.533 INFO:teuthology.orchestra.run.trial109.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-20T11:12:10.533 INFO:teuthology.orchestra.run.trial109.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-20T11:12:10.533 INFO:teuthology.orchestra.run.trial109.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-20T11:12:10.533 INFO:teuthology.orchestra.run.trial109.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-20T11:12:10.533 INFO:teuthology.orchestra.run.trial109.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-20T11:12:10.533 INFO:teuthology.orchestra.run.trial109.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-20T11:12:10.533 INFO:teuthology.orchestra.run.trial109.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-20T11:12:10.533 INFO:teuthology.orchestra.run.trial109.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-20T11:12:10.533 INFO:teuthology.orchestra.run.trial109.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-20T11:12:10.533 INFO:teuthology.orchestra.run.trial109.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-20T11:12:10.533 INFO:teuthology.orchestra.run.trial109.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-20T11:12:10.533 INFO:teuthology.orchestra.run.trial109.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-20T11:12:10.533 INFO:teuthology.orchestra.run.trial109.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-20T11:12:10.533 INFO:teuthology.orchestra.run.trial109.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-20T11:12:10.533 INFO:teuthology.orchestra.run.trial109.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-20T11:12:10.533 INFO:teuthology.orchestra.run.trial109.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-20T11:12:10.533 INFO:teuthology.orchestra.run.trial109.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-20T11:12:10.533 INFO:teuthology.orchestra.run.trial109.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-20T11:12:10.533 INFO:teuthology.orchestra.run.trial109.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-20T11:12:10.533 INFO:teuthology.orchestra.run.trial109.stdout: python3-ply-3.11-14.el9.noarch 2026-02-20T11:12:10.533 INFO:teuthology.orchestra.run.trial109.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-20T11:12:10.533 INFO:teuthology.orchestra.run.trial109.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-20T11:12:10.534 INFO:teuthology.orchestra.run.trial109.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-20T11:12:10.534 INFO:teuthology.orchestra.run.trial109.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2026-02-20T11:12:10.534 INFO:teuthology.orchestra.run.trial109.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2026-02-20T11:12:10.534 INFO:teuthology.orchestra.run.trial109.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-20T11:12:10.534 INFO:teuthology.orchestra.run.trial109.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-20T11:12:10.534 INFO:teuthology.orchestra.run.trial109.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-20T11:12:10.534 INFO:teuthology.orchestra.run.trial109.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-20T11:12:10.534 INFO:teuthology.orchestra.run.trial109.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-20T11:12:10.534 INFO:teuthology.orchestra.run.trial109.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-20T11:12:10.534 INFO:teuthology.orchestra.run.trial109.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-20T11:12:10.534 INFO:teuthology.orchestra.run.trial109.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-20T11:12:10.534 INFO:teuthology.orchestra.run.trial109.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-20T11:12:10.534 INFO:teuthology.orchestra.run.trial109.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-20T11:12:10.534 INFO:teuthology.orchestra.run.trial109.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-20T11:12:10.534 INFO:teuthology.orchestra.run.trial109.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-20T11:12:10.534 INFO:teuthology.orchestra.run.trial109.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-20T11:12:10.534 INFO:teuthology.orchestra.run.trial109.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-20T11:12:10.534 INFO:teuthology.orchestra.run.trial109.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-20T11:12:10.534 INFO:teuthology.orchestra.run.trial109.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-20T11:12:10.534 INFO:teuthology.orchestra.run.trial109.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-20T11:12:10.534 INFO:teuthology.orchestra.run.trial109.stdout: unzip-6.0-59.el9.x86_64 2026-02-20T11:12:10.534 INFO:teuthology.orchestra.run.trial109.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-20T11:12:10.534 INFO:teuthology.orchestra.run.trial109.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-20T11:12:10.534 INFO:teuthology.orchestra.run.trial109.stdout: zip-3.0-35.el9.x86_64 2026-02-20T11:12:10.534 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:10.534 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:12:10.601 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install ceph-base 2026-02-20T11:12:10.709 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:01:21 ago on Fri 20 Feb 2026 11:10:49 AM UTC. 2026-02-20T11:12:10.773 INFO:teuthology.orchestra.run.trial043.stdout:Package ceph-mgr-cephadm-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:12:10.791 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:12:10.791 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:12:10.791 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:12:10.814 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-osd 2026-02-20T11:12:10.912 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:01:19 ago on Fri 20 Feb 2026 11:10:51 AM UTC. 2026-02-20T11:12:10.977 INFO:teuthology.orchestra.run.trial109.stdout:Package ceph-base-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:12:10.995 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:12:10.996 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:12:10.996 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:12:11.018 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install cephadm 2026-02-20T11:12:11.126 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:01:22 ago on Fri 20 Feb 2026 11:10:49 AM UTC. 2026-02-20T11:12:11.190 INFO:teuthology.orchestra.run.trial043.stdout:Package ceph-osd-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:12:11.209 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:12:11.209 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:12:11.210 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:12:11.231 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-osd-classic 2026-02-20T11:12:11.325 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:01:20 ago on Fri 20 Feb 2026 11:10:51 AM UTC. 2026-02-20T11:12:11.390 INFO:teuthology.orchestra.run.trial109.stdout:Package cephadm-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:12:11.408 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:12:11.409 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:12:11.409 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:12:11.429 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install ceph-immutable-object-cache 2026-02-20T11:12:11.536 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:01:22 ago on Fri 20 Feb 2026 11:10:49 AM UTC. 2026-02-20T11:12:11.600 INFO:teuthology.orchestra.run.trial043.stdout:Package ceph-osd-classic-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:12:11.618 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:12:11.619 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:12:11.619 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:12:11.640 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-fuse 2026-02-20T11:12:11.730 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:01:20 ago on Fri 20 Feb 2026 11:10:51 AM UTC. 2026-02-20T11:12:11.814 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:12:11.814 INFO:teuthology.orchestra.run.trial109.stdout:================================================================================ 2026-02-20T11:12:11.814 INFO:teuthology.orchestra.run.trial109.stdout: Package Arch Version Repo Size 2026-02-20T11:12:11.814 INFO:teuthology.orchestra.run.trial109.stdout:================================================================================ 2026-02-20T11:12:11.814 INFO:teuthology.orchestra.run.trial109.stdout:Installing: 2026-02-20T11:12:11.814 INFO:teuthology.orchestra.run.trial109.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 149 k 2026-02-20T11:12:11.814 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:11.814 INFO:teuthology.orchestra.run.trial109.stdout:Transaction Summary 2026-02-20T11:12:11.814 INFO:teuthology.orchestra.run.trial109.stdout:================================================================================ 2026-02-20T11:12:11.814 INFO:teuthology.orchestra.run.trial109.stdout:Install 1 Package 2026-02-20T11:12:11.814 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:11.814 INFO:teuthology.orchestra.run.trial109.stdout:Total download size: 149 k 2026-02-20T11:12:11.814 INFO:teuthology.orchestra.run.trial109.stdout:Installed size: 455 k 2026-02-20T11:12:11.814 INFO:teuthology.orchestra.run.trial109.stdout:Downloading Packages: 2026-02-20T11:12:11.948 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:01:22 ago on Fri 20 Feb 2026 11:10:49 AM UTC. 2026-02-20T11:12:12.031 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:12:12.032 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-20T11:12:12.032 INFO:teuthology.orchestra.run.trial043.stdout: Package Arch Version Repo Size 2026-02-20T11:12:12.032 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-20T11:12:12.032 INFO:teuthology.orchestra.run.trial043.stdout:Installing: 2026-02-20T11:12:12.032 INFO:teuthology.orchestra.run.trial043.stdout: ceph-fuse x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 914 k 2026-02-20T11:12:12.032 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:12.032 INFO:teuthology.orchestra.run.trial043.stdout:Transaction Summary 2026-02-20T11:12:12.032 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-20T11:12:12.032 INFO:teuthology.orchestra.run.trial043.stdout:Install 1 Package 2026-02-20T11:12:12.032 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:12.033 INFO:teuthology.orchestra.run.trial043.stdout:Total download size: 914 k 2026-02-20T11:12:12.033 INFO:teuthology.orchestra.run.trial043.stdout:Installed size: 2.9 M 2026-02-20T11:12:12.033 INFO:teuthology.orchestra.run.trial043.stdout:Downloading Packages: 2026-02-20T11:12:12.208 INFO:teuthology.orchestra.run.trial109.stdout:ceph-immutable-object-cache-20.3.0-5357.g0c3148 381 kB/s | 149 kB 00:00 2026-02-20T11:12:12.208 INFO:teuthology.orchestra.run.trial109.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:12:12.209 INFO:teuthology.orchestra.run.trial109.stdout:Total 380 kB/s | 149 kB 00:00 2026-02-20T11:12:12.209 INFO:teuthology.orchestra.run.trial109.stdout:Running transaction check 2026-02-20T11:12:12.213 INFO:teuthology.orchestra.run.trial109.stdout:Transaction check succeeded. 2026-02-20T11:12:12.213 INFO:teuthology.orchestra.run.trial109.stdout:Running transaction test 2026-02-20T11:12:12.244 INFO:teuthology.orchestra.run.trial109.stdout:Transaction test succeeded. 2026-02-20T11:12:12.245 INFO:teuthology.orchestra.run.trial109.stdout:Running transaction 2026-02-20T11:12:12.299 INFO:teuthology.orchestra.run.trial109.stdout: Preparing : 1/1 2026-02-20T11:12:12.300 INFO:teuthology.orchestra.run.trial109.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5357.g0c3148ca. 1/1 2026-02-20T11:12:12.322 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5357.g0c3148ca. 1/1 2026-02-20T11:12:12.322 INFO:teuthology.orchestra.run.trial109.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T11:12:12.322 INFO:teuthology.orchestra.run.trial109.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-20T11:12:12.322 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:12.623 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5357.g0c3148ca. 1/1 2026-02-20T11:12:12.623 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:12.623 INFO:teuthology.orchestra.run.trial109.stdout:Installed: 2026-02-20T11:12:12.623 INFO:teuthology.orchestra.run.trial109.stdout: ceph-immutable-object-cache-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:12:12.623 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:12.623 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:12:12.650 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install ceph-mgr 2026-02-20T11:12:12.751 INFO:teuthology.orchestra.run.trial043.stdout:ceph-fuse-20.3.0-5357.g0c3148ca.el9.x86_64.rpm 1.2 MB/s | 914 kB 00:00 2026-02-20T11:12:12.752 INFO:teuthology.orchestra.run.trial043.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:12:12.752 INFO:teuthology.orchestra.run.trial043.stdout:Total 1.2 MB/s | 914 kB 00:00 2026-02-20T11:12:12.753 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction check 2026-02-20T11:12:12.757 INFO:teuthology.orchestra.run.trial043.stdout:Transaction check succeeded. 2026-02-20T11:12:12.757 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction test 2026-02-20T11:12:12.790 INFO:teuthology.orchestra.run.trial043.stdout:Transaction test succeeded. 2026-02-20T11:12:12.790 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction 2026-02-20T11:12:12.881 INFO:teuthology.orchestra.run.trial043.stdout: Preparing : 1/1 2026-02-20T11:12:12.898 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:12:12.961 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:01:21 ago on Fri 20 Feb 2026 11:10:51 AM UTC. 2026-02-20T11:12:13.028 INFO:teuthology.orchestra.run.trial109.stdout:Package ceph-mgr-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:12:13.046 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:12:13.046 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:12:13.046 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:12:13.067 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install ceph-mgr-dashboard 2026-02-20T11:12:13.148 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:12:13.205 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:12:13.205 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:13.206 INFO:teuthology.orchestra.run.trial043.stdout:Installed: 2026-02-20T11:12:13.206 INFO:teuthology.orchestra.run.trial043.stdout: ceph-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:12:13.206 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:13.206 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:12:13.234 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-volume 2026-02-20T11:12:13.375 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:01:22 ago on Fri 20 Feb 2026 11:10:51 AM UTC. 2026-02-20T11:12:13.440 INFO:teuthology.orchestra.run.trial109.stdout:Package ceph-mgr-dashboard-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:12:13.458 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:12:13.459 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:12:13.459 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:12:13.480 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-20T11:12:13.546 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:01:24 ago on Fri 20 Feb 2026 11:10:49 AM UTC. 2026-02-20T11:12:13.611 INFO:teuthology.orchestra.run.trial043.stdout:Package ceph-volume-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:12:13.629 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:12:13.630 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:12:13.630 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:12:13.651 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install librados-devel 2026-02-20T11:12:13.791 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:01:22 ago on Fri 20 Feb 2026 11:10:51 AM UTC. 2026-02-20T11:12:13.856 INFO:teuthology.orchestra.run.trial109.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:12:13.874 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:12:13.875 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:12:13.875 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:12:13.897 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install ceph-mgr-rook 2026-02-20T11:12:13.961 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:01:24 ago on Fri 20 Feb 2026 11:10:49 AM UTC. 2026-02-20T11:12:14.043 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:12:14.043 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-20T11:12:14.044 INFO:teuthology.orchestra.run.trial043.stdout: Package Arch Version Repo Size 2026-02-20T11:12:14.044 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-20T11:12:14.044 INFO:teuthology.orchestra.run.trial043.stdout:Installing: 2026-02-20T11:12:14.044 INFO:teuthology.orchestra.run.trial043.stdout: librados-devel x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 124 k 2026-02-20T11:12:14.044 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:14.044 INFO:teuthology.orchestra.run.trial043.stdout:Transaction Summary 2026-02-20T11:12:14.044 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-20T11:12:14.044 INFO:teuthology.orchestra.run.trial043.stdout:Install 1 Package 2026-02-20T11:12:14.044 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:14.044 INFO:teuthology.orchestra.run.trial043.stdout:Total download size: 124 k 2026-02-20T11:12:14.044 INFO:teuthology.orchestra.run.trial043.stdout:Installed size: 434 k 2026-02-20T11:12:14.045 INFO:teuthology.orchestra.run.trial043.stdout:Downloading Packages: 2026-02-20T11:12:14.203 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:01:23 ago on Fri 20 Feb 2026 11:10:51 AM UTC. 2026-02-20T11:12:14.269 INFO:teuthology.orchestra.run.trial109.stdout:Package ceph-mgr-rook-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:12:14.287 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:12:14.287 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:12:14.288 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:12:14.309 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install ceph-mgr-cephadm 2026-02-20T11:12:14.432 INFO:teuthology.orchestra.run.trial043.stdout:librados-devel-20.3.0-5357.g0c3148ca.el9.x86_64 319 kB/s | 124 kB 00:00 2026-02-20T11:12:14.433 INFO:teuthology.orchestra.run.trial043.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:12:14.433 INFO:teuthology.orchestra.run.trial043.stdout:Total 318 kB/s | 124 kB 00:00 2026-02-20T11:12:14.433 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction check 2026-02-20T11:12:14.435 INFO:teuthology.orchestra.run.trial043.stdout:Transaction check succeeded. 2026-02-20T11:12:14.435 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction test 2026-02-20T11:12:14.470 INFO:teuthology.orchestra.run.trial043.stdout:Transaction test succeeded. 2026-02-20T11:12:14.470 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction 2026-02-20T11:12:14.526 INFO:teuthology.orchestra.run.trial043.stdout: Preparing : 1/1 2026-02-20T11:12:14.543 INFO:teuthology.orchestra.run.trial043.stdout: Installing : librados-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:12:14.616 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:01:23 ago on Fri 20 Feb 2026 11:10:51 AM UTC. 2026-02-20T11:12:14.622 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: librados-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:12:14.680 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : librados-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:12:14.680 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:14.680 INFO:teuthology.orchestra.run.trial043.stdout:Installed: 2026-02-20T11:12:14.680 INFO:teuthology.orchestra.run.trial043.stdout: librados-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:12:14.681 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:14.681 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:12:14.681 INFO:teuthology.orchestra.run.trial109.stdout:Package ceph-mgr-cephadm-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:12:14.698 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:12:14.699 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:12:14.699 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:12:14.708 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install libcephfs2 2026-02-20T11:12:14.720 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install ceph-osd 2026-02-20T11:12:15.023 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:01:26 ago on Fri 20 Feb 2026 11:10:49 AM UTC. 2026-02-20T11:12:15.028 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:01:24 ago on Fri 20 Feb 2026 11:10:51 AM UTC. 2026-02-20T11:12:15.087 INFO:teuthology.orchestra.run.trial043.stdout:Package libcephfs2-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:12:15.092 INFO:teuthology.orchestra.run.trial109.stdout:Package ceph-osd-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:12:15.106 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:12:15.106 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:12:15.107 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:12:15.111 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:12:15.111 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:12:15.111 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:12:15.128 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install libcephfs-devel 2026-02-20T11:12:15.134 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install ceph-osd-classic 2026-02-20T11:12:15.435 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:01:26 ago on Fri 20 Feb 2026 11:10:49 AM UTC. 2026-02-20T11:12:15.448 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:01:24 ago on Fri 20 Feb 2026 11:10:51 AM UTC. 2026-02-20T11:12:15.511 INFO:teuthology.orchestra.run.trial109.stdout:Package ceph-osd-classic-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:12:15.517 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:12:15.518 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-20T11:12:15.518 INFO:teuthology.orchestra.run.trial043.stdout: Package Arch Version Repo Size 2026-02-20T11:12:15.518 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-20T11:12:15.518 INFO:teuthology.orchestra.run.trial043.stdout:Installing: 2026-02-20T11:12:15.519 INFO:teuthology.orchestra.run.trial043.stdout: libcephfs-devel x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 37 k 2026-02-20T11:12:15.519 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:15.519 INFO:teuthology.orchestra.run.trial043.stdout:Transaction Summary 2026-02-20T11:12:15.519 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-20T11:12:15.519 INFO:teuthology.orchestra.run.trial043.stdout:Install 1 Package 2026-02-20T11:12:15.519 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:15.519 INFO:teuthology.orchestra.run.trial043.stdout:Total download size: 37 k 2026-02-20T11:12:15.519 INFO:teuthology.orchestra.run.trial043.stdout:Installed size: 171 k 2026-02-20T11:12:15.519 INFO:teuthology.orchestra.run.trial043.stdout:Downloading Packages: 2026-02-20T11:12:15.529 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:12:15.530 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:12:15.530 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:12:15.551 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install ceph-fuse 2026-02-20T11:12:15.804 INFO:teuthology.orchestra.run.trial043.stdout:libcephfs-devel-20.3.0-5357.g0c3148ca.el9.x86_6 131 kB/s | 37 kB 00:00 2026-02-20T11:12:15.804 INFO:teuthology.orchestra.run.trial043.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:12:15.804 INFO:teuthology.orchestra.run.trial043.stdout:Total 130 kB/s | 37 kB 00:00 2026-02-20T11:12:15.804 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction check 2026-02-20T11:12:15.807 INFO:teuthology.orchestra.run.trial043.stdout:Transaction check succeeded. 2026-02-20T11:12:15.807 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction test 2026-02-20T11:12:15.809 INFO:teuthology.orchestra.run.trial043.stdout:Transaction test succeeded. 2026-02-20T11:12:15.810 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction 2026-02-20T11:12:15.831 INFO:teuthology.orchestra.run.trial043.stdout: Preparing : 1/1 2026-02-20T11:12:15.847 INFO:teuthology.orchestra.run.trial043.stdout: Installing : libcephfs-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:12:15.863 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:01:24 ago on Fri 20 Feb 2026 11:10:51 AM UTC. 2026-02-20T11:12:15.907 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:12:15.946 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:12:15.947 INFO:teuthology.orchestra.run.trial109.stdout:================================================================================ 2026-02-20T11:12:15.947 INFO:teuthology.orchestra.run.trial109.stdout: Package Arch Version Repo Size 2026-02-20T11:12:15.947 INFO:teuthology.orchestra.run.trial109.stdout:================================================================================ 2026-02-20T11:12:15.947 INFO:teuthology.orchestra.run.trial109.stdout:Installing: 2026-02-20T11:12:15.947 INFO:teuthology.orchestra.run.trial109.stdout: ceph-fuse x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 914 k 2026-02-20T11:12:15.947 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:15.947 INFO:teuthology.orchestra.run.trial109.stdout:Transaction Summary 2026-02-20T11:12:15.947 INFO:teuthology.orchestra.run.trial109.stdout:================================================================================ 2026-02-20T11:12:15.947 INFO:teuthology.orchestra.run.trial109.stdout:Install 1 Package 2026-02-20T11:12:15.947 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:15.948 INFO:teuthology.orchestra.run.trial109.stdout:Total download size: 914 k 2026-02-20T11:12:15.948 INFO:teuthology.orchestra.run.trial109.stdout:Installed size: 2.9 M 2026-02-20T11:12:15.948 INFO:teuthology.orchestra.run.trial109.stdout:Downloading Packages: 2026-02-20T11:12:15.964 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : libcephfs-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:12:15.964 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:15.964 INFO:teuthology.orchestra.run.trial043.stdout:Installed: 2026-02-20T11:12:15.965 INFO:teuthology.orchestra.run.trial043.stdout: libcephfs-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:12:15.965 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:15.965 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:12:15.990 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install librados2 2026-02-20T11:12:16.298 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:01:27 ago on Fri 20 Feb 2026 11:10:49 AM UTC. 2026-02-20T11:12:16.363 INFO:teuthology.orchestra.run.trial043.stdout:Package librados2-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:12:16.381 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:12:16.382 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:12:16.382 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:12:16.403 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install librbd1 2026-02-20T11:12:16.549 INFO:teuthology.orchestra.run.trial109.stdout:ceph-fuse-20.3.0-5357.g0c3148ca.el9.x86_64.rpm 1.5 MB/s | 914 kB 00:00 2026-02-20T11:12:16.549 INFO:teuthology.orchestra.run.trial109.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:12:16.549 INFO:teuthology.orchestra.run.trial109.stdout:Total 1.5 MB/s | 914 kB 00:00 2026-02-20T11:12:16.549 INFO:teuthology.orchestra.run.trial109.stdout:Running transaction check 2026-02-20T11:12:16.554 INFO:teuthology.orchestra.run.trial109.stdout:Transaction check succeeded. 2026-02-20T11:12:16.554 INFO:teuthology.orchestra.run.trial109.stdout:Running transaction test 2026-02-20T11:12:16.587 INFO:teuthology.orchestra.run.trial109.stdout:Transaction test succeeded. 2026-02-20T11:12:16.588 INFO:teuthology.orchestra.run.trial109.stdout:Running transaction 2026-02-20T11:12:16.678 INFO:teuthology.orchestra.run.trial109.stdout: Preparing : 1/1 2026-02-20T11:12:16.694 INFO:teuthology.orchestra.run.trial109.stdout: Installing : ceph-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:12:16.708 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:01:27 ago on Fri 20 Feb 2026 11:10:49 AM UTC. 2026-02-20T11:12:16.773 INFO:teuthology.orchestra.run.trial043.stdout:Package librbd1-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:12:16.791 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:12:16.791 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:12:16.792 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:12:16.813 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install python3-rados 2026-02-20T11:12:16.938 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:12:16.995 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : ceph-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:12:16.995 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:16.995 INFO:teuthology.orchestra.run.trial109.stdout:Installed: 2026-02-20T11:12:16.995 INFO:teuthology.orchestra.run.trial109.stdout: ceph-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:12:16.995 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:16.995 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:12:17.024 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install ceph-volume 2026-02-20T11:12:17.118 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:01:28 ago on Fri 20 Feb 2026 11:10:49 AM UTC. 2026-02-20T11:12:17.183 INFO:teuthology.orchestra.run.trial043.stdout:Package python3-rados-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:12:17.202 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:12:17.202 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:12:17.202 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:12:17.225 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install python3-rgw 2026-02-20T11:12:17.338 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:01:26 ago on Fri 20 Feb 2026 11:10:51 AM UTC. 2026-02-20T11:12:17.404 INFO:teuthology.orchestra.run.trial109.stdout:Package ceph-volume-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:12:17.423 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:12:17.423 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:12:17.424 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:12:17.445 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install librados-devel 2026-02-20T11:12:17.533 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:01:28 ago on Fri 20 Feb 2026 11:10:49 AM UTC. 2026-02-20T11:12:17.599 INFO:teuthology.orchestra.run.trial043.stdout:Package python3-rgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:12:17.617 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:12:17.618 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:12:17.618 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:12:17.639 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install python3-cephfs 2026-02-20T11:12:17.756 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:01:26 ago on Fri 20 Feb 2026 11:10:51 AM UTC. 2026-02-20T11:12:17.839 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:12:17.839 INFO:teuthology.orchestra.run.trial109.stdout:================================================================================ 2026-02-20T11:12:17.840 INFO:teuthology.orchestra.run.trial109.stdout: Package Arch Version Repo Size 2026-02-20T11:12:17.840 INFO:teuthology.orchestra.run.trial109.stdout:================================================================================ 2026-02-20T11:12:17.840 INFO:teuthology.orchestra.run.trial109.stdout:Installing: 2026-02-20T11:12:17.840 INFO:teuthology.orchestra.run.trial109.stdout: librados-devel x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 124 k 2026-02-20T11:12:17.840 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:17.840 INFO:teuthology.orchestra.run.trial109.stdout:Transaction Summary 2026-02-20T11:12:17.840 INFO:teuthology.orchestra.run.trial109.stdout:================================================================================ 2026-02-20T11:12:17.840 INFO:teuthology.orchestra.run.trial109.stdout:Install 1 Package 2026-02-20T11:12:17.840 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:17.840 INFO:teuthology.orchestra.run.trial109.stdout:Total download size: 124 k 2026-02-20T11:12:17.840 INFO:teuthology.orchestra.run.trial109.stdout:Installed size: 434 k 2026-02-20T11:12:17.840 INFO:teuthology.orchestra.run.trial109.stdout:Downloading Packages: 2026-02-20T11:12:17.947 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:01:28 ago on Fri 20 Feb 2026 11:10:49 AM UTC. 2026-02-20T11:12:18.012 INFO:teuthology.orchestra.run.trial043.stdout:Package python3-cephfs-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:12:18.030 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:12:18.031 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:12:18.031 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:12:18.052 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install python3-rbd 2026-02-20T11:12:18.305 INFO:teuthology.orchestra.run.trial109.stdout:librados-devel-20.3.0-5357.g0c3148ca.el9.x86_64 266 kB/s | 124 kB 00:00 2026-02-20T11:12:18.306 INFO:teuthology.orchestra.run.trial109.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:12:18.306 INFO:teuthology.orchestra.run.trial109.stdout:Total 265 kB/s | 124 kB 00:00 2026-02-20T11:12:18.306 INFO:teuthology.orchestra.run.trial109.stdout:Running transaction check 2026-02-20T11:12:18.309 INFO:teuthology.orchestra.run.trial109.stdout:Transaction check succeeded. 2026-02-20T11:12:18.309 INFO:teuthology.orchestra.run.trial109.stdout:Running transaction test 2026-02-20T11:12:18.344 INFO:teuthology.orchestra.run.trial109.stdout:Transaction test succeeded. 2026-02-20T11:12:18.344 INFO:teuthology.orchestra.run.trial109.stdout:Running transaction 2026-02-20T11:12:18.355 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:01:29 ago on Fri 20 Feb 2026 11:10:49 AM UTC. 2026-02-20T11:12:18.400 INFO:teuthology.orchestra.run.trial109.stdout: Preparing : 1/1 2026-02-20T11:12:18.417 INFO:teuthology.orchestra.run.trial109.stdout: Installing : librados-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:12:18.420 INFO:teuthology.orchestra.run.trial043.stdout:Package python3-rbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:12:18.438 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:12:18.439 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:12:18.439 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:12:18.459 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install rbd-fuse 2026-02-20T11:12:18.498 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: librados-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:12:18.556 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : librados-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:12:18.557 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:18.557 INFO:teuthology.orchestra.run.trial109.stdout:Installed: 2026-02-20T11:12:18.557 INFO:teuthology.orchestra.run.trial109.stdout: librados-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:12:18.557 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:18.557 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:12:18.585 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install libcephfs2 2026-02-20T11:12:18.764 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:01:29 ago on Fri 20 Feb 2026 11:10:49 AM UTC. 2026-02-20T11:12:18.848 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:12:18.849 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-20T11:12:18.849 INFO:teuthology.orchestra.run.trial043.stdout: Package Architecture Version Repository Size 2026-02-20T11:12:18.849 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-20T11:12:18.849 INFO:teuthology.orchestra.run.trial043.stdout:Installing: 2026-02-20T11:12:18.849 INFO:teuthology.orchestra.run.trial043.stdout: rbd-fuse x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 86 k 2026-02-20T11:12:18.849 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:18.850 INFO:teuthology.orchestra.run.trial043.stdout:Transaction Summary 2026-02-20T11:12:18.850 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-20T11:12:18.850 INFO:teuthology.orchestra.run.trial043.stdout:Install 1 Package 2026-02-20T11:12:18.850 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:18.850 INFO:teuthology.orchestra.run.trial043.stdout:Total download size: 86 k 2026-02-20T11:12:18.850 INFO:teuthology.orchestra.run.trial043.stdout:Installed size: 236 k 2026-02-20T11:12:18.850 INFO:teuthology.orchestra.run.trial043.stdout:Downloading Packages: 2026-02-20T11:12:18.895 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:01:27 ago on Fri 20 Feb 2026 11:10:51 AM UTC. 2026-02-20T11:12:18.961 INFO:teuthology.orchestra.run.trial109.stdout:Package libcephfs2-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:12:18.979 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:12:18.980 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:12:18.980 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:12:19.002 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install libcephfs-devel 2026-02-20T11:12:19.259 INFO:teuthology.orchestra.run.trial043.stdout:rbd-fuse-20.3.0-5357.g0c3148ca.el9.x86_64.rpm 211 kB/s | 86 kB 00:00 2026-02-20T11:12:19.259 INFO:teuthology.orchestra.run.trial043.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:12:19.260 INFO:teuthology.orchestra.run.trial043.stdout:Total 210 kB/s | 86 kB 00:00 2026-02-20T11:12:19.260 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction check 2026-02-20T11:12:19.263 INFO:teuthology.orchestra.run.trial043.stdout:Transaction check succeeded. 2026-02-20T11:12:19.264 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction test 2026-02-20T11:12:19.294 INFO:teuthology.orchestra.run.trial043.stdout:Transaction test succeeded. 2026-02-20T11:12:19.295 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction 2026-02-20T11:12:19.312 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:01:28 ago on Fri 20 Feb 2026 11:10:51 AM UTC. 2026-02-20T11:12:19.346 INFO:teuthology.orchestra.run.trial043.stdout: Preparing : 1/1 2026-02-20T11:12:19.362 INFO:teuthology.orchestra.run.trial043.stdout: Installing : rbd-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:12:19.394 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:12:19.395 INFO:teuthology.orchestra.run.trial109.stdout:================================================================================ 2026-02-20T11:12:19.395 INFO:teuthology.orchestra.run.trial109.stdout: Package Arch Version Repo Size 2026-02-20T11:12:19.395 INFO:teuthology.orchestra.run.trial109.stdout:================================================================================ 2026-02-20T11:12:19.395 INFO:teuthology.orchestra.run.trial109.stdout:Installing: 2026-02-20T11:12:19.396 INFO:teuthology.orchestra.run.trial109.stdout: libcephfs-devel x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 37 k 2026-02-20T11:12:19.396 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:19.396 INFO:teuthology.orchestra.run.trial109.stdout:Transaction Summary 2026-02-20T11:12:19.396 INFO:teuthology.orchestra.run.trial109.stdout:================================================================================ 2026-02-20T11:12:19.396 INFO:teuthology.orchestra.run.trial109.stdout:Install 1 Package 2026-02-20T11:12:19.396 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:19.396 INFO:teuthology.orchestra.run.trial109.stdout:Total download size: 37 k 2026-02-20T11:12:19.396 INFO:teuthology.orchestra.run.trial109.stdout:Installed size: 171 k 2026-02-20T11:12:19.396 INFO:teuthology.orchestra.run.trial109.stdout:Downloading Packages: 2026-02-20T11:12:19.442 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:12:19.500 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : rbd-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:12:19.500 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:19.501 INFO:teuthology.orchestra.run.trial043.stdout:Installed: 2026-02-20T11:12:19.501 INFO:teuthology.orchestra.run.trial043.stdout: rbd-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:12:19.501 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:19.501 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:12:19.528 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install rbd-mirror 2026-02-20T11:12:19.733 INFO:teuthology.orchestra.run.trial109.stdout:libcephfs-devel-20.3.0-5357.g0c3148ca.el9.x86_6 111 kB/s | 37 kB 00:00 2026-02-20T11:12:19.733 INFO:teuthology.orchestra.run.trial109.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:12:19.733 INFO:teuthology.orchestra.run.trial109.stdout:Total 110 kB/s | 37 kB 00:00 2026-02-20T11:12:19.733 INFO:teuthology.orchestra.run.trial109.stdout:Running transaction check 2026-02-20T11:12:19.736 INFO:teuthology.orchestra.run.trial109.stdout:Transaction check succeeded. 2026-02-20T11:12:19.736 INFO:teuthology.orchestra.run.trial109.stdout:Running transaction test 2026-02-20T11:12:19.738 INFO:teuthology.orchestra.run.trial109.stdout:Transaction test succeeded. 2026-02-20T11:12:19.739 INFO:teuthology.orchestra.run.trial109.stdout:Running transaction 2026-02-20T11:12:19.761 INFO:teuthology.orchestra.run.trial109.stdout: Preparing : 1/1 2026-02-20T11:12:19.777 INFO:teuthology.orchestra.run.trial109.stdout: Installing : libcephfs-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:12:19.834 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:12:19.836 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:01:30 ago on Fri 20 Feb 2026 11:10:49 AM UTC. 2026-02-20T11:12:19.892 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : libcephfs-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:12:19.892 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:19.892 INFO:teuthology.orchestra.run.trial109.stdout:Installed: 2026-02-20T11:12:19.892 INFO:teuthology.orchestra.run.trial109.stdout: libcephfs-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:12:19.892 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:19.892 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:12:19.920 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install librados2 2026-02-20T11:12:19.921 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:12:19.921 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-20T11:12:19.921 INFO:teuthology.orchestra.run.trial043.stdout: Package Arch Version Repo Size 2026-02-20T11:12:19.921 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-20T11:12:19.922 INFO:teuthology.orchestra.run.trial043.stdout:Installing: 2026-02-20T11:12:19.922 INFO:teuthology.orchestra.run.trial043.stdout: rbd-mirror x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 2.6 M 2026-02-20T11:12:19.922 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:19.922 INFO:teuthology.orchestra.run.trial043.stdout:Transaction Summary 2026-02-20T11:12:19.922 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-20T11:12:19.922 INFO:teuthology.orchestra.run.trial043.stdout:Install 1 Package 2026-02-20T11:12:19.922 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:19.922 INFO:teuthology.orchestra.run.trial043.stdout:Total download size: 2.6 M 2026-02-20T11:12:19.922 INFO:teuthology.orchestra.run.trial043.stdout:Installed size: 11 M 2026-02-20T11:12:19.922 INFO:teuthology.orchestra.run.trial043.stdout:Downloading Packages: 2026-02-20T11:12:20.229 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:01:29 ago on Fri 20 Feb 2026 11:10:51 AM UTC. 2026-02-20T11:12:20.293 INFO:teuthology.orchestra.run.trial109.stdout:Package librados2-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:12:20.312 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:12:20.313 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:12:20.313 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:12:20.334 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install librbd1 2026-02-20T11:12:20.645 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:01:29 ago on Fri 20 Feb 2026 11:10:51 AM UTC. 2026-02-20T11:12:20.710 INFO:teuthology.orchestra.run.trial109.stdout:Package librbd1-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:12:20.728 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:12:20.729 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:12:20.729 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:12:20.750 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install python3-rados 2026-02-20T11:12:20.755 INFO:teuthology.orchestra.run.trial043.stdout:rbd-mirror-20.3.0-5357.g0c3148ca.el9.x86_64.rpm 3.1 MB/s | 2.6 MB 00:00 2026-02-20T11:12:20.755 INFO:teuthology.orchestra.run.trial043.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:12:20.756 INFO:teuthology.orchestra.run.trial043.stdout:Total 3.1 MB/s | 2.6 MB 00:00 2026-02-20T11:12:20.756 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction check 2026-02-20T11:12:20.761 INFO:teuthology.orchestra.run.trial043.stdout:Transaction check succeeded. 2026-02-20T11:12:20.761 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction test 2026-02-20T11:12:20.800 INFO:teuthology.orchestra.run.trial043.stdout:Transaction test succeeded. 2026-02-20T11:12:20.800 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction 2026-02-20T11:12:20.965 INFO:teuthology.orchestra.run.trial043.stdout: Preparing : 1/1 2026-02-20T11:12:20.967 INFO:teuthology.orchestra.run.trial043.stdout: Installing : rbd-mirror-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:12:20.988 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:12:20.989 INFO:teuthology.orchestra.run.trial043.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T11:12:20.989 INFO:teuthology.orchestra.run.trial043.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-20T11:12:20.989 INFO:teuthology.orchestra.run.trial043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-20T11:12:20.989 INFO:teuthology.orchestra.run.trial043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-20T11:12:20.989 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:21.065 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:01:30 ago on Fri 20 Feb 2026 11:10:51 AM UTC. 2026-02-20T11:12:21.130 INFO:teuthology.orchestra.run.trial109.stdout:Package python3-rados-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:12:21.148 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:12:21.149 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:12:21.149 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:12:21.171 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install python3-rgw 2026-02-20T11:12:21.309 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : rbd-mirror-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:12:21.310 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:21.310 INFO:teuthology.orchestra.run.trial043.stdout:Installed: 2026-02-20T11:12:21.310 INFO:teuthology.orchestra.run.trial043.stdout: rbd-mirror-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:12:21.310 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:21.310 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:12:21.337 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install rbd-nbd 2026-02-20T11:12:21.484 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:01:30 ago on Fri 20 Feb 2026 11:10:51 AM UTC. 2026-02-20T11:12:21.549 INFO:teuthology.orchestra.run.trial109.stdout:Package python3-rgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:12:21.567 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:12:21.568 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:12:21.568 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:12:21.589 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install python3-cephfs 2026-02-20T11:12:21.645 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:01:32 ago on Fri 20 Feb 2026 11:10:49 AM UTC. 2026-02-20T11:12:21.729 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:12:21.729 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-20T11:12:21.730 INFO:teuthology.orchestra.run.trial043.stdout: Package Architecture Version Repository Size 2026-02-20T11:12:21.730 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-20T11:12:21.730 INFO:teuthology.orchestra.run.trial043.stdout:Installing: 2026-02-20T11:12:21.730 INFO:teuthology.orchestra.run.trial043.stdout: rbd-nbd x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 167 k 2026-02-20T11:12:21.730 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:21.730 INFO:teuthology.orchestra.run.trial043.stdout:Transaction Summary 2026-02-20T11:12:21.730 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-20T11:12:21.730 INFO:teuthology.orchestra.run.trial043.stdout:Install 1 Package 2026-02-20T11:12:21.730 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:21.730 INFO:teuthology.orchestra.run.trial043.stdout:Total download size: 167 k 2026-02-20T11:12:21.731 INFO:teuthology.orchestra.run.trial043.stdout:Installed size: 496 k 2026-02-20T11:12:21.731 INFO:teuthology.orchestra.run.trial043.stdout:Downloading Packages: 2026-02-20T11:12:21.906 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:01:30 ago on Fri 20 Feb 2026 11:10:51 AM UTC. 2026-02-20T11:12:21.970 INFO:teuthology.orchestra.run.trial109.stdout:Package python3-cephfs-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:12:21.988 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:12:21.989 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:12:21.989 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:12:22.011 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install python3-rbd 2026-02-20T11:12:22.198 INFO:teuthology.orchestra.run.trial043.stdout:rbd-nbd-20.3.0-5357.g0c3148ca.el9.x86_64.rpm 356 kB/s | 167 kB 00:00 2026-02-20T11:12:22.198 INFO:teuthology.orchestra.run.trial043.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:12:22.198 INFO:teuthology.orchestra.run.trial043.stdout:Total 355 kB/s | 167 kB 00:00 2026-02-20T11:12:22.198 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction check 2026-02-20T11:12:22.203 INFO:teuthology.orchestra.run.trial043.stdout:Transaction check succeeded. 2026-02-20T11:12:22.203 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction test 2026-02-20T11:12:22.237 INFO:teuthology.orchestra.run.trial043.stdout:Transaction test succeeded. 2026-02-20T11:12:22.237 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction 2026-02-20T11:12:22.295 INFO:teuthology.orchestra.run.trial043.stdout: Preparing : 1/1 2026-02-20T11:12:22.313 INFO:teuthology.orchestra.run.trial043.stdout: Installing : rbd-nbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:12:22.317 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:01:31 ago on Fri 20 Feb 2026 11:10:51 AM UTC. 2026-02-20T11:12:22.384 INFO:teuthology.orchestra.run.trial109.stdout:Package python3-rbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:12:22.395 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:12:22.402 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:12:22.403 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:12:22.403 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:12:22.425 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install rbd-fuse 2026-02-20T11:12:22.452 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : rbd-nbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:12:22.453 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:22.453 INFO:teuthology.orchestra.run.trial043.stdout:Installed: 2026-02-20T11:12:22.453 INFO:teuthology.orchestra.run.trial043.stdout: rbd-nbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:12:22.453 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:22.453 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:12:22.481 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-osd-crimson 2026-02-20T11:12:22.733 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:01:31 ago on Fri 20 Feb 2026 11:10:51 AM UTC. 2026-02-20T11:12:22.787 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:01:33 ago on Fri 20 Feb 2026 11:10:49 AM UTC. 2026-02-20T11:12:22.817 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:12:22.817 INFO:teuthology.orchestra.run.trial109.stdout:================================================================================ 2026-02-20T11:12:22.818 INFO:teuthology.orchestra.run.trial109.stdout: Package Architecture Version Repository Size 2026-02-20T11:12:22.818 INFO:teuthology.orchestra.run.trial109.stdout:================================================================================ 2026-02-20T11:12:22.818 INFO:teuthology.orchestra.run.trial109.stdout:Installing: 2026-02-20T11:12:22.818 INFO:teuthology.orchestra.run.trial109.stdout: rbd-fuse x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 86 k 2026-02-20T11:12:22.818 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:22.819 INFO:teuthology.orchestra.run.trial109.stdout:Transaction Summary 2026-02-20T11:12:22.819 INFO:teuthology.orchestra.run.trial109.stdout:================================================================================ 2026-02-20T11:12:22.819 INFO:teuthology.orchestra.run.trial109.stdout:Install 1 Package 2026-02-20T11:12:22.819 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:22.819 INFO:teuthology.orchestra.run.trial109.stdout:Total download size: 86 k 2026-02-20T11:12:22.819 INFO:teuthology.orchestra.run.trial109.stdout:Installed size: 236 k 2026-02-20T11:12:22.819 INFO:teuthology.orchestra.run.trial109.stdout:Downloading Packages: 2026-02-20T11:12:22.851 INFO:teuthology.orchestra.run.trial043.stdout:Package ceph-osd-crimson-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:12:22.869 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:12:22.869 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:12:22.870 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:12:22.891 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-osd-crimson-debuginfo 2026-02-20T11:12:23.201 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:01:34 ago on Fri 20 Feb 2026 11:10:49 AM UTC. 2026-02-20T11:12:23.228 INFO:teuthology.orchestra.run.trial109.stdout:rbd-fuse-20.3.0-5357.g0c3148ca.el9.x86_64.rpm 210 kB/s | 86 kB 00:00 2026-02-20T11:12:23.228 INFO:teuthology.orchestra.run.trial109.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:12:23.228 INFO:teuthology.orchestra.run.trial109.stdout:Total 210 kB/s | 86 kB 00:00 2026-02-20T11:12:23.228 INFO:teuthology.orchestra.run.trial109.stdout:Running transaction check 2026-02-20T11:12:23.232 INFO:teuthology.orchestra.run.trial109.stdout:Transaction check succeeded. 2026-02-20T11:12:23.233 INFO:teuthology.orchestra.run.trial109.stdout:Running transaction test 2026-02-20T11:12:23.264 INFO:teuthology.orchestra.run.trial109.stdout:Transaction test succeeded. 2026-02-20T11:12:23.265 INFO:teuthology.orchestra.run.trial109.stdout:Running transaction 2026-02-20T11:12:23.285 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:12:23.285 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-20T11:12:23.285 INFO:teuthology.orchestra.run.trial043.stdout: Package Arch Version Repo Size 2026-02-20T11:12:23.285 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-20T11:12:23.285 INFO:teuthology.orchestra.run.trial043.stdout:Installing: 2026-02-20T11:12:23.285 INFO:teuthology.orchestra.run.trial043.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 433 M 2026-02-20T11:12:23.285 INFO:teuthology.orchestra.run.trial043.stdout:Installing weak dependencies: 2026-02-20T11:12:23.286 INFO:teuthology.orchestra.run.trial043.stdout: ceph-debugsource x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 12 M 2026-02-20T11:12:23.286 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:23.286 INFO:teuthology.orchestra.run.trial043.stdout:Transaction Summary 2026-02-20T11:12:23.286 INFO:teuthology.orchestra.run.trial043.stdout:================================================================================ 2026-02-20T11:12:23.286 INFO:teuthology.orchestra.run.trial043.stdout:Install 2 Packages 2026-02-20T11:12:23.286 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:12:23.286 INFO:teuthology.orchestra.run.trial043.stdout:Total download size: 445 M 2026-02-20T11:12:23.286 INFO:teuthology.orchestra.run.trial043.stdout:Installed size: 2.7 G 2026-02-20T11:12:23.286 INFO:teuthology.orchestra.run.trial043.stdout:Downloading Packages: 2026-02-20T11:12:23.315 INFO:teuthology.orchestra.run.trial109.stdout: Preparing : 1/1 2026-02-20T11:12:23.332 INFO:teuthology.orchestra.run.trial109.stdout: Installing : rbd-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:12:23.411 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:12:23.469 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : rbd-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:12:23.469 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:23.470 INFO:teuthology.orchestra.run.trial109.stdout:Installed: 2026-02-20T11:12:23.470 INFO:teuthology.orchestra.run.trial109.stdout: rbd-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:12:23.470 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:23.470 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:12:23.498 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install rbd-mirror 2026-02-20T11:12:23.806 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:01:32 ago on Fri 20 Feb 2026 11:10:51 AM UTC. 2026-02-20T11:12:23.890 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:12:23.890 INFO:teuthology.orchestra.run.trial109.stdout:================================================================================ 2026-02-20T11:12:23.890 INFO:teuthology.orchestra.run.trial109.stdout: Package Arch Version Repo Size 2026-02-20T11:12:23.890 INFO:teuthology.orchestra.run.trial109.stdout:================================================================================ 2026-02-20T11:12:23.890 INFO:teuthology.orchestra.run.trial109.stdout:Installing: 2026-02-20T11:12:23.890 INFO:teuthology.orchestra.run.trial109.stdout: rbd-mirror x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 2.6 M 2026-02-20T11:12:23.890 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:23.890 INFO:teuthology.orchestra.run.trial109.stdout:Transaction Summary 2026-02-20T11:12:23.890 INFO:teuthology.orchestra.run.trial109.stdout:================================================================================ 2026-02-20T11:12:23.890 INFO:teuthology.orchestra.run.trial109.stdout:Install 1 Package 2026-02-20T11:12:23.890 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:23.890 INFO:teuthology.orchestra.run.trial109.stdout:Total download size: 2.6 M 2026-02-20T11:12:23.890 INFO:teuthology.orchestra.run.trial109.stdout:Installed size: 11 M 2026-02-20T11:12:23.890 INFO:teuthology.orchestra.run.trial109.stdout:Downloading Packages: 2026-02-20T11:12:24.877 INFO:teuthology.orchestra.run.trial109.stdout:rbd-mirror-20.3.0-5357.g0c3148ca.el9.x86_64.rpm 2.6 MB/s | 2.6 MB 00:00 2026-02-20T11:12:24.877 INFO:teuthology.orchestra.run.trial109.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:12:24.878 INFO:teuthology.orchestra.run.trial109.stdout:Total 2.6 MB/s | 2.6 MB 00:00 2026-02-20T11:12:24.878 INFO:teuthology.orchestra.run.trial109.stdout:Running transaction check 2026-02-20T11:12:24.883 INFO:teuthology.orchestra.run.trial109.stdout:Transaction check succeeded. 2026-02-20T11:12:24.883 INFO:teuthology.orchestra.run.trial109.stdout:Running transaction test 2026-02-20T11:12:24.923 INFO:teuthology.orchestra.run.trial109.stdout:Transaction test succeeded. 2026-02-20T11:12:24.923 INFO:teuthology.orchestra.run.trial109.stdout:Running transaction 2026-02-20T11:12:25.086 INFO:teuthology.orchestra.run.trial109.stdout: Preparing : 1/1 2026-02-20T11:12:25.087 INFO:teuthology.orchestra.run.trial109.stdout: Installing : rbd-mirror-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:12:25.110 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:12:25.110 INFO:teuthology.orchestra.run.trial109.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T11:12:25.110 INFO:teuthology.orchestra.run.trial109.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-20T11:12:25.110 INFO:teuthology.orchestra.run.trial109.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-20T11:12:25.110 INFO:teuthology.orchestra.run.trial109.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-20T11:12:25.110 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:25.414 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : rbd-mirror-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:12:25.414 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:25.415 INFO:teuthology.orchestra.run.trial109.stdout:Installed: 2026-02-20T11:12:25.415 INFO:teuthology.orchestra.run.trial109.stdout: rbd-mirror-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:12:25.415 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:25.415 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:12:25.444 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install rbd-nbd 2026-02-20T11:12:25.755 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:01:34 ago on Fri 20 Feb 2026 11:10:51 AM UTC. 2026-02-20T11:12:25.839 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:12:25.839 INFO:teuthology.orchestra.run.trial109.stdout:================================================================================ 2026-02-20T11:12:25.840 INFO:teuthology.orchestra.run.trial109.stdout: Package Architecture Version Repository Size 2026-02-20T11:12:25.840 INFO:teuthology.orchestra.run.trial109.stdout:================================================================================ 2026-02-20T11:12:25.840 INFO:teuthology.orchestra.run.trial109.stdout:Installing: 2026-02-20T11:12:25.840 INFO:teuthology.orchestra.run.trial109.stdout: rbd-nbd x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 167 k 2026-02-20T11:12:25.840 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:25.840 INFO:teuthology.orchestra.run.trial109.stdout:Transaction Summary 2026-02-20T11:12:25.840 INFO:teuthology.orchestra.run.trial109.stdout:================================================================================ 2026-02-20T11:12:25.840 INFO:teuthology.orchestra.run.trial109.stdout:Install 1 Package 2026-02-20T11:12:25.840 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:25.840 INFO:teuthology.orchestra.run.trial109.stdout:Total download size: 167 k 2026-02-20T11:12:25.840 INFO:teuthology.orchestra.run.trial109.stdout:Installed size: 496 k 2026-02-20T11:12:25.841 INFO:teuthology.orchestra.run.trial109.stdout:Downloading Packages: 2026-02-20T11:12:26.607 INFO:teuthology.orchestra.run.trial109.stdout:rbd-nbd-20.3.0-5357.g0c3148ca.el9.x86_64.rpm 217 kB/s | 167 kB 00:00 2026-02-20T11:12:26.608 INFO:teuthology.orchestra.run.trial109.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:12:26.608 INFO:teuthology.orchestra.run.trial109.stdout:Total 217 kB/s | 167 kB 00:00 2026-02-20T11:12:26.608 INFO:teuthology.orchestra.run.trial109.stdout:Running transaction check 2026-02-20T11:12:26.614 INFO:teuthology.orchestra.run.trial109.stdout:Transaction check succeeded. 2026-02-20T11:12:26.614 INFO:teuthology.orchestra.run.trial109.stdout:Running transaction test 2026-02-20T11:12:26.647 INFO:teuthology.orchestra.run.trial109.stdout:Transaction test succeeded. 2026-02-20T11:12:26.648 INFO:teuthology.orchestra.run.trial109.stdout:Running transaction 2026-02-20T11:12:26.711 INFO:teuthology.orchestra.run.trial109.stdout: Preparing : 1/1 2026-02-20T11:12:26.736 INFO:teuthology.orchestra.run.trial109.stdout: Installing : rbd-nbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:12:26.816 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:12:26.873 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : rbd-nbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:12:26.874 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:26.874 INFO:teuthology.orchestra.run.trial109.stdout:Installed: 2026-02-20T11:12:26.874 INFO:teuthology.orchestra.run.trial109.stdout: rbd-nbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:12:26.874 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:26.874 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:12:26.903 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install ceph-osd-crimson 2026-02-20T11:12:27.216 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:01:36 ago on Fri 20 Feb 2026 11:10:51 AM UTC. 2026-02-20T11:12:27.280 INFO:teuthology.orchestra.run.trial109.stdout:Package ceph-osd-crimson-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:12:27.299 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:12:27.299 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:12:27.299 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:12:27.321 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install ceph-osd-crimson-debuginfo 2026-02-20T11:12:27.635 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:01:36 ago on Fri 20 Feb 2026 11:10:51 AM UTC. 2026-02-20T11:12:27.718 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:12:27.718 INFO:teuthology.orchestra.run.trial109.stdout:================================================================================ 2026-02-20T11:12:27.718 INFO:teuthology.orchestra.run.trial109.stdout: Package Arch Version Repo Size 2026-02-20T11:12:27.718 INFO:teuthology.orchestra.run.trial109.stdout:================================================================================ 2026-02-20T11:12:27.718 INFO:teuthology.orchestra.run.trial109.stdout:Installing: 2026-02-20T11:12:27.718 INFO:teuthology.orchestra.run.trial109.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 433 M 2026-02-20T11:12:27.718 INFO:teuthology.orchestra.run.trial109.stdout:Installing weak dependencies: 2026-02-20T11:12:27.718 INFO:teuthology.orchestra.run.trial109.stdout: ceph-debugsource x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 12 M 2026-02-20T11:12:27.719 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:27.719 INFO:teuthology.orchestra.run.trial109.stdout:Transaction Summary 2026-02-20T11:12:27.719 INFO:teuthology.orchestra.run.trial109.stdout:================================================================================ 2026-02-20T11:12:27.719 INFO:teuthology.orchestra.run.trial109.stdout:Install 2 Packages 2026-02-20T11:12:27.719 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:12:27.719 INFO:teuthology.orchestra.run.trial109.stdout:Total download size: 445 M 2026-02-20T11:12:27.719 INFO:teuthology.orchestra.run.trial109.stdout:Installed size: 2.7 G 2026-02-20T11:12:27.719 INFO:teuthology.orchestra.run.trial109.stdout:Downloading Packages: 2026-02-20T11:12:27.791 INFO:teuthology.orchestra.run.trial043.stdout:(1/2): ceph-debugsource-20.3.0-5357.g0c3148ca.e 2.7 MB/s | 12 MB 00:04 2026-02-20T11:12:32.398 INFO:teuthology.orchestra.run.trial109.stdout:(1/2): ceph-debugsource-20.3.0-5357.g0c3148ca.e 2.6 MB/s | 12 MB 00:04 2026-02-20T11:13:21.192 INFO:teuthology.orchestra.run.trial043.stdout:(2/2): ceph-osd-crimson-debuginfo-20.3.0-5357.g 7.5 MB/s | 433 MB 00:57 2026-02-20T11:13:21.193 INFO:teuthology.orchestra.run.trial043.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:13:21.193 INFO:teuthology.orchestra.run.trial043.stdout:Total 7.7 MB/s | 445 MB 00:57 2026-02-20T11:13:21.193 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction check 2026-02-20T11:13:21.198 INFO:teuthology.orchestra.run.trial043.stdout:Transaction check succeeded. 2026-02-20T11:13:21.198 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction test 2026-02-20T11:13:22.569 INFO:teuthology.orchestra.run.trial043.stdout:Transaction test succeeded. 2026-02-20T11:13:22.570 INFO:teuthology.orchestra.run.trial043.stdout:Running transaction 2026-02-20T11:13:26.147 INFO:teuthology.orchestra.run.trial043.stdout: Preparing : 1/1 2026-02-20T11:13:32.102 INFO:teuthology.orchestra.run.trial109.stdout:(2/2): ceph-osd-crimson-debuginfo-20.3.0-5357.g 6.7 MB/s | 433 MB 01:04 2026-02-20T11:13:32.103 INFO:teuthology.orchestra.run.trial109.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:13:32.104 INFO:teuthology.orchestra.run.trial109.stdout:Total 6.9 MB/s | 445 MB 01:04 2026-02-20T11:13:32.104 INFO:teuthology.orchestra.run.trial109.stdout:Running transaction check 2026-02-20T11:13:32.108 INFO:teuthology.orchestra.run.trial109.stdout:Transaction check succeeded. 2026-02-20T11:13:32.108 INFO:teuthology.orchestra.run.trial109.stdout:Running transaction test 2026-02-20T11:13:33.488 INFO:teuthology.orchestra.run.trial109.stdout:Transaction test succeeded. 2026-02-20T11:13:33.488 INFO:teuthology.orchestra.run.trial109.stdout:Running transaction 2026-02-20T11:13:37.062 INFO:teuthology.orchestra.run.trial109.stdout: Preparing : 1/1 2026-02-20T11:13:48.502 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-debugsource-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/2 2026-02-20T11:13:48.519 INFO:teuthology.orchestra.run.trial043.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5357.g0c3148ca.e 2/2 2026-02-20T11:13:50.511 INFO:teuthology.orchestra.run.trial043.stdout: Running scriptlet: ceph-osd-crimson-debuginfo-2:20.3.0-5357.g0c3148ca.e 2/2 2026-02-20T11:13:50.511 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-debugsource-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/2 2026-02-20T11:13:50.570 INFO:teuthology.orchestra.run.trial043.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5357.g0c3148ca.e 2/2 2026-02-20T11:13:50.570 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:13:50.570 INFO:teuthology.orchestra.run.trial043.stdout:Installed: 2026-02-20T11:13:50.570 INFO:teuthology.orchestra.run.trial043.stdout: ceph-debugsource-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:13:50.570 INFO:teuthology.orchestra.run.trial043.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:13:50.570 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:13:50.570 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:13:50.641 DEBUG:teuthology.parallel:result is None 2026-02-20T11:13:59.299 INFO:teuthology.orchestra.run.trial109.stdout: Installing : ceph-debugsource-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/2 2026-02-20T11:13:59.316 INFO:teuthology.orchestra.run.trial109.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5357.g0c3148ca.e 2/2 2026-02-20T11:14:01.310 INFO:teuthology.orchestra.run.trial109.stdout: Running scriptlet: ceph-osd-crimson-debuginfo-2:20.3.0-5357.g0c3148ca.e 2/2 2026-02-20T11:14:01.310 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : ceph-debugsource-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/2 2026-02-20T11:14:01.370 INFO:teuthology.orchestra.run.trial109.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5357.g0c3148ca.e 2/2 2026-02-20T11:14:01.370 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:14:01.370 INFO:teuthology.orchestra.run.trial109.stdout:Installed: 2026-02-20T11:14:01.370 INFO:teuthology.orchestra.run.trial109.stdout: ceph-debugsource-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:14:01.370 INFO:teuthology.orchestra.run.trial109.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:14:01.370 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:14:01.370 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:14:01.446 DEBUG:teuthology.parallel:result is None 2026-02-20T11:14:01.447 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=0c3148ca19c827ca2a7a9b92c2554d99507fd1f4 2026-02-20T11:14:01.574 DEBUG:teuthology.orchestra.run.trial043:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-20T11:14:01.595 INFO:teuthology.orchestra.run.trial043.stdout:20.3.0-5357.g0c3148ca.el9 2026-02-20T11:14:01.595 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5357.g0c3148ca.el9 2026-02-20T11:14:01.595 INFO:teuthology.task.install:The correct ceph version 20.3.0-5357.g0c3148ca is installed. 2026-02-20T11:14:01.597 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=0c3148ca19c827ca2a7a9b92c2554d99507fd1f4 2026-02-20T11:14:01.737 DEBUG:teuthology.orchestra.run.trial109:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-20T11:14:01.759 INFO:teuthology.orchestra.run.trial109.stdout:20.3.0-5357.g0c3148ca.el9 2026-02-20T11:14:01.759 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5357.g0c3148ca.el9 2026-02-20T11:14:01.759 INFO:teuthology.task.install:The correct ceph version 20.3.0-5357.g0c3148ca is installed. 2026-02-20T11:14:01.761 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-20T11:14:01.761 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-20T11:14:01.761 DEBUG:teuthology.orchestra.run.trial043:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-20T11:14:01.788 DEBUG:teuthology.orchestra.run.trial109:> set -ex 2026-02-20T11:14:01.788 DEBUG:teuthology.orchestra.run.trial109:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-20T11:14:01.825 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-20T11:14:01.825 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-20T11:14:01.825 DEBUG:teuthology.orchestra.run.trial043:> sudo dd of=/usr/bin/daemon-helper 2026-02-20T11:14:01.853 DEBUG:teuthology.orchestra.run.trial043:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-20T11:14:01.918 DEBUG:teuthology.orchestra.run.trial109:> set -ex 2026-02-20T11:14:01.918 DEBUG:teuthology.orchestra.run.trial109:> sudo dd of=/usr/bin/daemon-helper 2026-02-20T11:14:01.945 DEBUG:teuthology.orchestra.run.trial109:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-20T11:14:02.010 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-20T11:14:02.010 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-20T11:14:02.010 DEBUG:teuthology.orchestra.run.trial043:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-20T11:14:02.036 DEBUG:teuthology.orchestra.run.trial043:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-20T11:14:02.098 DEBUG:teuthology.orchestra.run.trial109:> set -ex 2026-02-20T11:14:02.098 DEBUG:teuthology.orchestra.run.trial109:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-20T11:14:02.123 DEBUG:teuthology.orchestra.run.trial109:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-20T11:14:02.187 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-20T11:14:02.188 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-20T11:14:02.188 DEBUG:teuthology.orchestra.run.trial043:> sudo dd of=/usr/bin/stdin-killer 2026-02-20T11:14:02.213 DEBUG:teuthology.orchestra.run.trial043:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-20T11:14:02.276 DEBUG:teuthology.orchestra.run.trial109:> set -ex 2026-02-20T11:14:02.276 DEBUG:teuthology.orchestra.run.trial109:> sudo dd of=/usr/bin/stdin-killer 2026-02-20T11:14:02.301 DEBUG:teuthology.orchestra.run.trial109:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-20T11:14:02.365 INFO:teuthology.run_tasks:Running task pexec... 2026-02-20T11:14:02.382 INFO:teuthology.task.pexec:Executing custom commands... 2026-02-20T11:14:02.382 DEBUG:teuthology.orchestra.run.trial043:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-20T11:14:02.383 DEBUG:teuthology.orchestra.run.trial109:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-20T11:14:02.386 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial043.front.sepia.ceph.com 2026-02-20T11:14:02.386 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-20T11:14:02.407 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial109.front.sepia.ceph.com 2026-02-20T11:14:02.407 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-20T11:14:02.436 DEBUG:teuthology.parallel:result is None 2026-02-20T11:14:02.457 DEBUG:teuthology.parallel:result is None 2026-02-20T11:14:02.457 INFO:teuthology.run_tasks:Running task install... 2026-02-20T11:14:02.468 DEBUG:teuthology.task.install:project ceph 2026-02-20T11:14:02.468 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': '0c3148ca19c827ca2a7a9b92c2554d99507fd1f4'}} 2026-02-20T11:14:02.468 DEBUG:teuthology.task.install:config {'flavor': 'debug', 'sha1': '0c3148ca19c827ca2a7a9b92c2554d99507fd1f4'} 2026-02-20T11:14:02.469 INFO:teuthology.task.install:Using flavor: debug 2026-02-20T11:14:02.472 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-20T11:14:02.473 INFO:teuthology.task.install:extra packages: [] 2026-02-20T11:14:02.473 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0c3148ca19c827ca2a7a9b92c2554d99507fd1f4', 'tag': None, 'wait_for_package': False} 2026-02-20T11:14:02.473 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=0c3148ca19c827ca2a7a9b92c2554d99507fd1f4 2026-02-20T11:14:02.475 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0c3148ca19c827ca2a7a9b92c2554d99507fd1f4', 'tag': None, 'wait_for_package': False} 2026-02-20T11:14:02.475 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=0c3148ca19c827ca2a7a9b92c2554d99507fd1f4 2026-02-20T11:14:02.661 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-19-02-2026-PR64439/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/centos/9/flavors/debug/ 2026-02-20T11:14:02.662 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5357.g0c3148ca 2026-02-20T11:14:02.671 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-19-02-2026-PR64439/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/centos/9/flavors/debug/ 2026-02-20T11:14:02.672 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5357.g0c3148ca 2026-02-20T11:14:02.831 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-19-02-2026-PR64439/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-19-02-2026-PR64439/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-19-02-2026-PR64439/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-20T11:14:02.832 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-20T11:14:02.832 DEBUG:teuthology.orchestra.run.trial043:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-20T11:14:02.840 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-19-02-2026-PR64439/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-19-02-2026-PR64439/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-19-02-2026-PR64439/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-20T11:14:02.840 DEBUG:teuthology.orchestra.run.trial109:> set -ex 2026-02-20T11:14:02.840 DEBUG:teuthology.orchestra.run.trial109:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-20T11:14:02.855 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-20T11:14:02.855 DEBUG:teuthology.orchestra.run.trial043:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-20T11:14:02.864 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-20T11:14:02.864 DEBUG:teuthology.orchestra.run.trial109:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-20T11:14:02.922 DEBUG:teuthology.orchestra.run.trial043:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-20T11:14:02.929 DEBUG:teuthology.orchestra.run.trial109:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-20T11:14:02.986 DEBUG:teuthology.orchestra.run.trial043:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-20T11:14:02.992 DEBUG:teuthology.orchestra.run.trial109:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-20T11:14:03.042 INFO:teuthology.orchestra.run.trial043.stdout:check_obsoletes = 1 2026-02-20T11:14:03.049 INFO:teuthology.orchestra.run.trial109.stdout:check_obsoletes = 1 2026-02-20T11:14:03.052 DEBUG:teuthology.orchestra.run.trial043:> sudo yum clean all 2026-02-20T11:14:03.058 DEBUG:teuthology.orchestra.run.trial109:> sudo yum clean all 2026-02-20T11:14:03.211 INFO:teuthology.orchestra.run.trial043.stdout:56 files removed 2026-02-20T11:14:03.219 INFO:teuthology.orchestra.run.trial109.stdout:56 files removed 2026-02-20T11:14:03.226 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-radosgw 2026-02-20T11:14:03.234 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install ceph-radosgw 2026-02-20T11:14:03.518 INFO:teuthology.orchestra.run.trial109.stdout:ceph packages for x86_64 704 kB/s | 94 kB 00:00 2026-02-20T11:14:03.519 INFO:teuthology.orchestra.run.trial043.stdout:ceph packages for x86_64 655 kB/s | 94 kB 00:00 2026-02-20T11:14:03.672 INFO:teuthology.orchestra.run.trial109.stdout:ceph noarch packages 196 kB/s | 24 kB 00:00 2026-02-20T11:14:03.804 INFO:teuthology.orchestra.run.trial109.stdout:ceph source packages 19 kB/s | 2.1 kB 00:00 2026-02-20T11:14:05.318 INFO:teuthology.orchestra.run.trial109.stdout:CentOS Stream 9 - BaseOS 6.0 MB/s | 8.9 MB 00:01 2026-02-20T11:14:08.621 INFO:teuthology.orchestra.run.trial109.stdout:CentOS Stream 9 - AppStream 9.7 MB/s | 27 MB 00:02 2026-02-20T11:14:09.963 INFO:teuthology.orchestra.run.trial043.stdout:ceph noarch packages 3.8 kB/s | 24 kB 00:06 2026-02-20T11:14:10.499 INFO:teuthology.orchestra.run.trial043.stdout:ceph source packages 4.0 kB/s | 2.1 kB 00:00 2026-02-20T11:14:12.640 INFO:teuthology.orchestra.run.trial043.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.9 MB 00:02 2026-02-20T11:14:12.709 INFO:teuthology.orchestra.run.trial109.stdout:CentOS Stream 9 - CRB 4.5 MB/s | 7.8 MB 00:01 2026-02-20T11:14:13.597 INFO:teuthology.orchestra.run.trial109.stdout:CentOS Stream 9 - Extras packages 129 kB/s | 20 kB 00:00 2026-02-20T11:14:13.956 INFO:teuthology.orchestra.run.trial109.stdout:Extra Packages for Enterprise Linux 68 MB/s | 20 MB 00:00 2026-02-20T11:14:15.743 INFO:teuthology.orchestra.run.trial043.stdout:CentOS Stream 9 - AppStream 10 MB/s | 27 MB 00:02 2026-02-20T11:14:17.172 INFO:teuthology.orchestra.run.trial109.stdout:lab-extras 2.6 MB/s | 50 kB 00:00 2026-02-20T11:14:18.342 INFO:teuthology.orchestra.run.trial109.stdout:Package ceph-radosgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:18.361 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:14:18.362 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:14:18.362 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:14:18.386 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install ceph-test 2026-02-20T11:14:18.689 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:00:01 ago on Fri 20 Feb 2026 11:14:17 AM UTC. 2026-02-20T11:14:18.754 INFO:teuthology.orchestra.run.trial109.stdout:Package ceph-test-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:18.772 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:14:18.773 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:14:18.773 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:14:18.794 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install ceph 2026-02-20T11:14:19.108 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:00:02 ago on Fri 20 Feb 2026 11:14:17 AM UTC. 2026-02-20T11:14:19.174 INFO:teuthology.orchestra.run.trial109.stdout:Package ceph-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:19.192 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:14:19.193 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:14:19.193 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:14:19.215 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install ceph-base 2026-02-20T11:14:19.523 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:00:02 ago on Fri 20 Feb 2026 11:14:17 AM UTC. 2026-02-20T11:14:19.587 INFO:teuthology.orchestra.run.trial109.stdout:Package ceph-base-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:19.606 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:14:19.606 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:14:19.606 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:14:19.627 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install cephadm 2026-02-20T11:14:19.936 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:00:02 ago on Fri 20 Feb 2026 11:14:17 AM UTC. 2026-02-20T11:14:20.001 INFO:teuthology.orchestra.run.trial109.stdout:Package cephadm-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:14:20.019 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:14:20.020 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:14:20.020 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:14:20.042 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install ceph-immutable-object-cache 2026-02-20T11:14:20.120 INFO:teuthology.orchestra.run.trial043.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 7.8 MB 00:01 2026-02-20T11:14:20.353 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:00:03 ago on Fri 20 Feb 2026 11:14:17 AM UTC. 2026-02-20T11:14:20.418 INFO:teuthology.orchestra.run.trial109.stdout:Package ceph-immutable-object-cache-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:20.436 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:14:20.437 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:14:20.437 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:14:20.459 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install ceph-mgr 2026-02-20T11:14:20.775 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:00:03 ago on Fri 20 Feb 2026 11:14:17 AM UTC. 2026-02-20T11:14:20.840 INFO:teuthology.orchestra.run.trial109.stdout:Package ceph-mgr-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:20.858 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:14:20.859 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:14:20.859 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:14:20.881 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install ceph-mgr-dashboard 2026-02-20T11:14:21.046 INFO:teuthology.orchestra.run.trial043.stdout:CentOS Stream 9 - Extras packages 108 kB/s | 20 kB 00:00 2026-02-20T11:14:21.192 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:00:04 ago on Fri 20 Feb 2026 11:14:17 AM UTC. 2026-02-20T11:14:21.256 INFO:teuthology.orchestra.run.trial109.stdout:Package ceph-mgr-dashboard-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:14:21.275 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:14:21.276 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:14:21.276 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:14:21.298 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-20T11:14:21.387 INFO:teuthology.orchestra.run.trial043.stdout:Extra Packages for Enterprise Linux 74 MB/s | 20 MB 00:00 2026-02-20T11:14:21.609 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:00:04 ago on Fri 20 Feb 2026 11:14:17 AM UTC. 2026-02-20T11:14:21.673 INFO:teuthology.orchestra.run.trial109.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:14:21.692 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:14:21.692 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:14:21.693 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:14:21.714 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install ceph-mgr-rook 2026-02-20T11:14:22.025 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:00:05 ago on Fri 20 Feb 2026 11:14:17 AM UTC. 2026-02-20T11:14:22.090 INFO:teuthology.orchestra.run.trial109.stdout:Package ceph-mgr-rook-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:14:22.109 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:14:22.109 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:14:22.109 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:14:22.132 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install ceph-mgr-cephadm 2026-02-20T11:14:22.442 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:00:05 ago on Fri 20 Feb 2026 11:14:17 AM UTC. 2026-02-20T11:14:22.507 INFO:teuthology.orchestra.run.trial109.stdout:Package ceph-mgr-cephadm-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:14:22.525 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:14:22.526 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:14:22.526 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:14:22.548 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install ceph-osd 2026-02-20T11:14:22.854 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:00:05 ago on Fri 20 Feb 2026 11:14:17 AM UTC. 2026-02-20T11:14:22.919 INFO:teuthology.orchestra.run.trial109.stdout:Package ceph-osd-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:22.937 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:14:22.938 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:14:22.938 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:14:22.960 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install ceph-osd-classic 2026-02-20T11:14:23.271 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:00:06 ago on Fri 20 Feb 2026 11:14:17 AM UTC. 2026-02-20T11:14:23.336 INFO:teuthology.orchestra.run.trial109.stdout:Package ceph-osd-classic-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:23.355 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:14:23.355 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:14:23.356 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:14:23.378 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install ceph-fuse 2026-02-20T11:14:23.687 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:00:06 ago on Fri 20 Feb 2026 11:14:17 AM UTC. 2026-02-20T11:14:23.754 INFO:teuthology.orchestra.run.trial109.stdout:Package ceph-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:23.772 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:14:23.773 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:14:23.773 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:14:23.794 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install ceph-volume 2026-02-20T11:14:24.108 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:00:07 ago on Fri 20 Feb 2026 11:14:17 AM UTC. 2026-02-20T11:14:24.173 INFO:teuthology.orchestra.run.trial109.stdout:Package ceph-volume-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:14:24.191 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:14:24.192 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:14:24.192 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:14:24.215 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install librados-devel 2026-02-20T11:14:24.529 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:00:07 ago on Fri 20 Feb 2026 11:14:17 AM UTC. 2026-02-20T11:14:24.574 INFO:teuthology.orchestra.run.trial043.stdout:lab-extras 2.7 MB/s | 50 kB 00:00 2026-02-20T11:14:24.594 INFO:teuthology.orchestra.run.trial109.stdout:Package librados-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:24.612 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:14:24.613 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:14:24.613 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:14:24.635 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install libcephfs2 2026-02-20T11:14:24.947 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:00:07 ago on Fri 20 Feb 2026 11:14:17 AM UTC. 2026-02-20T11:14:25.012 INFO:teuthology.orchestra.run.trial109.stdout:Package libcephfs2-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:25.030 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:14:25.031 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:14:25.031 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:14:25.053 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install libcephfs-devel 2026-02-20T11:14:25.363 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:00:08 ago on Fri 20 Feb 2026 11:14:17 AM UTC. 2026-02-20T11:14:25.428 INFO:teuthology.orchestra.run.trial109.stdout:Package libcephfs-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:25.446 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:14:25.447 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:14:25.447 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:14:25.468 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install librados2 2026-02-20T11:14:25.761 INFO:teuthology.orchestra.run.trial043.stdout:Package ceph-radosgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:25.776 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:00:08 ago on Fri 20 Feb 2026 11:14:17 AM UTC. 2026-02-20T11:14:25.780 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:14:25.781 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:14:25.781 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:14:25.803 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-test 2026-02-20T11:14:25.841 INFO:teuthology.orchestra.run.trial109.stdout:Package librados2-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:25.859 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:14:25.860 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:14:25.860 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:14:25.883 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install librbd1 2026-02-20T11:14:26.105 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:02 ago on Fri 20 Feb 2026 11:14:24 AM UTC. 2026-02-20T11:14:26.170 INFO:teuthology.orchestra.run.trial043.stdout:Package ceph-test-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:26.192 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:00:09 ago on Fri 20 Feb 2026 11:14:17 AM UTC. 2026-02-20T11:14:26.193 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:14:26.193 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:14:26.193 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:14:26.214 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph 2026-02-20T11:14:26.257 INFO:teuthology.orchestra.run.trial109.stdout:Package librbd1-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:26.275 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:14:26.276 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:14:26.276 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:14:26.299 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install python3-rados 2026-02-20T11:14:26.522 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:02 ago on Fri 20 Feb 2026 11:14:24 AM UTC. 2026-02-20T11:14:26.586 INFO:teuthology.orchestra.run.trial043.stdout:Package ceph-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:26.605 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:14:26.605 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:14:26.605 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:14:26.616 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:00:09 ago on Fri 20 Feb 2026 11:14:17 AM UTC. 2026-02-20T11:14:26.626 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-base 2026-02-20T11:14:26.680 INFO:teuthology.orchestra.run.trial109.stdout:Package python3-rados-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:26.699 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:14:26.700 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:14:26.700 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:14:26.722 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install python3-rgw 2026-02-20T11:14:26.932 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:02 ago on Fri 20 Feb 2026 11:14:24 AM UTC. 2026-02-20T11:14:26.996 INFO:teuthology.orchestra.run.trial043.stdout:Package ceph-base-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:27.015 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:14:27.016 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:14:27.016 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:14:27.028 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:00:10 ago on Fri 20 Feb 2026 11:14:17 AM UTC. 2026-02-20T11:14:27.036 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install cephadm 2026-02-20T11:14:27.093 INFO:teuthology.orchestra.run.trial109.stdout:Package python3-rgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:27.112 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:14:27.113 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:14:27.113 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:14:27.135 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install python3-cephfs 2026-02-20T11:14:27.343 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:03 ago on Fri 20 Feb 2026 11:14:24 AM UTC. 2026-02-20T11:14:27.407 INFO:teuthology.orchestra.run.trial043.stdout:Package cephadm-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:14:27.426 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:14:27.426 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:14:27.426 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:14:27.442 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:00:10 ago on Fri 20 Feb 2026 11:14:17 AM UTC. 2026-02-20T11:14:27.447 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-immutable-object-cache 2026-02-20T11:14:27.507 INFO:teuthology.orchestra.run.trial109.stdout:Package python3-cephfs-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:27.526 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:14:27.526 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:14:27.526 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:14:27.549 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install python3-rbd 2026-02-20T11:14:27.759 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:03 ago on Fri 20 Feb 2026 11:14:24 AM UTC. 2026-02-20T11:14:27.823 INFO:teuthology.orchestra.run.trial043.stdout:Package ceph-immutable-object-cache-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:27.842 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:14:27.842 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:14:27.843 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:14:27.860 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:00:10 ago on Fri 20 Feb 2026 11:14:17 AM UTC. 2026-02-20T11:14:27.864 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-mgr 2026-02-20T11:14:27.925 INFO:teuthology.orchestra.run.trial109.stdout:Package python3-rbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:27.944 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:14:27.945 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:14:27.945 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:14:27.967 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install rbd-fuse 2026-02-20T11:14:28.173 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:04 ago on Fri 20 Feb 2026 11:14:24 AM UTC. 2026-02-20T11:14:28.237 INFO:teuthology.orchestra.run.trial043.stdout:Package ceph-mgr-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:28.255 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:14:28.256 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:14:28.256 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:14:28.277 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-mgr-dashboard 2026-02-20T11:14:28.281 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:00:11 ago on Fri 20 Feb 2026 11:14:17 AM UTC. 2026-02-20T11:14:28.346 INFO:teuthology.orchestra.run.trial109.stdout:Package rbd-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:28.364 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:14:28.365 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:14:28.365 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:14:28.388 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install rbd-mirror 2026-02-20T11:14:28.584 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:04 ago on Fri 20 Feb 2026 11:14:24 AM UTC. 2026-02-20T11:14:28.648 INFO:teuthology.orchestra.run.trial043.stdout:Package ceph-mgr-dashboard-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:14:28.666 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:14:28.667 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:14:28.667 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:14:28.688 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-20T11:14:28.699 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:00:11 ago on Fri 20 Feb 2026 11:14:17 AM UTC. 2026-02-20T11:14:28.766 INFO:teuthology.orchestra.run.trial109.stdout:Package rbd-mirror-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:28.785 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:14:28.785 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:14:28.786 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:14:28.808 DEBUG:teuthology.orchestra.run.trial109:> sudo yum -y install rbd-nbd 2026-02-20T11:14:28.995 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:04 ago on Fri 20 Feb 2026 11:14:24 AM UTC. 2026-02-20T11:14:29.061 INFO:teuthology.orchestra.run.trial043.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:14:29.079 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:14:29.080 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:14:29.080 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:14:29.101 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-mgr-rook 2026-02-20T11:14:29.120 INFO:teuthology.orchestra.run.trial109.stdout:Last metadata expiration check: 0:00:12 ago on Fri 20 Feb 2026 11:14:17 AM UTC. 2026-02-20T11:14:29.186 INFO:teuthology.orchestra.run.trial109.stdout:Package rbd-nbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:29.204 INFO:teuthology.orchestra.run.trial109.stdout:Dependencies resolved. 2026-02-20T11:14:29.205 INFO:teuthology.orchestra.run.trial109.stdout:Nothing to do. 2026-02-20T11:14:29.205 INFO:teuthology.orchestra.run.trial109.stdout:Complete! 2026-02-20T11:14:29.231 DEBUG:teuthology.parallel:result is None 2026-02-20T11:14:29.410 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:05 ago on Fri 20 Feb 2026 11:14:24 AM UTC. 2026-02-20T11:14:29.475 INFO:teuthology.orchestra.run.trial043.stdout:Package ceph-mgr-rook-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:14:29.493 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:14:29.494 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:14:29.494 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:14:29.515 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-mgr-cephadm 2026-02-20T11:14:29.824 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:05 ago on Fri 20 Feb 2026 11:14:24 AM UTC. 2026-02-20T11:14:29.889 INFO:teuthology.orchestra.run.trial043.stdout:Package ceph-mgr-cephadm-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:14:29.908 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:14:29.908 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:14:29.908 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:14:29.930 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-osd 2026-02-20T11:14:30.235 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:06 ago on Fri 20 Feb 2026 11:14:24 AM UTC. 2026-02-20T11:14:30.299 INFO:teuthology.orchestra.run.trial043.stdout:Package ceph-osd-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:30.317 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:14:30.318 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:14:30.318 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:14:30.339 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-osd-classic 2026-02-20T11:14:30.648 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:06 ago on Fri 20 Feb 2026 11:14:24 AM UTC. 2026-02-20T11:14:30.713 INFO:teuthology.orchestra.run.trial043.stdout:Package ceph-osd-classic-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:30.732 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:14:30.732 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:14:30.732 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:14:30.754 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-fuse 2026-02-20T11:14:31.062 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:07 ago on Fri 20 Feb 2026 11:14:24 AM UTC. 2026-02-20T11:14:31.126 INFO:teuthology.orchestra.run.trial043.stdout:Package ceph-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:31.145 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:14:31.145 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:14:31.146 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:14:31.167 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install ceph-volume 2026-02-20T11:14:31.477 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:07 ago on Fri 20 Feb 2026 11:14:24 AM UTC. 2026-02-20T11:14:31.542 INFO:teuthology.orchestra.run.trial043.stdout:Package ceph-volume-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:14:31.561 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:14:31.561 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:14:31.561 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:14:31.583 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install librados-devel 2026-02-20T11:14:31.896 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:07 ago on Fri 20 Feb 2026 11:14:24 AM UTC. 2026-02-20T11:14:31.960 INFO:teuthology.orchestra.run.trial043.stdout:Package librados-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:31.979 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:14:31.980 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:14:31.980 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:14:32.001 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install libcephfs2 2026-02-20T11:14:32.313 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:08 ago on Fri 20 Feb 2026 11:14:24 AM UTC. 2026-02-20T11:14:32.377 INFO:teuthology.orchestra.run.trial043.stdout:Package libcephfs2-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:32.396 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:14:32.397 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:14:32.397 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:14:32.418 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install libcephfs-devel 2026-02-20T11:14:32.731 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:08 ago on Fri 20 Feb 2026 11:14:24 AM UTC. 2026-02-20T11:14:32.796 INFO:teuthology.orchestra.run.trial043.stdout:Package libcephfs-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:32.815 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:14:32.816 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:14:32.816 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:14:32.838 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install librados2 2026-02-20T11:14:33.150 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:09 ago on Fri 20 Feb 2026 11:14:24 AM UTC. 2026-02-20T11:14:33.215 INFO:teuthology.orchestra.run.trial043.stdout:Package librados2-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:33.234 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:14:33.235 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:14:33.235 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:14:33.255 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install librbd1 2026-02-20T11:14:33.564 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:09 ago on Fri 20 Feb 2026 11:14:24 AM UTC. 2026-02-20T11:14:33.628 INFO:teuthology.orchestra.run.trial043.stdout:Package librbd1-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:33.647 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:14:33.647 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:14:33.647 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:14:33.667 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install python3-rados 2026-02-20T11:14:33.977 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:09 ago on Fri 20 Feb 2026 11:14:24 AM UTC. 2026-02-20T11:14:34.041 INFO:teuthology.orchestra.run.trial043.stdout:Package python3-rados-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:34.060 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:14:34.061 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:14:34.061 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:14:34.082 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install python3-rgw 2026-02-20T11:14:34.392 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:10 ago on Fri 20 Feb 2026 11:14:24 AM UTC. 2026-02-20T11:14:34.456 INFO:teuthology.orchestra.run.trial043.stdout:Package python3-rgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:34.475 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:14:34.475 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:14:34.476 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:14:34.497 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install python3-cephfs 2026-02-20T11:14:34.805 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:10 ago on Fri 20 Feb 2026 11:14:24 AM UTC. 2026-02-20T11:14:34.869 INFO:teuthology.orchestra.run.trial043.stdout:Package python3-cephfs-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:34.888 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:14:34.889 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:14:34.889 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:14:34.910 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install python3-rbd 2026-02-20T11:14:35.218 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:11 ago on Fri 20 Feb 2026 11:14:24 AM UTC. 2026-02-20T11:14:35.283 INFO:teuthology.orchestra.run.trial043.stdout:Package python3-rbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:35.302 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:14:35.302 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:14:35.303 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:14:35.324 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install rbd-fuse 2026-02-20T11:14:35.630 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:11 ago on Fri 20 Feb 2026 11:14:24 AM UTC. 2026-02-20T11:14:35.694 INFO:teuthology.orchestra.run.trial043.stdout:Package rbd-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:35.713 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:14:35.714 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:14:35.714 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:14:35.735 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install rbd-mirror 2026-02-20T11:14:36.046 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:12 ago on Fri 20 Feb 2026 11:14:24 AM UTC. 2026-02-20T11:14:36.111 INFO:teuthology.orchestra.run.trial043.stdout:Package rbd-mirror-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:36.129 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:14:36.130 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:14:36.130 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:14:36.152 DEBUG:teuthology.orchestra.run.trial043:> sudo yum -y install rbd-nbd 2026-02-20T11:14:36.458 INFO:teuthology.orchestra.run.trial043.stdout:Last metadata expiration check: 0:00:12 ago on Fri 20 Feb 2026 11:14:24 AM UTC. 2026-02-20T11:14:36.522 INFO:teuthology.orchestra.run.trial043.stdout:Package rbd-nbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:14:36.541 INFO:teuthology.orchestra.run.trial043.stdout:Dependencies resolved. 2026-02-20T11:14:36.542 INFO:teuthology.orchestra.run.trial043.stdout:Nothing to do. 2026-02-20T11:14:36.542 INFO:teuthology.orchestra.run.trial043.stdout:Complete! 2026-02-20T11:14:36.566 DEBUG:teuthology.parallel:result is None 2026-02-20T11:14:36.566 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=0c3148ca19c827ca2a7a9b92c2554d99507fd1f4 2026-02-20T11:14:36.731 DEBUG:teuthology.orchestra.run.trial043:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-20T11:14:36.753 INFO:teuthology.orchestra.run.trial043.stdout:20.3.0-5357.g0c3148ca.el9 2026-02-20T11:14:36.753 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5357.g0c3148ca.el9 2026-02-20T11:14:36.753 INFO:teuthology.task.install:The correct ceph version 20.3.0-5357.g0c3148ca is installed. 2026-02-20T11:14:36.756 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=0c3148ca19c827ca2a7a9b92c2554d99507fd1f4 2026-02-20T11:14:36.892 DEBUG:teuthology.orchestra.run.trial109:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-20T11:14:36.912 INFO:teuthology.orchestra.run.trial109.stdout:20.3.0-5357.g0c3148ca.el9 2026-02-20T11:14:36.913 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5357.g0c3148ca.el9 2026-02-20T11:14:36.913 INFO:teuthology.task.install:The correct ceph version 20.3.0-5357.g0c3148ca is installed. 2026-02-20T11:14:36.915 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2026-02-20T11:14:36.915 INFO:teuthology.run_tasks:Running task ceph... 2026-02-20T11:14:37.012 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-20T11:14:37.012 DEBUG:teuthology.orchestra.run.trial043:> sudo chmod 777 /var/log/ceph 2026-02-20T11:14:37.014 DEBUG:teuthology.orchestra.run.trial109:> sudo chmod 777 /var/log/ceph 2026-02-20T11:14:37.039 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-20T11:14:37.039 DEBUG:teuthology.orchestra.run.trial043:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-20T11:14:37.079 DEBUG:teuthology.orchestra.run.trial109:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-20T11:14:37.102 INFO:tasks.ceph:Creating extra log directories... 2026-02-20T11:14:37.102 DEBUG:teuthology.orchestra.run.trial043:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-20T11:14:37.143 DEBUG:teuthology.orchestra.run.trial109:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-20T11:14:37.182 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-20T11:14:37.182 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd target transaction size': 5, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Degraded data redundancy', 'is down', 'osds down', 'pg .*? is .*?degraded.*?, acting', 'pg .*? is stuck', 'pg degraded', 'PG_BACKFILL_FULL', 'Low space hindering backfill .*? backfill_toofull', 'OSD_ROOT_DOWN', 'pgs degraded', 'pgs undersized', 'is active.*backfill_toofull.*'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-20T11:14:37.182 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/jjperez-2026-02-20_10:21:31-crimson-rados-wip-perezjos-crimson-only-19-02-2026-PR64439-distro-debug-trial/60477', 'branch': 'wip-perezjos-crimson-only-19-02-2026-PR64439', 'description': 'crimson-rados/thrash/{0-crimson_install 0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} clusters/{crimson-fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm} thrashers/{default} thrashosds-health workloads/pool-snaps-few-objects}', 'email': None, 'first_in_suite': False, 'flavor': 'debug', 'job_id': '60477', 'kernel': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'trial', 'name': 'jjperez-2026-02-20_10:21:31-crimson-rados-wip-perezjos-crimson-only-19-02-2026-PR64439-distro-debug-trial', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-perezjos-crimson-only-19-02-2026-PR64439'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd target transaction size': 5, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Degraded data redundancy', 'is down', 'osds down', 'pg .*? is .*?degraded.*?, acting', 'pg .*? is stuck', 'pg degraded', 'PG_BACKFILL_FULL', 'Low space hindering backfill .*? backfill_toofull', 'OSD_ROOT_DOWN', 'pgs degraded', 'pgs undersized', 'is active.*backfill_toofull.*'], 'sha1': '0c3148ca19c827ca2a7a9b92c2554d99507fd1f4'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'conf': {'osd': {'osd deep scrub update digest min age': 0}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': '0c3148ca19c827ca2a7a9b92c2554d99507fd1f4'}}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '445d5c9ba18a4040e70e25303230c36c77773acb'}}, 'owner': 'scheduled_jjperez@soko04.front.sepia.ceph.com', 'priority': 101, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'], ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 1736, 'sha1': '0c3148ca19c827ca2a7a9b92c2554d99507fd1f4', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': '445d5c9ba18a4040e70e25303230c36c77773acb', 'targets': {'trial043.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC9KROEZxDEnRzwsYioaSBZAaVX82KUX6mpdTEBGlCxJ1/oKHlwjv0iX6bqUNS8vM7byIN+eIeisLxGilvuexMo=', 'trial109.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMbRy1ZZTQoDxKIW3KAcBEW3M+EMItJWa8K9r3kapqYEYab49qlHZdbYtkkIjSY3EjWrQ0PL7DjwuV79dEIDcOE='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': '0c3148ca19c827ca2a7a9b92c2554d99507fd1f4'}}, {'pexec': {'all': ['sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson']}}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd target transaction size': 5, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Degraded data redundancy', 'is down', 'osds down', 'pg .*? is .*?degraded.*?, acting', 'pg .*? is stuck', 'pg degraded', 'PG_BACKFILL_FULL', 'Low space hindering backfill .*? backfill_toofull', 'OSD_ROOT_DOWN', 'pgs degraded', 'pgs undersized', 'is active.*backfill_toofull.*'], 'sha1': '0c3148ca19c827ca2a7a9b92c2554d99507fd1f4', 'cluster': 'ceph'}}, {'thrashosds': {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_trim_stale_osdmaps': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400}}, {'rados': {'clients': ['client.0'], 'objects': 50, 'op_weights': {'copy_from': 0, 'delete': 50, 'read': 100, 'rollback': 0, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000, 'pool_snaps': True}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'reimage-ssh-down', 'teuthology_sha1': 'e6d9099ce03dc21193fda653159ec82465c3a94a', 'timestamp': '2026-02-20_10:21:31', 'tube': 'trial', 'user': 'jjperez', 'verbose': False, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.1540699'} 2026-02-20T11:14:37.183 DEBUG:teuthology.orchestra.run.trial043:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-20T11:14:37.209 DEBUG:teuthology.orchestra.run.trial109:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-20T11:14:37.225 DEBUG:teuthology.orchestra.run.trial043:> sudo install -d -m0777 -- /var/run/ceph 2026-02-20T11:14:37.266 DEBUG:teuthology.orchestra.run.trial109:> sudo install -d -m0777 -- /var/run/ceph 2026-02-20T11:14:37.292 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-20T11:14:37.293 DEBUG:teuthology.orchestra.run.trial043:> dd if=/scratch_devs of=/dev/stdout 2026-02-20T11:14:37.347 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-20T11:14:37.347 DEBUG:teuthology.orchestra.run.trial043:> stat /dev/vg_nvme/lv_1 2026-02-20T11:14:37.404 INFO:teuthology.orchestra.run.trial043.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-20T11:14:37.405 INFO:teuthology.orchestra.run.trial043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-20T11:14:37.405 INFO:teuthology.orchestra.run.trial043.stdout:Device: 6h/6d Inode: 1312 Links: 1 2026-02-20T11:14:37.405 INFO:teuthology.orchestra.run.trial043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-20T11:14:37.405 INFO:teuthology.orchestra.run.trial043.stdout:Context: system_u:object_r:device_t:s0 2026-02-20T11:14:37.405 INFO:teuthology.orchestra.run.trial043.stdout:Access: 2026-02-20 11:12:21.127785337 +0000 2026-02-20T11:14:37.405 INFO:teuthology.orchestra.run.trial043.stdout:Modify: 2026-02-20 11:11:03.840068166 +0000 2026-02-20T11:14:37.405 INFO:teuthology.orchestra.run.trial043.stdout:Change: 2026-02-20 11:11:03.840068166 +0000 2026-02-20T11:14:37.405 INFO:teuthology.orchestra.run.trial043.stdout: Birth: 2026-02-20 11:11:03.840068166 +0000 2026-02-20T11:14:37.406 DEBUG:teuthology.orchestra.run.trial043:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-20T11:14:37.469 INFO:teuthology.orchestra.run.trial043.stderr:1+0 records in 2026-02-20T11:14:37.469 INFO:teuthology.orchestra.run.trial043.stderr:1+0 records out 2026-02-20T11:14:37.469 INFO:teuthology.orchestra.run.trial043.stderr:512 bytes copied, 0.000158103 s, 3.2 MB/s 2026-02-20T11:14:37.470 DEBUG:teuthology.orchestra.run.trial043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-20T11:14:37.528 DEBUG:teuthology.orchestra.run.trial043:> stat /dev/vg_nvme/lv_2 2026-02-20T11:14:37.585 INFO:teuthology.orchestra.run.trial043.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-20T11:14:37.585 INFO:teuthology.orchestra.run.trial043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-20T11:14:37.585 INFO:teuthology.orchestra.run.trial043.stdout:Device: 6h/6d Inode: 1308 Links: 1 2026-02-20T11:14:37.585 INFO:teuthology.orchestra.run.trial043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-20T11:14:37.585 INFO:teuthology.orchestra.run.trial043.stdout:Context: system_u:object_r:device_t:s0 2026-02-20T11:14:37.586 INFO:teuthology.orchestra.run.trial043.stdout:Access: 2026-02-20 11:12:21.127785337 +0000 2026-02-20T11:14:37.586 INFO:teuthology.orchestra.run.trial043.stdout:Modify: 2026-02-20 11:11:03.839068179 +0000 2026-02-20T11:14:37.586 INFO:teuthology.orchestra.run.trial043.stdout:Change: 2026-02-20 11:11:03.839068179 +0000 2026-02-20T11:14:37.586 INFO:teuthology.orchestra.run.trial043.stdout: Birth: 2026-02-20 11:11:03.839068179 +0000 2026-02-20T11:14:37.586 DEBUG:teuthology.orchestra.run.trial043:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-20T11:14:37.648 INFO:teuthology.orchestra.run.trial043.stderr:1+0 records in 2026-02-20T11:14:37.648 INFO:teuthology.orchestra.run.trial043.stderr:1+0 records out 2026-02-20T11:14:37.648 INFO:teuthology.orchestra.run.trial043.stderr:512 bytes copied, 0.000146823 s, 3.5 MB/s 2026-02-20T11:14:37.649 DEBUG:teuthology.orchestra.run.trial043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-20T11:14:37.709 DEBUG:teuthology.orchestra.run.trial043:> stat /dev/vg_nvme/lv_3 2026-02-20T11:14:37.766 INFO:teuthology.orchestra.run.trial043.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-20T11:14:37.766 INFO:teuthology.orchestra.run.trial043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-20T11:14:37.766 INFO:teuthology.orchestra.run.trial043.stdout:Device: 6h/6d Inode: 1302 Links: 1 2026-02-20T11:14:37.766 INFO:teuthology.orchestra.run.trial043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-20T11:14:37.766 INFO:teuthology.orchestra.run.trial043.stdout:Context: system_u:object_r:device_t:s0 2026-02-20T11:14:37.766 INFO:teuthology.orchestra.run.trial043.stdout:Access: 2026-02-20 11:12:21.127785337 +0000 2026-02-20T11:14:37.766 INFO:teuthology.orchestra.run.trial043.stdout:Modify: 2026-02-20 11:11:03.839068179 +0000 2026-02-20T11:14:37.766 INFO:teuthology.orchestra.run.trial043.stdout:Change: 2026-02-20 11:11:03.839068179 +0000 2026-02-20T11:14:37.766 INFO:teuthology.orchestra.run.trial043.stdout: Birth: 2026-02-20 11:11:03.839068179 +0000 2026-02-20T11:14:37.767 DEBUG:teuthology.orchestra.run.trial043:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-20T11:14:37.830 INFO:teuthology.orchestra.run.trial043.stderr:1+0 records in 2026-02-20T11:14:37.830 INFO:teuthology.orchestra.run.trial043.stderr:1+0 records out 2026-02-20T11:14:37.830 INFO:teuthology.orchestra.run.trial043.stderr:512 bytes copied, 0.000167561 s, 3.1 MB/s 2026-02-20T11:14:37.831 DEBUG:teuthology.orchestra.run.trial043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-20T11:14:37.889 DEBUG:teuthology.orchestra.run.trial043:> stat /dev/vg_nvme/lv_4 2026-02-20T11:14:37.946 INFO:teuthology.orchestra.run.trial043.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-20T11:14:37.947 INFO:teuthology.orchestra.run.trial043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-20T11:14:37.947 INFO:teuthology.orchestra.run.trial043.stdout:Device: 6h/6d Inode: 1295 Links: 1 2026-02-20T11:14:37.947 INFO:teuthology.orchestra.run.trial043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-20T11:14:37.947 INFO:teuthology.orchestra.run.trial043.stdout:Context: system_u:object_r:device_t:s0 2026-02-20T11:14:37.947 INFO:teuthology.orchestra.run.trial043.stdout:Access: 2026-02-20 11:12:21.128785322 +0000 2026-02-20T11:14:37.947 INFO:teuthology.orchestra.run.trial043.stdout:Modify: 2026-02-20 11:11:03.839068179 +0000 2026-02-20T11:14:37.947 INFO:teuthology.orchestra.run.trial043.stdout:Change: 2026-02-20 11:11:03.839068179 +0000 2026-02-20T11:14:37.947 INFO:teuthology.orchestra.run.trial043.stdout: Birth: 2026-02-20 11:11:03.839068179 +0000 2026-02-20T11:14:37.948 DEBUG:teuthology.orchestra.run.trial043:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-20T11:14:38.011 INFO:teuthology.orchestra.run.trial043.stderr:1+0 records in 2026-02-20T11:14:38.011 INFO:teuthology.orchestra.run.trial043.stderr:1+0 records out 2026-02-20T11:14:38.011 INFO:teuthology.orchestra.run.trial043.stderr:512 bytes copied, 0.000138988 s, 3.7 MB/s 2026-02-20T11:14:38.012 DEBUG:teuthology.orchestra.run.trial043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-20T11:14:38.070 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2026-02-20T11:14:38.071 DEBUG:teuthology.orchestra.run.trial109:> set -ex 2026-02-20T11:14:38.071 DEBUG:teuthology.orchestra.run.trial109:> dd if=/scratch_devs of=/dev/stdout 2026-02-20T11:14:38.089 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-20T11:14:38.089 DEBUG:teuthology.orchestra.run.trial109:> stat /dev/vg_nvme/lv_1 2026-02-20T11:14:38.147 INFO:teuthology.orchestra.run.trial109.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-20T11:14:38.148 INFO:teuthology.orchestra.run.trial109.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-20T11:14:38.148 INFO:teuthology.orchestra.run.trial109.stdout:Device: 6h/6d Inode: 1300 Links: 1 2026-02-20T11:14:38.148 INFO:teuthology.orchestra.run.trial109.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-20T11:14:38.148 INFO:teuthology.orchestra.run.trial109.stdout:Context: system_u:object_r:device_t:s0 2026-02-20T11:14:38.148 INFO:teuthology.orchestra.run.trial109.stdout:Access: 2026-02-20 11:12:25.258174475 +0000 2026-02-20T11:14:38.148 INFO:teuthology.orchestra.run.trial109.stdout:Modify: 2026-02-20 11:11:03.360243374 +0000 2026-02-20T11:14:38.148 INFO:teuthology.orchestra.run.trial109.stdout:Change: 2026-02-20 11:11:03.360243374 +0000 2026-02-20T11:14:38.148 INFO:teuthology.orchestra.run.trial109.stdout: Birth: 2026-02-20 11:11:03.360243374 +0000 2026-02-20T11:14:38.149 DEBUG:teuthology.orchestra.run.trial109:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-20T11:14:38.212 INFO:teuthology.orchestra.run.trial109.stderr:1+0 records in 2026-02-20T11:14:38.213 INFO:teuthology.orchestra.run.trial109.stderr:1+0 records out 2026-02-20T11:14:38.213 INFO:teuthology.orchestra.run.trial109.stderr:512 bytes copied, 0.00015119 s, 3.4 MB/s 2026-02-20T11:14:38.214 DEBUG:teuthology.orchestra.run.trial109:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-20T11:14:38.273 DEBUG:teuthology.orchestra.run.trial109:> stat /dev/vg_nvme/lv_2 2026-02-20T11:14:38.330 INFO:teuthology.orchestra.run.trial109.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-20T11:14:38.330 INFO:teuthology.orchestra.run.trial109.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-20T11:14:38.330 INFO:teuthology.orchestra.run.trial109.stdout:Device: 6h/6d Inode: 1310 Links: 1 2026-02-20T11:14:38.330 INFO:teuthology.orchestra.run.trial109.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-20T11:14:38.330 INFO:teuthology.orchestra.run.trial109.stdout:Context: system_u:object_r:device_t:s0 2026-02-20T11:14:38.330 INFO:teuthology.orchestra.run.trial109.stdout:Access: 2026-02-20 11:12:25.259174437 +0000 2026-02-20T11:14:38.330 INFO:teuthology.orchestra.run.trial109.stdout:Modify: 2026-02-20 11:11:03.360243374 +0000 2026-02-20T11:14:38.330 INFO:teuthology.orchestra.run.trial109.stdout:Change: 2026-02-20 11:11:03.360243374 +0000 2026-02-20T11:14:38.330 INFO:teuthology.orchestra.run.trial109.stdout: Birth: 2026-02-20 11:11:03.360243374 +0000 2026-02-20T11:14:38.331 DEBUG:teuthology.orchestra.run.trial109:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-20T11:14:38.393 INFO:teuthology.orchestra.run.trial109.stderr:1+0 records in 2026-02-20T11:14:38.393 INFO:teuthology.orchestra.run.trial109.stderr:1+0 records out 2026-02-20T11:14:38.393 INFO:teuthology.orchestra.run.trial109.stderr:512 bytes copied, 0.00016707 s, 3.1 MB/s 2026-02-20T11:14:38.394 DEBUG:teuthology.orchestra.run.trial109:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-20T11:14:38.452 DEBUG:teuthology.orchestra.run.trial109:> stat /dev/vg_nvme/lv_3 2026-02-20T11:14:38.508 INFO:teuthology.orchestra.run.trial109.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-20T11:14:38.509 INFO:teuthology.orchestra.run.trial109.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-20T11:14:38.509 INFO:teuthology.orchestra.run.trial109.stdout:Device: 6h/6d Inode: 1301 Links: 1 2026-02-20T11:14:38.509 INFO:teuthology.orchestra.run.trial109.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-20T11:14:38.509 INFO:teuthology.orchestra.run.trial109.stdout:Context: system_u:object_r:device_t:s0 2026-02-20T11:14:38.509 INFO:teuthology.orchestra.run.trial109.stdout:Access: 2026-02-20 11:12:25.259174437 +0000 2026-02-20T11:14:38.509 INFO:teuthology.orchestra.run.trial109.stdout:Modify: 2026-02-20 11:11:03.360243374 +0000 2026-02-20T11:14:38.509 INFO:teuthology.orchestra.run.trial109.stdout:Change: 2026-02-20 11:11:03.360243374 +0000 2026-02-20T11:14:38.509 INFO:teuthology.orchestra.run.trial109.stdout: Birth: 2026-02-20 11:11:03.360243374 +0000 2026-02-20T11:14:38.510 DEBUG:teuthology.orchestra.run.trial109:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-20T11:14:38.571 INFO:teuthology.orchestra.run.trial109.stderr:1+0 records in 2026-02-20T11:14:38.571 INFO:teuthology.orchestra.run.trial109.stderr:1+0 records out 2026-02-20T11:14:38.571 INFO:teuthology.orchestra.run.trial109.stderr:512 bytes copied, 0.000165667 s, 3.1 MB/s 2026-02-20T11:14:38.573 DEBUG:teuthology.orchestra.run.trial109:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-20T11:14:38.631 DEBUG:teuthology.orchestra.run.trial109:> stat /dev/vg_nvme/lv_4 2026-02-20T11:14:38.688 INFO:teuthology.orchestra.run.trial109.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-20T11:14:38.688 INFO:teuthology.orchestra.run.trial109.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-20T11:14:38.688 INFO:teuthology.orchestra.run.trial109.stdout:Device: 6h/6d Inode: 1291 Links: 1 2026-02-20T11:14:38.688 INFO:teuthology.orchestra.run.trial109.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-20T11:14:38.689 INFO:teuthology.orchestra.run.trial109.stdout:Context: system_u:object_r:device_t:s0 2026-02-20T11:14:38.689 INFO:teuthology.orchestra.run.trial109.stdout:Access: 2026-02-20 11:12:25.259174437 +0000 2026-02-20T11:14:38.689 INFO:teuthology.orchestra.run.trial109.stdout:Modify: 2026-02-20 11:11:03.359243412 +0000 2026-02-20T11:14:38.689 INFO:teuthology.orchestra.run.trial109.stdout:Change: 2026-02-20 11:11:03.359243412 +0000 2026-02-20T11:14:38.689 INFO:teuthology.orchestra.run.trial109.stdout: Birth: 2026-02-20 11:11:03.359243412 +0000 2026-02-20T11:14:38.690 DEBUG:teuthology.orchestra.run.trial109:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-20T11:14:38.750 INFO:teuthology.orchestra.run.trial109.stderr:1+0 records in 2026-02-20T11:14:38.751 INFO:teuthology.orchestra.run.trial109.stderr:1+0 records out 2026-02-20T11:14:38.751 INFO:teuthology.orchestra.run.trial109.stderr:512 bytes copied, 0.00013435 s, 3.8 MB/s 2026-02-20T11:14:38.753 DEBUG:teuthology.orchestra.run.trial109:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-20T11:14:38.809 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2026-02-20T11:14:38.809 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial043.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@trial109.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2026-02-20T11:14:38.810 INFO:tasks.ceph:Generating config... 2026-02-20T11:14:38.811 INFO:tasks.ceph:[osd] debug monc = 20 2026-02-20T11:14:38.811 INFO:tasks.ceph:[osd] crimson cpu num = 4 2026-02-20T11:14:38.811 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2026-02-20T11:14:38.811 INFO:tasks.ceph:[osd] debug ms = 20 2026-02-20T11:14:38.811 INFO:tasks.ceph:[osd] debug osd = 20 2026-02-20T11:14:38.811 INFO:tasks.ceph:[osd] debug seastore = 20 2026-02-20T11:14:38.811 INFO:tasks.ceph:[osd] debug seastore backref = 20 2026-02-20T11:14:38.811 INFO:tasks.ceph:[osd] debug seastore cache = 20 2026-02-20T11:14:38.811 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2026-02-20T11:14:38.811 INFO:tasks.ceph:[osd] debug seastore device = 20 2026-02-20T11:14:38.811 INFO:tasks.ceph:[osd] debug seastore epm = 20 2026-02-20T11:14:38.811 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2026-02-20T11:14:38.811 INFO:tasks.ceph:[osd] debug seastore journal = 20 2026-02-20T11:14:38.811 INFO:tasks.ceph:[osd] debug seastore lba = 20 2026-02-20T11:14:38.811 INFO:tasks.ceph:[osd] debug seastore odata = 20 2026-02-20T11:14:38.811 INFO:tasks.ceph:[osd] debug seastore omap = 20 2026-02-20T11:14:38.811 INFO:tasks.ceph:[osd] debug seastore onode = 20 2026-02-20T11:14:38.812 INFO:tasks.ceph:[osd] debug seastore t = 20 2026-02-20T11:14:38.812 INFO:tasks.ceph:[osd] debug seastore tm = 20 2026-02-20T11:14:38.812 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2026-02-20T11:14:38.812 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2026-02-20T11:14:38.812 INFO:tasks.ceph:[osd] osd delete sleep = 1 2026-02-20T11:14:38.812 INFO:tasks.ceph:[osd] osd max backfills = 3 2026-02-20T11:14:38.812 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2026-02-20T11:14:38.812 INFO:tasks.ceph:[osd] osd objectstore = seastore 2026-02-20T11:14:38.812 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2026-02-20T11:14:38.812 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2026-02-20T11:14:38.812 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2026-02-20T11:14:38.812 INFO:tasks.ceph:[osd] osd snap trim sleep = 2 2026-02-20T11:14:38.812 INFO:tasks.ceph:[osd] osd target transaction size = 5 2026-02-20T11:14:38.812 INFO:tasks.ceph:[osd] seastore cachepin size pershard = 64M 2026-02-20T11:14:38.812 INFO:tasks.ceph:[osd] seastore main device type = RANDOM_BLOCK_SSD 2026-02-20T11:14:38.812 INFO:tasks.ceph:[osd] seastore max concurrent transactions = 8 2026-02-20T11:14:38.812 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2026-02-20T11:14:38.812 INFO:tasks.ceph:[global] ms client mode = crc 2026-02-20T11:14:38.812 INFO:tasks.ceph:[global] ms cluster mode = crc 2026-02-20T11:14:38.812 INFO:tasks.ceph:[global] ms mon client mode = crc 2026-02-20T11:14:38.812 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2026-02-20T11:14:38.812 INFO:tasks.ceph:[global] ms mon service mode = crc 2026-02-20T11:14:38.812 INFO:tasks.ceph:[global] ms service mode = crc 2026-02-20T11:14:38.813 INFO:tasks.ceph:[global] osd_max_pg_log_entries = 2 2026-02-20T11:14:38.813 INFO:tasks.ceph:[global] osd_min_pg_log_entries = 1 2026-02-20T11:14:38.813 INFO:tasks.ceph:[global] osd_pg_log_trim_min = 0 2026-02-20T11:14:38.813 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2026-02-20T11:14:38.813 INFO:tasks.ceph:[global] osd_pool_default_size = 3 2026-02-20T11:14:38.813 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-20T11:14:38.813 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-20T11:14:38.813 INFO:tasks.ceph:[mon] debug mon = 20 2026-02-20T11:14:38.813 INFO:tasks.ceph:[mon] debug ms = 1 2026-02-20T11:14:38.813 INFO:tasks.ceph:[mon] debug paxos = 20 2026-02-20T11:14:38.813 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2026-02-20T11:14:38.813 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2026-02-20T11:14:38.813 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2026-02-20T11:14:38.813 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2026-02-20T11:14:38.814 INFO:tasks.ceph:[mon] osd pool default crimson = True 2026-02-20T11:14:38.814 INFO:tasks.ceph:[mon] paxos service trim min = 10 2026-02-20T11:14:38.814 INFO:tasks.ceph:Setting up mon.a... 2026-02-20T11:14:38.814 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-20T11:14:38.858 INFO:teuthology.orchestra.run.trial043.stdout:creating /etc/ceph/ceph.keyring 2026-02-20T11:14:38.860 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-20T11:14:38.947 DEBUG:teuthology.orchestra.run.trial043:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-20T11:14:38.972 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.43')] 2026-02-20T11:14:38.972 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.43', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': 'true', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd target transaction size': 5, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2026-02-20T11:14:38.974 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-20T11:14:38.974 DEBUG:teuthology.orchestra.run.trial043:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-20T11:14:39.028 DEBUG:teuthology.orchestra.run.trial043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.43 --print /home/ubuntu/cephtest/ceph.monmap 2026-02-20T11:14:39.106 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:14:39.106+0000 7f24be28af40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:14:39.107 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:14:39.106+0000 7f24be28af40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:14:39.107 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:14:39.106+0000 7f24be28af40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:14:39.108 INFO:teuthology.orchestra.run.trial043.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-20T11:14:39.108 INFO:teuthology.orchestra.run.trial043.stdout:monmaptool: generated fsid dfee6dd4-7cac-4c06-aaed-6868a4b5b9a0 2026-02-20T11:14:39.108 INFO:teuthology.orchestra.run.trial043.stdout:setting min_mon_release = tentacle 2026-02-20T11:14:39.108 INFO:teuthology.orchestra.run.trial043.stdout:epoch 0 2026-02-20T11:14:39.108 INFO:teuthology.orchestra.run.trial043.stdout:fsid dfee6dd4-7cac-4c06-aaed-6868a4b5b9a0 2026-02-20T11:14:39.109 INFO:teuthology.orchestra.run.trial043.stdout:last_changed 2026-02-20T11:14:39.107568+0000 2026-02-20T11:14:39.109 INFO:teuthology.orchestra.run.trial043.stdout:created 2026-02-20T11:14:39.107568+0000 2026-02-20T11:14:39.109 INFO:teuthology.orchestra.run.trial043.stdout:min_mon_release 20 (tentacle) 2026-02-20T11:14:39.109 INFO:teuthology.orchestra.run.trial043.stdout:election_strategy: 1 2026-02-20T11:14:39.109 INFO:teuthology.orchestra.run.trial043.stdout:0: [v2:10.20.193.43:3300/0,v1:10.20.193.43:6789/0] mon.a 2026-02-20T11:14:39.109 INFO:teuthology.orchestra.run.trial043.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2026-02-20T11:14:39.110 DEBUG:teuthology.orchestra.run.trial043:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-20T11:14:39.165 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID dfee6dd4-7cac-4c06-aaed-6868a4b5b9a0... 2026-02-20T11:14:39.167 DEBUG:teuthology.orchestra.run.trial043:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-20T11:14:39.208 DEBUG:teuthology.orchestra.run.trial109:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-20T11:14:39.247 INFO:teuthology.orchestra.run.trial043.stdout:[global] 2026-02-20T11:14:39.247 INFO:teuthology.orchestra.run.trial043.stdout: chdir = "" 2026-02-20T11:14:39.247 INFO:teuthology.orchestra.run.trial043.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-20T11:14:39.247 INFO:teuthology.orchestra.run.trial043.stdout: auth supported = cephx 2026-02-20T11:14:39.247 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:14:39.248 INFO:teuthology.orchestra.run.trial043.stdout: filestore xattr use omap = true 2026-02-20T11:14:39.248 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:14:39.248 INFO:teuthology.orchestra.run.trial043.stdout: mon clock drift allowed = 1.000 2026-02-20T11:14:39.248 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:14:39.248 INFO:teuthology.orchestra.run.trial043.stdout: osd crush chooseleaf type = 0 2026-02-20T11:14:39.248 INFO:teuthology.orchestra.run.trial043.stdout: auth debug = true 2026-02-20T11:14:39.248 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:14:39.248 INFO:teuthology.orchestra.run.trial043.stdout: ms die on old message = true 2026-02-20T11:14:39.248 INFO:teuthology.orchestra.run.trial043.stdout: ms die on bug = true 2026-02-20T11:14:39.248 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:14:39.248 INFO:teuthology.orchestra.run.trial043.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-20T11:14:39.249 INFO:teuthology.orchestra.run.trial043.stdout: mon pg warn max object skew = 0 2026-02-20T11:14:39.249 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:14:39.249 INFO:teuthology.orchestra.run.trial043.stdout: # disable pg_autoscaler by default for new pools 2026-02-20T11:14:39.249 INFO:teuthology.orchestra.run.trial043.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-20T11:14:39.249 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:14:39.249 INFO:teuthology.orchestra.run.trial043.stdout: osd pool default size = 2 2026-02-20T11:14:39.249 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:14:39.249 INFO:teuthology.orchestra.run.trial043.stdout: mon osd allow primary affinity = true 2026-02-20T11:14:39.249 INFO:teuthology.orchestra.run.trial043.stdout: mon osd allow pg remap = true 2026-02-20T11:14:39.250 INFO:teuthology.orchestra.run.trial043.stdout: mon warn on legacy crush tunables = false 2026-02-20T11:14:39.250 INFO:teuthology.orchestra.run.trial043.stdout: mon warn on crush straw calc version zero = false 2026-02-20T11:14:39.250 INFO:teuthology.orchestra.run.trial043.stdout: mon warn on no sortbitwise = false 2026-02-20T11:14:39.250 INFO:teuthology.orchestra.run.trial043.stdout: mon warn on osd down out interval zero = false 2026-02-20T11:14:39.250 INFO:teuthology.orchestra.run.trial043.stdout: mon warn on too few osds = false 2026-02-20T11:14:39.250 INFO:teuthology.orchestra.run.trial043.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-20T11:14:39.250 INFO:teuthology.orchestra.run.trial043.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-20T11:14:39.250 INFO:teuthology.orchestra.run.trial043.stdout: mon_allow_pool_size_one = true 2026-02-20T11:14:39.250 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:14:39.250 INFO:teuthology.orchestra.run.trial043.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-20T11:14:39.251 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:14:39.251 INFO:teuthology.orchestra.run.trial043.stdout: osd default data pool replay window = 5 2026-02-20T11:14:39.251 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:14:39.251 INFO:teuthology.orchestra.run.trial043.stdout: mon allow pool delete = true 2026-02-20T11:14:39.251 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:14:39.251 INFO:teuthology.orchestra.run.trial043.stdout: mon cluster log file level = debug 2026-02-20T11:14:39.251 INFO:teuthology.orchestra.run.trial043.stdout: debug asserts on shutdown = true 2026-02-20T11:14:39.251 INFO:teuthology.orchestra.run.trial043.stdout: mon health detail to clog = false 2026-02-20T11:14:39.251 INFO:teuthology.orchestra.run.trial043.stdout: mon host = 10.20.193.43 2026-02-20T11:14:39.251 INFO:teuthology.orchestra.run.trial043.stdout: enable experimental unrecoverable data corrupting features = crimson 2026-02-20T11:14:39.251 INFO:teuthology.orchestra.run.trial043.stdout: ms client mode = crc 2026-02-20T11:14:39.251 INFO:teuthology.orchestra.run.trial043.stdout: ms cluster mode = crc 2026-02-20T11:14:39.252 INFO:teuthology.orchestra.run.trial043.stdout: ms mon client mode = crc 2026-02-20T11:14:39.252 INFO:teuthology.orchestra.run.trial043.stdout: ms mon cluster mode = crc 2026-02-20T11:14:39.252 INFO:teuthology.orchestra.run.trial043.stdout: ms mon service mode = crc 2026-02-20T11:14:39.252 INFO:teuthology.orchestra.run.trial043.stdout: ms service mode = crc 2026-02-20T11:14:39.252 INFO:teuthology.orchestra.run.trial043.stdout: osd_max_pg_log_entries = 2 2026-02-20T11:14:39.252 INFO:teuthology.orchestra.run.trial043.stdout: osd_min_pg_log_entries = 1 2026-02-20T11:14:39.252 INFO:teuthology.orchestra.run.trial043.stdout: osd_pg_log_trim_min = 0 2026-02-20T11:14:39.252 INFO:teuthology.orchestra.run.trial043.stdout: osd_pool_default_min_size = 2 2026-02-20T11:14:39.252 INFO:teuthology.orchestra.run.trial043.stdout: osd_pool_default_size = 3 2026-02-20T11:14:39.252 INFO:teuthology.orchestra.run.trial043.stdout: fsid = dfee6dd4-7cac-4c06-aaed-6868a4b5b9a0 2026-02-20T11:14:39.252 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:14:39.252 INFO:teuthology.orchestra.run.trial043.stdout:[osd] 2026-02-20T11:14:39.253 INFO:teuthology.orchestra.run.trial043.stdout: osd journal size = 100 2026-02-20T11:14:39.253 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:14:39.253 INFO:teuthology.orchestra.run.trial043.stdout: osd scrub load threshold = 5.0 2026-02-20T11:14:39.253 INFO:teuthology.orchestra.run.trial043.stdout: osd scrub max interval = 120 2026-02-20T11:14:39.253 INFO:teuthology.orchestra.run.trial043.stdout: osd mclock profile = high_recovery_ops 2026-02-20T11:14:39.253 INFO:teuthology.orchestra.run.trial043.stdout: osd mclock skip benchmark = true 2026-02-20T11:14:39.253 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:14:39.253 INFO:teuthology.orchestra.run.trial043.stdout: osd recover clone overlap = true 2026-02-20T11:14:39.253 INFO:teuthology.orchestra.run.trial043.stdout: osd recovery max chunk = 1048576 2026-02-20T11:14:39.253 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:14:39.253 INFO:teuthology.orchestra.run.trial043.stdout: osd debug shutdown = true 2026-02-20T11:14:39.253 INFO:teuthology.orchestra.run.trial043.stdout: osd debug op order = true 2026-02-20T11:14:39.253 INFO:teuthology.orchestra.run.trial043.stdout: osd debug verify stray on activate = true 2026-02-20T11:14:39.254 INFO:teuthology.orchestra.run.trial043.stdout: osd debug trim objects = true 2026-02-20T11:14:39.254 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:14:39.254 INFO:teuthology.orchestra.run.trial043.stdout: osd open classes on start = true 2026-02-20T11:14:39.254 INFO:teuthology.orchestra.run.trial043.stdout: osd debug pg log writeout = true 2026-02-20T11:14:39.254 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:14:39.254 INFO:teuthology.orchestra.run.trial043.stdout: osd deep scrub update digest min age = 30 2026-02-20T11:14:39.254 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:14:39.254 INFO:teuthology.orchestra.run.trial043.stdout: osd map max advance = 10 2026-02-20T11:14:39.254 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:14:39.254 INFO:teuthology.orchestra.run.trial043.stdout: journal zero on create = true 2026-02-20T11:14:39.254 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:14:39.254 INFO:teuthology.orchestra.run.trial043.stdout: filestore ondisk finisher threads = 3 2026-02-20T11:14:39.254 INFO:teuthology.orchestra.run.trial043.stdout: filestore apply finisher threads = 3 2026-02-20T11:14:39.255 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:14:39.255 INFO:teuthology.orchestra.run.trial043.stdout: bdev debug aio = true 2026-02-20T11:14:39.255 INFO:teuthology.orchestra.run.trial043.stdout: osd debug misdirected ops = true 2026-02-20T11:14:39.255 INFO:teuthology.orchestra.run.trial043.stdout: debug monc = 20 2026-02-20T11:14:39.255 INFO:teuthology.orchestra.run.trial043.stdout: crimson cpu num = 4 2026-02-20T11:14:39.255 INFO:teuthology.orchestra.run.trial043.stdout: crimson osd obc lru size = 10 2026-02-20T11:14:39.255 INFO:teuthology.orchestra.run.trial043.stdout: debug ms = 20 2026-02-20T11:14:39.255 INFO:teuthology.orchestra.run.trial043.stdout: debug osd = 20 2026-02-20T11:14:39.255 INFO:teuthology.orchestra.run.trial043.stdout: debug seastore = 20 2026-02-20T11:14:39.255 INFO:teuthology.orchestra.run.trial043.stdout: debug seastore backref = 20 2026-02-20T11:14:39.255 INFO:teuthology.orchestra.run.trial043.stdout: debug seastore cache = 20 2026-02-20T11:14:39.255 INFO:teuthology.orchestra.run.trial043.stdout: debug seastore cleaner = 20 2026-02-20T11:14:39.256 INFO:teuthology.orchestra.run.trial043.stdout: debug seastore device = 20 2026-02-20T11:14:39.256 INFO:teuthology.orchestra.run.trial043.stdout: debug seastore epm = 20 2026-02-20T11:14:39.256 INFO:teuthology.orchestra.run.trial043.stdout: debug seastore fixedkv tree = 20 2026-02-20T11:14:39.256 INFO:teuthology.orchestra.run.trial043.stdout: debug seastore journal = 20 2026-02-20T11:14:39.256 INFO:teuthology.orchestra.run.trial043.stdout: debug seastore lba = 20 2026-02-20T11:14:39.256 INFO:teuthology.orchestra.run.trial043.stdout: debug seastore odata = 20 2026-02-20T11:14:39.256 INFO:teuthology.orchestra.run.trial043.stdout: debug seastore omap = 20 2026-02-20T11:14:39.256 INFO:teuthology.orchestra.run.trial043.stdout: debug seastore onode = 20 2026-02-20T11:14:39.256 INFO:teuthology.orchestra.run.trial043.stdout: debug seastore t = 20 2026-02-20T11:14:39.256 INFO:teuthology.orchestra.run.trial043.stdout: debug seastore tm = 20 2026-02-20T11:14:39.256 INFO:teuthology.orchestra.run.trial043.stdout: osd blocked scrub grace period = 3600 2026-02-20T11:14:39.256 INFO:teuthology.orchestra.run.trial043.stdout: osd debug reject backfill probability = 0.3 2026-02-20T11:14:39.256 INFO:teuthology.orchestra.run.trial043.stdout: osd delete sleep = 1 2026-02-20T11:14:39.256 INFO:teuthology.orchestra.run.trial043.stdout: osd max backfills = 3 2026-02-20T11:14:39.257 INFO:teuthology.orchestra.run.trial043.stdout: osd max markdown count = 1000 2026-02-20T11:14:39.257 INFO:teuthology.orchestra.run.trial043.stdout: osd objectstore = seastore 2026-02-20T11:14:39.257 INFO:teuthology.orchestra.run.trial043.stdout: osd scrub min interval = 60 2026-02-20T11:14:39.257 INFO:teuthology.orchestra.run.trial043.stdout: osd shutdown pgref assert = True 2026-02-20T11:14:39.257 INFO:teuthology.orchestra.run.trial043.stdout: osd snap trim sleep = 2 2026-02-20T11:14:39.257 INFO:teuthology.orchestra.run.trial043.stdout: osd target transaction size = 5 2026-02-20T11:14:39.257 INFO:teuthology.orchestra.run.trial043.stdout: seastore cachepin size pershard = 64M 2026-02-20T11:14:39.257 INFO:teuthology.orchestra.run.trial043.stdout: seastore main device type = RANDOM_BLOCK_SSD 2026-02-20T11:14:39.257 INFO:teuthology.orchestra.run.trial043.stdout: seastore max concurrent transactions = 8 2026-02-20T11:14:39.257 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:14:39.257 INFO:teuthology.orchestra.run.trial043.stdout:[mgr] 2026-02-20T11:14:39.257 INFO:teuthology.orchestra.run.trial043.stdout: debug ms = 1 2026-02-20T11:14:39.258 INFO:teuthology.orchestra.run.trial043.stdout: debug mgr = 20 2026-02-20T11:14:39.258 INFO:teuthology.orchestra.run.trial043.stdout: debug mon = 20 2026-02-20T11:14:39.258 INFO:teuthology.orchestra.run.trial043.stdout: debug auth = 20 2026-02-20T11:14:39.258 INFO:teuthology.orchestra.run.trial043.stdout: mon reweight min pgs per osd = 4 2026-02-20T11:14:39.258 INFO:teuthology.orchestra.run.trial043.stdout: mon reweight min bytes per osd = 10 2026-02-20T11:14:39.258 INFO:teuthology.orchestra.run.trial043.stdout: mgr/telemetry/nag = false 2026-02-20T11:14:39.258 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:14:39.258 INFO:teuthology.orchestra.run.trial043.stdout:[mon] 2026-02-20T11:14:39.258 INFO:teuthology.orchestra.run.trial043.stdout: debug ms = 1 2026-02-20T11:14:39.258 INFO:teuthology.orchestra.run.trial043.stdout: debug mon = 20 2026-02-20T11:14:39.259 INFO:teuthology.orchestra.run.trial043.stdout: debug paxos = 20 2026-02-20T11:14:39.259 INFO:teuthology.orchestra.run.trial043.stdout: debug auth = 20 2026-02-20T11:14:39.259 INFO:teuthology.orchestra.run.trial043.stdout: mon data avail warn = 5 2026-02-20T11:14:39.259 INFO:teuthology.orchestra.run.trial043.stdout: mon mgr mkfs grace = 240 2026-02-20T11:14:39.259 INFO:teuthology.orchestra.run.trial043.stdout: mon reweight min pgs per osd = 4 2026-02-20T11:14:39.259 INFO:teuthology.orchestra.run.trial043.stdout: mon osd reporter subtree level = osd 2026-02-20T11:14:39.259 INFO:teuthology.orchestra.run.trial043.stdout: mon osd prime pg temp = true 2026-02-20T11:14:39.259 INFO:teuthology.orchestra.run.trial043.stdout: mon reweight min bytes per osd = 10 2026-02-20T11:14:39.259 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:14:39.259 INFO:teuthology.orchestra.run.trial043.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-20T11:14:39.259 INFO:teuthology.orchestra.run.trial043.stdout: auth mon ticket ttl = 660 # 11m 2026-02-20T11:14:39.259 INFO:teuthology.orchestra.run.trial043.stdout: auth service ticket ttl = 240 # 4m 2026-02-20T11:14:39.259 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:14:39.260 INFO:teuthology.orchestra.run.trial043.stdout: # don't complain about insecure global_id in the test suite 2026-02-20T11:14:39.260 INFO:teuthology.orchestra.run.trial043.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-20T11:14:39.260 INFO:teuthology.orchestra.run.trial043.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-20T11:14:39.260 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:14:39.260 INFO:teuthology.orchestra.run.trial043.stdout: # 1m isn't quite enough 2026-02-20T11:14:39.260 INFO:teuthology.orchestra.run.trial043.stdout: mon_down_mkfs_grace = 2m 2026-02-20T11:14:39.260 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:14:39.260 INFO:teuthology.orchestra.run.trial043.stdout: mon_warn_on_filestore_osds = false 2026-02-20T11:14:39.260 INFO:teuthology.orchestra.run.trial043.stdout: mon min osdmap epochs = 50 2026-02-20T11:14:39.260 INFO:teuthology.orchestra.run.trial043.stdout: mon osdmap full prune interval = 2 2026-02-20T11:14:39.260 INFO:teuthology.orchestra.run.trial043.stdout: mon osdmap full prune min = 15 2026-02-20T11:14:39.260 INFO:teuthology.orchestra.run.trial043.stdout: mon osdmap full prune txsize = 2 2026-02-20T11:14:39.261 INFO:teuthology.orchestra.run.trial043.stdout: osd pool default crimson = True 2026-02-20T11:14:39.261 INFO:teuthology.orchestra.run.trial043.stdout: paxos service trim min = 10 2026-02-20T11:14:39.261 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:14:39.261 INFO:teuthology.orchestra.run.trial043.stdout:[client] 2026-02-20T11:14:39.261 INFO:teuthology.orchestra.run.trial043.stdout: rgw cache enabled = true 2026-02-20T11:14:39.261 INFO:teuthology.orchestra.run.trial043.stdout: rgw enable ops log = true 2026-02-20T11:14:39.261 INFO:teuthology.orchestra.run.trial043.stdout: rgw enable usage log = true 2026-02-20T11:14:39.261 INFO:teuthology.orchestra.run.trial043.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-20T11:14:39.261 INFO:teuthology.orchestra.run.trial043.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-20T11:14:39.261 INFO:teuthology.orchestra.run.trial043.stdout:[mon.a] 2026-02-20T11:14:39.263 INFO:teuthology.orchestra.run.trial109.stdout:[global] 2026-02-20T11:14:39.263 INFO:teuthology.orchestra.run.trial109.stdout: chdir = "" 2026-02-20T11:14:39.263 INFO:teuthology.orchestra.run.trial109.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-20T11:14:39.264 INFO:teuthology.orchestra.run.trial109.stdout: auth supported = cephx 2026-02-20T11:14:39.264 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:14:39.264 INFO:teuthology.orchestra.run.trial109.stdout: filestore xattr use omap = true 2026-02-20T11:14:39.264 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:14:39.264 INFO:teuthology.orchestra.run.trial109.stdout: mon clock drift allowed = 1.000 2026-02-20T11:14:39.264 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:14:39.264 INFO:teuthology.orchestra.run.trial109.stdout: osd crush chooseleaf type = 0 2026-02-20T11:14:39.264 INFO:teuthology.orchestra.run.trial109.stdout: auth debug = true 2026-02-20T11:14:39.264 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:14:39.264 INFO:teuthology.orchestra.run.trial109.stdout: ms die on old message = true 2026-02-20T11:14:39.264 INFO:teuthology.orchestra.run.trial109.stdout: ms die on bug = true 2026-02-20T11:14:39.264 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:14:39.265 INFO:teuthology.orchestra.run.trial109.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-20T11:14:39.265 INFO:teuthology.orchestra.run.trial109.stdout: mon pg warn max object skew = 0 2026-02-20T11:14:39.265 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:14:39.265 INFO:teuthology.orchestra.run.trial109.stdout: # disable pg_autoscaler by default for new pools 2026-02-20T11:14:39.265 INFO:teuthology.orchestra.run.trial109.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-20T11:14:39.265 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:14:39.265 INFO:teuthology.orchestra.run.trial109.stdout: osd pool default size = 2 2026-02-20T11:14:39.265 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:14:39.265 INFO:teuthology.orchestra.run.trial109.stdout: mon osd allow primary affinity = true 2026-02-20T11:14:39.266 INFO:teuthology.orchestra.run.trial109.stdout: mon osd allow pg remap = true 2026-02-20T11:14:39.266 INFO:teuthology.orchestra.run.trial109.stdout: mon warn on legacy crush tunables = false 2026-02-20T11:14:39.266 INFO:teuthology.orchestra.run.trial109.stdout: mon warn on crush straw calc version zero = false 2026-02-20T11:14:39.266 INFO:teuthology.orchestra.run.trial109.stdout: mon warn on no sortbitwise = false 2026-02-20T11:14:39.266 INFO:teuthology.orchestra.run.trial109.stdout: mon warn on osd down out interval zero = false 2026-02-20T11:14:39.266 INFO:teuthology.orchestra.run.trial109.stdout: mon warn on too few osds = false 2026-02-20T11:14:39.266 INFO:teuthology.orchestra.run.trial109.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-20T11:14:39.266 INFO:teuthology.orchestra.run.trial109.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-20T11:14:39.267 INFO:teuthology.orchestra.run.trial109.stdout: mon_allow_pool_size_one = true 2026-02-20T11:14:39.267 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:14:39.267 INFO:teuthology.orchestra.run.trial109.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-20T11:14:39.267 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:14:39.267 INFO:teuthology.orchestra.run.trial109.stdout: osd default data pool replay window = 5 2026-02-20T11:14:39.267 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:14:39.267 INFO:teuthology.orchestra.run.trial109.stdout: mon allow pool delete = true 2026-02-20T11:14:39.267 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:14:39.267 INFO:teuthology.orchestra.run.trial109.stdout: mon cluster log file level = debug 2026-02-20T11:14:39.267 INFO:teuthology.orchestra.run.trial109.stdout: debug asserts on shutdown = true 2026-02-20T11:14:39.268 INFO:teuthology.orchestra.run.trial109.stdout: mon health detail to clog = false 2026-02-20T11:14:39.268 INFO:teuthology.orchestra.run.trial109.stdout: mon host = 10.20.193.43 2026-02-20T11:14:39.268 INFO:teuthology.orchestra.run.trial109.stdout: enable experimental unrecoverable data corrupting features = crimson 2026-02-20T11:14:39.268 INFO:teuthology.orchestra.run.trial109.stdout: ms client mode = crc 2026-02-20T11:14:39.268 INFO:teuthology.orchestra.run.trial109.stdout: ms cluster mode = crc 2026-02-20T11:14:39.268 INFO:teuthology.orchestra.run.trial109.stdout: ms mon client mode = crc 2026-02-20T11:14:39.268 INFO:teuthology.orchestra.run.trial109.stdout: ms mon cluster mode = crc 2026-02-20T11:14:39.268 INFO:teuthology.orchestra.run.trial109.stdout: ms mon service mode = crc 2026-02-20T11:14:39.268 INFO:teuthology.orchestra.run.trial109.stdout: ms service mode = crc 2026-02-20T11:14:39.268 INFO:teuthology.orchestra.run.trial109.stdout: osd_max_pg_log_entries = 2 2026-02-20T11:14:39.269 INFO:teuthology.orchestra.run.trial109.stdout: osd_min_pg_log_entries = 1 2026-02-20T11:14:39.269 INFO:teuthology.orchestra.run.trial109.stdout: osd_pg_log_trim_min = 0 2026-02-20T11:14:39.269 INFO:teuthology.orchestra.run.trial109.stdout: osd_pool_default_min_size = 2 2026-02-20T11:14:39.269 INFO:teuthology.orchestra.run.trial109.stdout: osd_pool_default_size = 3 2026-02-20T11:14:39.269 INFO:teuthology.orchestra.run.trial109.stdout: fsid = dfee6dd4-7cac-4c06-aaed-6868a4b5b9a0 2026-02-20T11:14:39.269 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:14:39.269 INFO:teuthology.orchestra.run.trial109.stdout:[osd] 2026-02-20T11:14:39.269 INFO:teuthology.orchestra.run.trial109.stdout: osd journal size = 100 2026-02-20T11:14:39.269 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:14:39.269 INFO:teuthology.orchestra.run.trial109.stdout: osd scrub load threshold = 5.0 2026-02-20T11:14:39.270 INFO:teuthology.orchestra.run.trial109.stdout: osd scrub max interval = 120 2026-02-20T11:14:39.270 INFO:teuthology.orchestra.run.trial109.stdout: osd mclock profile = high_recovery_ops 2026-02-20T11:14:39.270 INFO:teuthology.orchestra.run.trial109.stdout: osd mclock skip benchmark = true 2026-02-20T11:14:39.270 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:14:39.270 INFO:teuthology.orchestra.run.trial109.stdout: osd recover clone overlap = true 2026-02-20T11:14:39.270 INFO:teuthology.orchestra.run.trial109.stdout: osd recovery max chunk = 1048576 2026-02-20T11:14:39.270 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:14:39.270 INFO:teuthology.orchestra.run.trial109.stdout: osd debug shutdown = true 2026-02-20T11:14:39.270 INFO:teuthology.orchestra.run.trial109.stdout: osd debug op order = true 2026-02-20T11:14:39.271 INFO:teuthology.orchestra.run.trial109.stdout: osd debug verify stray on activate = true 2026-02-20T11:14:39.271 INFO:teuthology.orchestra.run.trial109.stdout: osd debug trim objects = true 2026-02-20T11:14:39.271 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:14:39.271 INFO:teuthology.orchestra.run.trial109.stdout: osd open classes on start = true 2026-02-20T11:14:39.271 INFO:teuthology.orchestra.run.trial109.stdout: osd debug pg log writeout = true 2026-02-20T11:14:39.271 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:14:39.271 INFO:teuthology.orchestra.run.trial109.stdout: osd deep scrub update digest min age = 30 2026-02-20T11:14:39.271 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:14:39.271 INFO:teuthology.orchestra.run.trial109.stdout: osd map max advance = 10 2026-02-20T11:14:39.271 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:14:39.271 INFO:teuthology.orchestra.run.trial109.stdout: journal zero on create = true 2026-02-20T11:14:39.272 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:14:39.272 INFO:teuthology.orchestra.run.trial109.stdout: filestore ondisk finisher threads = 3 2026-02-20T11:14:39.272 INFO:teuthology.orchestra.run.trial109.stdout: filestore apply finisher threads = 3 2026-02-20T11:14:39.272 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:14:39.272 INFO:teuthology.orchestra.run.trial109.stdout: bdev debug aio = true 2026-02-20T11:14:39.272 INFO:teuthology.orchestra.run.trial109.stdout: osd debug misdirected ops = true 2026-02-20T11:14:39.272 INFO:teuthology.orchestra.run.trial109.stdout: debug monc = 20 2026-02-20T11:14:39.273 INFO:teuthology.orchestra.run.trial109.stdout: crimson cpu num = 4 2026-02-20T11:14:39.273 INFO:teuthology.orchestra.run.trial109.stdout: crimson osd obc lru size = 10 2026-02-20T11:14:39.273 INFO:teuthology.orchestra.run.trial109.stdout: debug ms = 20 2026-02-20T11:14:39.273 INFO:teuthology.orchestra.run.trial109.stdout: debug osd = 20 2026-02-20T11:14:39.273 INFO:teuthology.orchestra.run.trial109.stdout: debug seastore = 20 2026-02-20T11:14:39.273 INFO:teuthology.orchestra.run.trial109.stdout: debug seastore backref = 20 2026-02-20T11:14:39.273 INFO:teuthology.orchestra.run.trial109.stdout: debug seastore cache = 20 2026-02-20T11:14:39.273 INFO:teuthology.orchestra.run.trial109.stdout: debug seastore cleaner = 20 2026-02-20T11:14:39.274 INFO:teuthology.orchestra.run.trial109.stdout: debug seastore device = 20 2026-02-20T11:14:39.274 INFO:teuthology.orchestra.run.trial109.stdout: debug seastore epm = 20 2026-02-20T11:14:39.274 INFO:teuthology.orchestra.run.trial109.stdout: debug seastore fixedkv tree = 20 2026-02-20T11:14:39.274 INFO:teuthology.orchestra.run.trial109.stdout: debug seastore journal = 20 2026-02-20T11:14:39.274 INFO:teuthology.orchestra.run.trial109.stdout: debug seastore lba = 20 2026-02-20T11:14:39.274 INFO:teuthology.orchestra.run.trial109.stdout: debug seastore odata = 20 2026-02-20T11:14:39.274 INFO:teuthology.orchestra.run.trial109.stdout: debug seastore omap = 20 2026-02-20T11:14:39.274 INFO:teuthology.orchestra.run.trial109.stdout: debug seastore onode = 20 2026-02-20T11:14:39.274 INFO:teuthology.orchestra.run.trial109.stdout: debug seastore t = 20 2026-02-20T11:14:39.274 INFO:teuthology.orchestra.run.trial109.stdout: debug seastore tm = 20 2026-02-20T11:14:39.274 INFO:teuthology.orchestra.run.trial109.stdout: osd blocked scrub grace period = 3600 2026-02-20T11:14:39.274 INFO:teuthology.orchestra.run.trial109.stdout: osd debug reject backfill probability = 0.3 2026-02-20T11:14:39.274 INFO:teuthology.orchestra.run.trial109.stdout: osd delete sleep = 1 2026-02-20T11:14:39.275 INFO:teuthology.orchestra.run.trial109.stdout: osd max backfills = 3 2026-02-20T11:14:39.275 INFO:teuthology.orchestra.run.trial109.stdout: osd max markdown count = 1000 2026-02-20T11:14:39.275 INFO:teuthology.orchestra.run.trial109.stdout: osd objectstore = seastore 2026-02-20T11:14:39.275 INFO:teuthology.orchestra.run.trial109.stdout: osd scrub min interval = 60 2026-02-20T11:14:39.275 INFO:teuthology.orchestra.run.trial109.stdout: osd shutdown pgref assert = True 2026-02-20T11:14:39.275 INFO:teuthology.orchestra.run.trial109.stdout: osd snap trim sleep = 2 2026-02-20T11:14:39.275 INFO:teuthology.orchestra.run.trial109.stdout: osd target transaction size = 5 2026-02-20T11:14:39.275 INFO:teuthology.orchestra.run.trial109.stdout: seastore cachepin size pershard = 64M 2026-02-20T11:14:39.275 INFO:teuthology.orchestra.run.trial109.stdout: seastore main device type = RANDOM_BLOCK_SSD 2026-02-20T11:14:39.275 INFO:teuthology.orchestra.run.trial109.stdout: seastore max concurrent transactions = 8 2026-02-20T11:14:39.275 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:14:39.275 INFO:teuthology.orchestra.run.trial109.stdout:[mgr] 2026-02-20T11:14:39.276 INFO:teuthology.orchestra.run.trial109.stdout: debug ms = 1 2026-02-20T11:14:39.276 INFO:teuthology.orchestra.run.trial109.stdout: debug mgr = 20 2026-02-20T11:14:39.276 INFO:teuthology.orchestra.run.trial109.stdout: debug mon = 20 2026-02-20T11:14:39.276 INFO:teuthology.orchestra.run.trial109.stdout: debug auth = 20 2026-02-20T11:14:39.276 INFO:teuthology.orchestra.run.trial109.stdout: mon reweight min pgs per osd = 4 2026-02-20T11:14:39.276 INFO:teuthology.orchestra.run.trial109.stdout: mon reweight min bytes per osd = 10 2026-02-20T11:14:39.276 INFO:teuthology.orchestra.run.trial109.stdout: mgr/telemetry/nag = false 2026-02-20T11:14:39.276 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:14:39.276 INFO:teuthology.orchestra.run.trial109.stdout:[mon] 2026-02-20T11:14:39.276 INFO:teuthology.orchestra.run.trial109.stdout: debug ms = 1 2026-02-20T11:14:39.276 INFO:teuthology.orchestra.run.trial109.stdout: debug mon = 20 2026-02-20T11:14:39.276 INFO:teuthology.orchestra.run.trial109.stdout: debug paxos = 20 2026-02-20T11:14:39.277 INFO:teuthology.orchestra.run.trial109.stdout: debug auth = 20 2026-02-20T11:14:39.277 INFO:teuthology.orchestra.run.trial109.stdout: mon data avail warn = 5 2026-02-20T11:14:39.277 INFO:teuthology.orchestra.run.trial109.stdout: mon mgr mkfs grace = 240 2026-02-20T11:14:39.277 INFO:teuthology.orchestra.run.trial109.stdout: mon reweight min pgs per osd = 4 2026-02-20T11:14:39.277 INFO:teuthology.orchestra.run.trial109.stdout: mon osd reporter subtree level = osd 2026-02-20T11:14:39.277 INFO:teuthology.orchestra.run.trial109.stdout: mon osd prime pg temp = true 2026-02-20T11:14:39.277 INFO:teuthology.orchestra.run.trial109.stdout: mon reweight min bytes per osd = 10 2026-02-20T11:14:39.277 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:14:39.277 INFO:teuthology.orchestra.run.trial109.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-20T11:14:39.277 INFO:teuthology.orchestra.run.trial109.stdout: auth mon ticket ttl = 660 # 11m 2026-02-20T11:14:39.277 INFO:teuthology.orchestra.run.trial109.stdout: auth service ticket ttl = 240 # 4m 2026-02-20T11:14:39.277 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:14:39.277 INFO:teuthology.orchestra.run.trial109.stdout: # don't complain about insecure global_id in the test suite 2026-02-20T11:14:39.278 INFO:teuthology.orchestra.run.trial109.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-20T11:14:39.278 INFO:teuthology.orchestra.run.trial109.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-20T11:14:39.278 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:14:39.278 INFO:teuthology.orchestra.run.trial109.stdout: # 1m isn't quite enough 2026-02-20T11:14:39.278 INFO:teuthology.orchestra.run.trial109.stdout: mon_down_mkfs_grace = 2m 2026-02-20T11:14:39.278 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:14:39.278 INFO:teuthology.orchestra.run.trial109.stdout: mon_warn_on_filestore_osds = false 2026-02-20T11:14:39.278 INFO:teuthology.orchestra.run.trial109.stdout: mon min osdmap epochs = 50 2026-02-20T11:14:39.278 INFO:teuthology.orchestra.run.trial109.stdout: mon osdmap full prune interval = 2 2026-02-20T11:14:39.278 INFO:teuthology.orchestra.run.trial109.stdout: mon osdmap full prune min = 15 2026-02-20T11:14:39.278 INFO:teuthology.orchestra.run.trial109.stdout: mon osdmap full prune txsize = 2 2026-02-20T11:14:39.278 INFO:teuthology.orchestra.run.trial109.stdout: osd pool default crimson = True 2026-02-20T11:14:39.278 INFO:teuthology.orchestra.run.trial109.stdout: paxos service trim min = 10 2026-02-20T11:14:39.279 INFO:teuthology.orchestra.run.trial109.stdout: 2026-02-20T11:14:39.279 INFO:teuthology.orchestra.run.trial109.stdout:[client] 2026-02-20T11:14:39.279 INFO:teuthology.orchestra.run.trial109.stdout: rgw cache enabled = true 2026-02-20T11:14:39.279 INFO:teuthology.orchestra.run.trial109.stdout: rgw enable ops log = true 2026-02-20T11:14:39.279 INFO:teuthology.orchestra.run.trial109.stdout: rgw enable usage log = true 2026-02-20T11:14:39.279 INFO:teuthology.orchestra.run.trial109.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-20T11:14:39.279 INFO:teuthology.orchestra.run.trial109.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-20T11:14:39.279 INFO:teuthology.orchestra.run.trial109.stdout:[mon.a] 2026-02-20T11:14:39.279 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-20T11:14:39.280 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-20T11:14:39.355 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-20T11:14:39.355 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-20T11:14:39.355 DEBUG:teuthology.orchestra.run.trial043:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-20T11:14:39.372 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-20T11:14:39.372 DEBUG:teuthology.orchestra.run.trial043:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-20T11:14:39.429 INFO:tasks.ceph:Sending monmap to node ubuntu@trial043.front.sepia.ceph.com 2026-02-20T11:14:39.430 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-20T11:14:39.430 DEBUG:teuthology.orchestra.run.trial043:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-20T11:14:39.430 DEBUG:teuthology.orchestra.run.trial043:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-20T11:14:39.504 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-20T11:14:39.504 DEBUG:teuthology.orchestra.run.trial043:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-20T11:14:39.560 INFO:tasks.ceph:Sending monmap to node ubuntu@trial109.front.sepia.ceph.com 2026-02-20T11:14:39.560 DEBUG:teuthology.orchestra.run.trial109:> set -ex 2026-02-20T11:14:39.560 DEBUG:teuthology.orchestra.run.trial109:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-20T11:14:39.560 DEBUG:teuthology.orchestra.run.trial109:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-20T11:14:39.594 DEBUG:teuthology.orchestra.run.trial109:> set -ex 2026-02-20T11:14:39.595 DEBUG:teuthology.orchestra.run.trial109:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-20T11:14:39.651 INFO:tasks.ceph:Setting up mon nodes... 2026-02-20T11:14:39.651 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-20T11:14:39.651 DEBUG:teuthology.orchestra.run.trial109:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-20T11:14:39.741 INFO:teuthology.orchestra.run.trial109.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-20T11:14:39.743 INFO:tasks.ceph:Setting up mds nodes... 2026-02-20T11:14:39.744 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-20T11:14:39.744 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-20T11:14:39.787 INFO:teuthology.orchestra.run.trial043.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-20T11:14:39.800 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2026-02-20T11:14:39.841 INFO:teuthology.orchestra.run.trial109.stdout:creating /etc/ceph/ceph.client.1.keyring 2026-02-20T11:14:39.854 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-20T11:14:39.854 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial043.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@trial109.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2026-02-20T11:14:39.854 DEBUG:teuthology.orchestra.run.trial043:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-20T11:14:39.880 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2026-02-20T11:14:39.880 INFO:tasks.ceph:role: osd.0 2026-02-20T11:14:39.880 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial043.front.sepia.ceph.com 2026-02-20T11:14:39.881 DEBUG:teuthology.orchestra.run.trial043:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-20T11:14:39.944 INFO:teuthology.orchestra.run.trial043.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-20T11:14:39.945 INFO:teuthology.orchestra.run.trial043.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-20T11:14:39.945 INFO:teuthology.orchestra.run.trial043.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-20T11:14:39.945 INFO:teuthology.orchestra.run.trial043.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-20T11:14:39.945 INFO:teuthology.orchestra.run.trial043.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-20T11:14:39.945 INFO:teuthology.orchestra.run.trial043.stdout: = sunit=0 swidth=0 blks 2026-02-20T11:14:39.945 INFO:teuthology.orchestra.run.trial043.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-20T11:14:39.945 INFO:teuthology.orchestra.run.trial043.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-20T11:14:39.945 INFO:teuthology.orchestra.run.trial043.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-20T11:14:39.945 INFO:teuthology.orchestra.run.trial043.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-20T11:14:40.517 INFO:teuthology.orchestra.run.trial043.stdout:Discarding blocks...Done. 2026-02-20T11:14:40.520 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial043.front.sepia.ceph.com -o noatime 2026-02-20T11:14:40.520 DEBUG:teuthology.orchestra.run.trial043:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-20T11:14:40.588 DEBUG:teuthology.orchestra.run.trial043:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-20T11:14:40.654 DEBUG:teuthology.orchestra.run.trial043:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-20T11:14:40.720 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2026-02-20T11:14:40.720 INFO:tasks.ceph:role: osd.1 2026-02-20T11:14:40.720 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial043.front.sepia.ceph.com 2026-02-20T11:14:40.720 DEBUG:teuthology.orchestra.run.trial043:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-20T11:14:40.785 INFO:teuthology.orchestra.run.trial043.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-20T11:14:40.785 INFO:teuthology.orchestra.run.trial043.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-20T11:14:40.785 INFO:teuthology.orchestra.run.trial043.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-20T11:14:40.786 INFO:teuthology.orchestra.run.trial043.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-20T11:14:40.786 INFO:teuthology.orchestra.run.trial043.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-20T11:14:40.786 INFO:teuthology.orchestra.run.trial043.stdout: = sunit=0 swidth=0 blks 2026-02-20T11:14:40.786 INFO:teuthology.orchestra.run.trial043.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-20T11:14:40.786 INFO:teuthology.orchestra.run.trial043.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-20T11:14:40.786 INFO:teuthology.orchestra.run.trial043.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-20T11:14:40.786 INFO:teuthology.orchestra.run.trial043.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-20T11:14:41.346 INFO:teuthology.orchestra.run.trial043.stdout:Discarding blocks...Done. 2026-02-20T11:14:41.348 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial043.front.sepia.ceph.com -o noatime 2026-02-20T11:14:41.348 DEBUG:teuthology.orchestra.run.trial043:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-20T11:14:41.416 DEBUG:teuthology.orchestra.run.trial043:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-20T11:14:41.483 DEBUG:teuthology.orchestra.run.trial043:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-20T11:14:41.634 INFO:teuthology.orchestra.run.trial043.stderr:INFO  osd - OSD::main: parsing early config 2026-02-20T11:14:41.692 INFO:teuthology.orchestra.run.trial043.stderr:INFO 2026-02-20 11:14:41,693 seastar - Reactor backend: linux-aio 2026-02-20T11:14:41.702 INFO:teuthology.orchestra.run.trial043.stderr:INFO 2026-02-20 11:14:41,703 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T11:14:41.702 INFO:teuthology.orchestra.run.trial043.stderr:INFO 2026-02-20 11:14:41,703 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T11:14:41.704 INFO:teuthology.orchestra.run.trial043.stderr:==39468==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-20T11:14:41.761 INFO:teuthology.orchestra.run.trial043.stderr:INFO 2026-02-20 11:14:41,761 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-20T11:14:41.761 INFO:teuthology.orchestra.run.trial043.stderr:INFO 2026-02-20 11:14:41,761 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-20T11:14:41.761 INFO:teuthology.orchestra.run.trial043.stderr:INFO 2026-02-20 11:14:41,761 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-20T11:14:41.761 INFO:teuthology.orchestra.run.trial043.stderr:INFO 2026-02-20 11:14:41,761 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-20T11:14:41.761 INFO:teuthology.orchestra.run.trial043.stderr:DEBUG 2026-02-20 11:14:41,762 [shard 0:main] seastar - reactor::drain 2026-02-20T11:14:43.813 INFO:teuthology.orchestra.run.trial043.stderr:----------------------------------------------------- 2026-02-20T11:14:43.813 INFO:teuthology.orchestra.run.trial043.stderr:Suppressions used: 2026-02-20T11:14:43.813 INFO:teuthology.orchestra.run.trial043.stderr: count bytes template 2026-02-20T11:14:43.813 INFO:teuthology.orchestra.run.trial043.stderr: 1 8 MallocExtension::Register 2026-02-20T11:14:43.813 INFO:teuthology.orchestra.run.trial043.stderr:----------------------------------------------------- 2026-02-20T11:14:43.813 INFO:teuthology.orchestra.run.trial043.stderr: 2026-02-20T11:14:43.834 INFO:teuthology.orchestra.run.trial043.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-20T11:14:43.834 INFO:teuthology.orchestra.run.trial043.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-20T11:14:43.877 INFO:teuthology.orchestra.run.trial043.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-20T11:14:43.878 INFO:teuthology.orchestra.run.trial043.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-20T11:14:43.884 INFO:teuthology.orchestra.run.trial043.stderr:INFO 2026-02-20 11:14:43,885 seastar - Reactor backend: linux-aio 2026-02-20T11:14:43.894 INFO:teuthology.orchestra.run.trial043.stderr:INFO 2026-02-20 11:14:43,895 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T11:14:43.895 INFO:teuthology.orchestra.run.trial043.stderr:INFO 2026-02-20 11:14:43,895 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T11:14:43.898 INFO:teuthology.orchestra.run.trial043.stderr:==39466==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-20T11:14:43.898 INFO:teuthology.orchestra.run.trial043.stderr:INFO 2026-02-20 11:14:43,899 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-20T11:14:43.958 INFO:teuthology.orchestra.run.trial043.stderr:DEBUG 2026-02-20 11:14:43,958 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-20T11:14:44.037 INFO:teuthology.orchestra.run.trial043.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0x10a36cd 0xf4865a 0x9e1e2c5 0x9e1e78f 0x9e1ea02 0x9e20729 0x9f0b606 0x9f0d258 0x9f0d8b2 0x9f0ddf8 0x9f18086 0x9f1c4ff 0x9edc6d1 0x9edb6d1 0x7b9c74f 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f75f2c 0x6f79bf7 0x6f7f709 0x73d6e81 0x73d6f06 0x73d7008 0x73d7100 0x74a009e 0x7246aa6 0x727c254 0x73c1550 0x7413540 0x7246b2d 0x727c45b 0x756db47 0x73d7185 0x73d7263 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:14:44.038 INFO:teuthology.orchestra.run.trial043.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0x9e20645 0x9e20c76 0x9e20e75 0x9f0d69d 0x9f0ddf8 0x9f18086 0x9f1c4ff 0x9edc6d1 0x9edb6d1 0x7b9c74f 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f75f2c 0x6f79bf7 0x6f7f709 0x73d6e81 0x73d6f06 0x73d7008 0x73d7100 0x74a009e 0x7246aa6 0x727c254 0x73c1550 0x7413540 0x7246b2d 0x727c45b 0x756db47 0x73d7185 0x73d7263 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:14:44.038 INFO:teuthology.orchestra.run.trial043.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x151ab 0x31fb0c9 0x9e1e9f0 0x9e20729 0x9e20c76 0x9e20e75 0x9f0d69d 0x9f0ddf8 0x9f18086 0x9f1c4ff 0x9edc6d1 0x9edb6d1 0x7b9c74f 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f75f2c 0x6f79bf7 0x6f7f709 0x73d6e81 0x73d6f06 0x73d7008 0x73d7100 0x74a009e 0x7246aa6 0x727c254 0x73c1550 0x7413540 0x7246b2d 0x727c45b 0x756db47 0x73d7185 0x73d7263 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:14:44.191 INFO:teuthology.orchestra.run.trial043.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0x9e2762b 0x9e27e01 0x9f17e54 0x9f1c4ff 0x9edc6d1 0x9edb6d1 0x7b9c74f 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f2e256 0x6f30572 0x73d4684 0x73d4709 0x73d480b 0x73d4903 0x749ff22 0x7246045 0x727559c 0x73bfa6b 0x740df81 0x72460cc 0x72757a3 0x756b847 0x73d4988 0x73d4a66 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:14:44.191 INFO:teuthology.orchestra.run.trial043.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x341f /lib64/libasan.so.8+0x15197 0x31fb0c9 0x9e1e9f0 0x9e1ecb8 0x9e1f27a 0x9e27721 0x9e27e01 0x9f17e54 0x9f1c4ff 0x9edc6d1 0x9edb6d1 0x7b9c74f 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f2e256 0x6f30572 0x73d4684 0x73d4709 0x73d480b 0x73d4903 0x749ff22 0x7246045 0x727559c 0x73bfa6b 0x740df81 0x72460cc 0x72757a3 0x756b847 0x73d4988 0x73d4a66 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:14:44.191 INFO:teuthology.orchestra.run.trial043.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xe419a1 0x9d704b4 0x9d705f8 0x9d706dd 0x9d70939 0x9d70a2c 0x9d70b08 0x9d70d84 0x9d7132b 0x9d7172f 0x9d719ec 0x9f1a6cd 0x9edc6d1 0x9edb6d1 0x7b9c74f 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f2e256 0x6f30572 0x73d4684 0x73d4709 0x73d480b 0x73d4903 0x749ff22 0x7246045 0x727559c 0x73bfa6b 0x740df81 0x72460cc 0x72757a3 0x756b847 0x73d4988 0x73d4a66 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:14:44.191 INFO:teuthology.orchestra.run.trial043.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffa1a1f6b4 0xffffffffa1a1fb20 0xffffffffa189525e 0xffffffffa1be61a9 0xffffffffa1bc9715 0xffffffffa1bd3aeb 0xffffffffa1bd3f4b 0xffffffffa18910d2 0xffffffffa254d451 0xffffffffa2600ba2 2026-02-20T11:14:46.271 INFO:teuthology.orchestra.run.trial043.stderr:----------------------------------------------------- 2026-02-20T11:14:46.271 INFO:teuthology.orchestra.run.trial043.stderr:Suppressions used: 2026-02-20T11:14:46.271 INFO:teuthology.orchestra.run.trial043.stderr: count bytes template 2026-02-20T11:14:46.271 INFO:teuthology.orchestra.run.trial043.stderr: 1 8 MallocExtension::Register 2026-02-20T11:14:46.271 INFO:teuthology.orchestra.run.trial043.stderr:----------------------------------------------------- 2026-02-20T11:14:46.271 INFO:teuthology.orchestra.run.trial043.stderr: 2026-02-20T11:14:46.303 DEBUG:teuthology.orchestra.run.trial043:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-20T11:14:46.329 DEBUG:teuthology.orchestra.run.trial043:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-20T11:14:46.481 INFO:teuthology.orchestra.run.trial043.stderr:INFO  osd - OSD::main: parsing early config 2026-02-20T11:14:46.541 INFO:teuthology.orchestra.run.trial043.stderr:INFO 2026-02-20 11:14:46,542 seastar - Reactor backend: linux-aio 2026-02-20T11:14:46.554 INFO:teuthology.orchestra.run.trial043.stderr:INFO 2026-02-20 11:14:46,555 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T11:14:46.554 INFO:teuthology.orchestra.run.trial043.stderr:INFO 2026-02-20 11:14:46,555 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T11:14:46.556 INFO:teuthology.orchestra.run.trial043.stderr:==39534==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-20T11:14:46.613 INFO:teuthology.orchestra.run.trial043.stderr:INFO 2026-02-20 11:14:46,614 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-20T11:14:46.613 INFO:teuthology.orchestra.run.trial043.stderr:INFO 2026-02-20 11:14:46,614 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-20T11:14:46.613 INFO:teuthology.orchestra.run.trial043.stderr:INFO 2026-02-20 11:14:46,614 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-20T11:14:46.613 INFO:teuthology.orchestra.run.trial043.stderr:INFO 2026-02-20 11:14:46,614 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-20T11:14:46.613 INFO:teuthology.orchestra.run.trial043.stderr:DEBUG 2026-02-20 11:14:46,614 [shard 0:main] seastar - reactor::drain 2026-02-20T11:14:48.679 INFO:teuthology.orchestra.run.trial043.stderr:----------------------------------------------------- 2026-02-20T11:14:48.679 INFO:teuthology.orchestra.run.trial043.stderr:Suppressions used: 2026-02-20T11:14:48.680 INFO:teuthology.orchestra.run.trial043.stderr: count bytes template 2026-02-20T11:14:48.680 INFO:teuthology.orchestra.run.trial043.stderr: 1 8 MallocExtension::Register 2026-02-20T11:14:48.680 INFO:teuthology.orchestra.run.trial043.stderr:----------------------------------------------------- 2026-02-20T11:14:48.680 INFO:teuthology.orchestra.run.trial043.stderr: 2026-02-20T11:14:48.702 INFO:teuthology.orchestra.run.trial043.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-20T11:14:48.702 INFO:teuthology.orchestra.run.trial043.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-20T11:14:48.744 INFO:teuthology.orchestra.run.trial043.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-20T11:14:48.744 INFO:teuthology.orchestra.run.trial043.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-20T11:14:48.750 INFO:teuthology.orchestra.run.trial043.stderr:INFO 2026-02-20 11:14:48,751 seastar - Reactor backend: linux-aio 2026-02-20T11:14:48.761 INFO:teuthology.orchestra.run.trial043.stderr:INFO 2026-02-20 11:14:48,761 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T11:14:48.761 INFO:teuthology.orchestra.run.trial043.stderr:INFO 2026-02-20 11:14:48,761 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T11:14:48.764 INFO:teuthology.orchestra.run.trial043.stderr:==39532==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-20T11:14:48.765 INFO:teuthology.orchestra.run.trial043.stderr:INFO 2026-02-20 11:14:48,765 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-20T11:14:48.827 INFO:teuthology.orchestra.run.trial043.stderr:DEBUG 2026-02-20 11:14:48,827 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-20T11:14:48.908 INFO:teuthology.orchestra.run.trial043.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0x10a35c5 0xf4865a 0x9e1e492 0x9e1e78f 0x9e1ea02 0x9e1ecb8 0x9e1f27a 0x9e27721 0x9e27e01 0x9f17e54 0x9f1c4ff 0x9edc6d1 0x9edb6d1 0x7b9c74f 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f75f2c 0x6f79bf7 0x6f7f709 0x73d6e81 0x73d6f06 0x73d7008 0x73d7100 0x74a009e 0x7246aa6 0x727c254 0x73c1550 0x7413540 0x7246b2d 0x727c45b 0x756db47 0x73d7185 0x73d7263 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:14:48.909 INFO:teuthology.orchestra.run.trial043.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x359d229 0x9e30705 0x9e307c7 0x9ef7d44 0x9ef7ed9 0x9ef826b 0x9ef9851 0x9efab6c 0x9efaf8c 0x9f17817 0x9f1c4ff 0x9edc6d1 0x9edb6d1 0x7b9c74f 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f75f2c 0x6f79bf7 0x6f7f709 0x73d6e81 0x73d6f06 0x73d7008 0x73d7100 0x74a009e 0x7246aa6 0x727c254 0x73c1550 0x7413540 0x7246b2d 0x727c45b 0x756db47 0x73d7185 0x73d7263 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:14:48.909 INFO:teuthology.orchestra.run.trial043.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x11904b /lib64/libasan.so.8+0xc5f4 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x9cf0b5b 0x9ee0e7e 0x9ee0fac 0x9f137b3 0x9f13f56 0x9f1e423 0x78bcf19 0x7b9c686 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f75f2c 0x6f79bf7 0x6f7f709 0x73d6e81 0x73d6f06 0x73d7008 0x73d7100 0x74a009e 0x7246aa6 0x727c254 0x73c1550 0x7413540 0x7246b2d 0x727c45b 0x756db47 0x73d7185 0x73d7263 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:14:49.063 INFO:teuthology.orchestra.run.trial043.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0x31fb131 0x9e1e943 0x9e1fa02 0x9e1ff70 0x9e20295 0x9e203ff 0x9f07c75 0x9f1aa78 0x9edc6d1 0x9edb6d1 0x7b9c74f 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f2e256 0x6f30572 0x73d4684 0x73d4709 0x73d480b 0x73d4903 0x749ff22 0x7246045 0x727559c 0x73bfa6b 0x740df81 0x72460cc 0x72757a3 0x756b847 0x73d4988 0x73d4a66 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:14:49.064 INFO:teuthology.orchestra.run.trial043.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0x9f0fdfc 0x9f13918 0x9f13f56 0x9f1e423 0x78bcf19 0x7b9c686 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f2e256 0x6f30572 0x73d4684 0x73d4709 0x73d480b 0x73d4903 0x749ff22 0x7246045 0x727559c 0x73bfa6b 0x740df81 0x72460cc 0x72757a3 0x756b847 0x73d4988 0x73d4a66 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:14:49.064 INFO:teuthology.orchestra.run.trial043.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc589 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x9cf0b5b 0x9d70916 0x9d70a2c 0x9d70b08 0x9d70d84 0x9d7132b 0x9d7172f 0x9d719ec 0x9f1a6cd 0x9edc6d1 0x9edb6d1 0x7b9c74f 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f2e256 0x6f30572 0x73d4684 0x73d4709 0x73d480b 0x73d4903 0x749ff22 0x7246045 0x727559c 0x73bfa6b 0x740df81 0x72460cc 0x72757a3 0x756b847 0x73d4988 0x73d4a66 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:14:51.192 INFO:teuthology.orchestra.run.trial043.stderr:----------------------------------------------------- 2026-02-20T11:14:51.192 INFO:teuthology.orchestra.run.trial043.stderr:Suppressions used: 2026-02-20T11:14:51.192 INFO:teuthology.orchestra.run.trial043.stderr: count bytes template 2026-02-20T11:14:51.192 INFO:teuthology.orchestra.run.trial043.stderr: 1 8 MallocExtension::Register 2026-02-20T11:14:51.192 INFO:teuthology.orchestra.run.trial043.stderr:----------------------------------------------------- 2026-02-20T11:14:51.192 INFO:teuthology.orchestra.run.trial043.stderr: 2026-02-20T11:14:51.225 DEBUG:teuthology.orchestra.run.trial043:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-20T11:14:51.251 DEBUG:teuthology.orchestra.run.trial109:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-20T11:14:51.280 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2026-02-20T11:14:51.280 INFO:tasks.ceph:role: osd.2 2026-02-20T11:14:51.280 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial109.front.sepia.ceph.com 2026-02-20T11:14:51.280 DEBUG:teuthology.orchestra.run.trial109:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-20T11:14:51.347 INFO:teuthology.orchestra.run.trial109.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-20T11:14:51.347 INFO:teuthology.orchestra.run.trial109.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-20T11:14:51.348 INFO:teuthology.orchestra.run.trial109.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-20T11:14:51.348 INFO:teuthology.orchestra.run.trial109.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-20T11:14:51.348 INFO:teuthology.orchestra.run.trial109.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-20T11:14:51.348 INFO:teuthology.orchestra.run.trial109.stdout: = sunit=0 swidth=0 blks 2026-02-20T11:14:51.348 INFO:teuthology.orchestra.run.trial109.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-20T11:14:51.348 INFO:teuthology.orchestra.run.trial109.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-20T11:14:51.348 INFO:teuthology.orchestra.run.trial109.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-20T11:14:51.348 INFO:teuthology.orchestra.run.trial109.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-20T11:14:51.919 INFO:teuthology.orchestra.run.trial109.stdout:Discarding blocks...Done. 2026-02-20T11:14:51.921 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial109.front.sepia.ceph.com -o noatime 2026-02-20T11:14:51.921 DEBUG:teuthology.orchestra.run.trial109:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2026-02-20T11:14:51.990 DEBUG:teuthology.orchestra.run.trial109:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-20T11:14:52.058 DEBUG:teuthology.orchestra.run.trial109:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2026-02-20T11:14:52.124 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2026-02-20T11:14:52.124 INFO:tasks.ceph:role: osd.3 2026-02-20T11:14:52.124 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial109.front.sepia.ceph.com 2026-02-20T11:14:52.125 DEBUG:teuthology.orchestra.run.trial109:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-20T11:14:52.190 INFO:teuthology.orchestra.run.trial109.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-20T11:14:52.190 INFO:teuthology.orchestra.run.trial109.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-20T11:14:52.190 INFO:teuthology.orchestra.run.trial109.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-20T11:14:52.190 INFO:teuthology.orchestra.run.trial109.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-20T11:14:52.190 INFO:teuthology.orchestra.run.trial109.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-20T11:14:52.190 INFO:teuthology.orchestra.run.trial109.stdout: = sunit=0 swidth=0 blks 2026-02-20T11:14:52.190 INFO:teuthology.orchestra.run.trial109.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-20T11:14:52.191 INFO:teuthology.orchestra.run.trial109.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-20T11:14:52.191 INFO:teuthology.orchestra.run.trial109.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-20T11:14:52.191 INFO:teuthology.orchestra.run.trial109.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-20T11:14:52.750 INFO:teuthology.orchestra.run.trial109.stdout:Discarding blocks...Done. 2026-02-20T11:14:52.752 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial109.front.sepia.ceph.com -o noatime 2026-02-20T11:14:52.753 DEBUG:teuthology.orchestra.run.trial109:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-3 2026-02-20T11:14:52.821 DEBUG:teuthology.orchestra.run.trial109:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2026-02-20T11:14:52.888 DEBUG:teuthology.orchestra.run.trial109:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-20T11:14:53.042 INFO:teuthology.orchestra.run.trial109.stderr:INFO  osd - OSD::main: parsing early config 2026-02-20T11:14:53.102 INFO:teuthology.orchestra.run.trial109.stderr:INFO 2026-02-20 11:14:53,101 seastar - Reactor backend: linux-aio 2026-02-20T11:14:53.116 INFO:teuthology.orchestra.run.trial109.stderr:INFO 2026-02-20 11:14:53,115 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T11:14:53.116 INFO:teuthology.orchestra.run.trial109.stderr:INFO 2026-02-20 11:14:53,115 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T11:14:53.118 INFO:teuthology.orchestra.run.trial109.stderr:==38891==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-20T11:14:53.175 INFO:teuthology.orchestra.run.trial109.stderr:INFO 2026-02-20 11:14:53,173 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-20T11:14:53.175 INFO:teuthology.orchestra.run.trial109.stderr:INFO 2026-02-20 11:14:53,173 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-20T11:14:53.175 INFO:teuthology.orchestra.run.trial109.stderr:INFO 2026-02-20 11:14:53,173 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-20T11:14:53.175 INFO:teuthology.orchestra.run.trial109.stderr:INFO 2026-02-20 11:14:53,173 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-20T11:14:53.175 INFO:teuthology.orchestra.run.trial109.stderr:DEBUG 2026-02-20 11:14:53,174 [shard 0:main] seastar - reactor::drain 2026-02-20T11:14:55.241 INFO:teuthology.orchestra.run.trial109.stderr:----------------------------------------------------- 2026-02-20T11:14:55.241 INFO:teuthology.orchestra.run.trial109.stderr:Suppressions used: 2026-02-20T11:14:55.241 INFO:teuthology.orchestra.run.trial109.stderr: count bytes template 2026-02-20T11:14:55.241 INFO:teuthology.orchestra.run.trial109.stderr: 1 8 MallocExtension::Register 2026-02-20T11:14:55.241 INFO:teuthology.orchestra.run.trial109.stderr:----------------------------------------------------- 2026-02-20T11:14:55.241 INFO:teuthology.orchestra.run.trial109.stderr: 2026-02-20T11:14:55.260 INFO:teuthology.orchestra.run.trial109.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-20T11:14:55.261 INFO:teuthology.orchestra.run.trial109.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-20T11:14:55.302 INFO:teuthology.orchestra.run.trial109.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-20T11:14:55.302 INFO:teuthology.orchestra.run.trial109.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-20T11:14:55.308 INFO:teuthology.orchestra.run.trial109.stderr:INFO 2026-02-20 11:14:55,307 seastar - Reactor backend: linux-aio 2026-02-20T11:14:55.320 INFO:teuthology.orchestra.run.trial109.stderr:INFO 2026-02-20 11:14:55,318 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T11:14:55.320 INFO:teuthology.orchestra.run.trial109.stderr:INFO 2026-02-20 11:14:55,318 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T11:14:55.324 INFO:teuthology.orchestra.run.trial109.stderr:==38889==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-20T11:14:55.324 INFO:teuthology.orchestra.run.trial109.stderr:INFO 2026-02-20 11:14:55,323 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-20T11:14:55.383 INFO:teuthology.orchestra.run.trial109.stderr:DEBUG 2026-02-20 11:14:55,382 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-20T11:14:55.462 INFO:teuthology.orchestra.run.trial109.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0x9f069d7 0x9f07573 0x9f07a5e 0x9f07df1 0x9f1aa78 0x9edc6d1 0x9edb6d1 0x7b9c74f 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f75f2c 0x6f79bf7 0x6f7f709 0x73d6e81 0x73d6f06 0x73d7008 0x73d7100 0x74a009e 0x7246aa6 0x727c254 0x73c1550 0x7413540 0x7246b2d 0x727c45b 0x756db47 0x73d7185 0x73d7263 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:14:55.463 INFO:teuthology.orchestra.run.trial109.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x756a5f9 0x72714ec 0x712f14a 0x7b9c3d3 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f75f2c 0x6f79bf7 0x6f7f709 0x73d6e81 0x73d6f06 0x73d7008 0x73d7100 0x74a009e 0x7246aa6 0x727c254 0x73c1550 0x7413540 0x7246b2d 0x727c45b 0x756db47 0x73d7185 0x73d7263 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:14:55.463 INFO:teuthology.orchestra.run.trial109.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff94475c69 0xffffffff9461f8c7 0xffffffff9461fb20 0xffffffff9449525e 0xffffffff947e61a9 0xffffffff947c9715 0xffffffff947d3aeb 0xffffffff947d3f4b 0xffffffff944910d2 0xffffffff9514d451 0xffffffff95200ba2 2026-02-20T11:14:55.463 INFO:teuthology.orchestra.run.trial109.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0xccaa89 0x10a369e 0xf4865a 0x9e1e492 0x9e1e78f 0x9e1ea02 0x9e20729 0x9e20c76 0x9e20e75 0x9f0d69d 0x9f0ddf8 0x9f18086 0x9f1c4ff 0x9edc6d1 0x9edb6d1 0x7b9c74f 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f75f2c 0x6f79bf7 0x6f7f709 0x73d6e81 0x73d6f06 0x73d7008 0x73d7100 0x74a009e 0x7246aa6 0x727c254 0x73c1550 0x7413540 0x7246b2d 0x727c45b 0x756db47 0x73d7185 0x73d7263 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:14:55.612 INFO:teuthology.orchestra.run.trial109.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1583b 0x10a3723 0xf4865a 0x9e1e492 0x9e1e78f 0x9e1ea02 0x9e1fa02 0x9e27a85 0x9e27e01 0x9f17e54 0x9f1c4ff 0x9edc6d1 0x9edb6d1 0x7b9c74f 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f2e256 0x6f30572 0x73d4684 0x73d4709 0x73d480b 0x73d4903 0x749ff22 0x7246045 0x727559c 0x73bfa6b 0x740df81 0x72460cc 0x72757a3 0x756b847 0x73d4988 0x73d4a66 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:14:55.612 INFO:teuthology.orchestra.run.trial109.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x756a5f9 0x72714ec 0x712f14a 0x7b9bdd8 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f2e256 0x6f30572 0x73d4684 0x73d4709 0x73d480b 0x73d4903 0x749ff22 0x7246045 0x727559c 0x73bfa6b 0x740df81 0x72460cc 0x72757a3 0x756b847 0x73d4988 0x73d4a66 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:14:55.612 INFO:teuthology.orchestra.run.trial109.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff9461f6b4 0xffffffff9461fb20 0xffffffff9449525e 0xffffffff947e61a9 0xffffffff947c9715 0xffffffff947d3aeb 0xffffffff947d3f4b 0xffffffff944910d2 0xffffffff9514d451 0xffffffff95200ba2 2026-02-20T11:14:55.613 INFO:teuthology.orchestra.run.trial109.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xe93d /lib64/libasan.so.8+0xec776 0xcc3105 0x9f19117 0x9f1c4ff 0x9edc6d1 0x9edb6d1 0x7b9c74f 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f2e256 0x6f30572 0x73d4684 0x73d4709 0x73d480b 0x73d4903 0x749ff22 0x7246045 0x727559c 0x73bfa6b 0x740df81 0x72460cc 0x72757a3 0x756b847 0x73d4988 0x73d4a66 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:14:55.613 INFO:teuthology.orchestra.run.trial109.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff9459ce99 0xffffffff947d0ce3 0xffffffff944910a4 0xffffffff9514d451 0xffffffff95200ba2 2026-02-20T11:14:57.749 INFO:teuthology.orchestra.run.trial109.stderr:----------------------------------------------------- 2026-02-20T11:14:57.749 INFO:teuthology.orchestra.run.trial109.stderr:Suppressions used: 2026-02-20T11:14:57.750 INFO:teuthology.orchestra.run.trial109.stderr: count bytes template 2026-02-20T11:14:57.750 INFO:teuthology.orchestra.run.trial109.stderr: 1 8 MallocExtension::Register 2026-02-20T11:14:57.750 INFO:teuthology.orchestra.run.trial109.stderr:----------------------------------------------------- 2026-02-20T11:14:57.750 INFO:teuthology.orchestra.run.trial109.stderr: 2026-02-20T11:14:57.784 DEBUG:teuthology.orchestra.run.trial109:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-20T11:14:57.809 DEBUG:teuthology.orchestra.run.trial109:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-20T11:14:57.962 INFO:teuthology.orchestra.run.trial109.stderr:INFO  osd - OSD::main: parsing early config 2026-02-20T11:14:58.024 INFO:teuthology.orchestra.run.trial109.stderr:INFO 2026-02-20 11:14:58,023 seastar - Reactor backend: linux-aio 2026-02-20T11:14:58.037 INFO:teuthology.orchestra.run.trial109.stderr:INFO 2026-02-20 11:14:58,036 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T11:14:58.038 INFO:teuthology.orchestra.run.trial109.stderr:INFO 2026-02-20 11:14:58,036 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T11:14:58.039 INFO:teuthology.orchestra.run.trial109.stderr:==38955==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-20T11:14:58.096 INFO:teuthology.orchestra.run.trial109.stderr:INFO 2026-02-20 11:14:58,094 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-20T11:14:58.096 INFO:teuthology.orchestra.run.trial109.stderr:INFO 2026-02-20 11:14:58,094 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-20T11:14:58.096 INFO:teuthology.orchestra.run.trial109.stderr:INFO 2026-02-20 11:14:58,094 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-20T11:14:58.096 INFO:teuthology.orchestra.run.trial109.stderr:INFO 2026-02-20 11:14:58,094 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-20T11:14:58.096 INFO:teuthology.orchestra.run.trial109.stderr:DEBUG 2026-02-20 11:14:58,095 [shard 0:main] seastar - reactor::drain 2026-02-20T11:15:00.155 INFO:teuthology.orchestra.run.trial109.stderr:----------------------------------------------------- 2026-02-20T11:15:00.155 INFO:teuthology.orchestra.run.trial109.stderr:Suppressions used: 2026-02-20T11:15:00.155 INFO:teuthology.orchestra.run.trial109.stderr: count bytes template 2026-02-20T11:15:00.155 INFO:teuthology.orchestra.run.trial109.stderr: 1 8 MallocExtension::Register 2026-02-20T11:15:00.155 INFO:teuthology.orchestra.run.trial109.stderr:----------------------------------------------------- 2026-02-20T11:15:00.156 INFO:teuthology.orchestra.run.trial109.stderr: 2026-02-20T11:15:00.175 INFO:teuthology.orchestra.run.trial109.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-20T11:15:00.175 INFO:teuthology.orchestra.run.trial109.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-20T11:15:00.230 INFO:teuthology.orchestra.run.trial109.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-20T11:15:00.231 INFO:teuthology.orchestra.run.trial109.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-20T11:15:00.237 INFO:teuthology.orchestra.run.trial109.stderr:INFO 2026-02-20 11:15:00,236 seastar - Reactor backend: linux-aio 2026-02-20T11:15:00.247 INFO:teuthology.orchestra.run.trial109.stderr:INFO 2026-02-20 11:15:00,246 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T11:15:00.248 INFO:teuthology.orchestra.run.trial109.stderr:INFO 2026-02-20 11:15:00,246 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T11:15:00.251 INFO:teuthology.orchestra.run.trial109.stderr:==38953==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-20T11:15:00.252 INFO:teuthology.orchestra.run.trial109.stderr:INFO 2026-02-20 11:15:00,251 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-20T11:15:00.313 INFO:teuthology.orchestra.run.trial109.stderr:DEBUG 2026-02-20 11:15:00,312 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-20T11:15:00.393 INFO:teuthology.orchestra.run.trial109.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xb04e9 /lib64/libasan.so.8+0xb1437 0xb7b689 0x10a36ac 0xf4865a 0x9e22cec 0x9e22f99 0x9f17991 0x9f1c4ff 0x9edc6d1 0x9edb6d1 0x7b9c74f 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f75f2c 0x6f79bf7 0x6f7f709 0x73d6e81 0x73d6f06 0x73d7008 0x73d7100 0x74a009e 0x7246aa6 0x727c254 0x73c1550 0x7413540 0x7246b2d 0x727c45b 0x756db47 0x73d7185 0x73d7263 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:15:00.393 INFO:teuthology.orchestra.run.trial109.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc589 /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xe419a1 0x9f1e3ba 0x78bcf19 0x7b9c686 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f75f2c 0x6f79bf7 0x6f7f709 0x73d6e81 0x73d6f06 0x73d7008 0x73d7100 0x74a009e 0x7246aa6 0x727c254 0x73c1550 0x7413540 0x7246b2d 0x727c45b 0x756db47 0x73d7185 0x73d7263 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:15:00.393 INFO:teuthology.orchestra.run.trial109.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff944912ca 0xffffffff9514d451 0xffffffff95200ba2 2026-02-20T11:15:00.394 INFO:teuthology.orchestra.run.trial109.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xfcaef /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x1138fc8 0xf4864f 0x9e1e492 0x9e1e78f 0x9e1ea02 0x9e1ecb8 0x9e1f767 0x9e1fc65 0x9e20295 0x9e203ff 0x9f07c75 0x9f1aa78 0x9edc6d1 0x9edb6d1 0x7b9c74f 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f75f2c 0x6f79bf7 0x6f7f709 0x73d6e81 0x73d6f06 0x73d7008 0x73d7100 0x74a009e 0x7246aa6 0x727c254 0x73c1550 0x7413540 0x7246b2d 0x727c45b 0x756db47 0x73d7185 0x73d7263 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:15:00.544 INFO:teuthology.orchestra.run.trial109.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1521e 0x1138fc8 0xf4864f 0x9f0887d 0x9f08bc2 0x9f08f45 0x9f1909c 0x9f1c4ff 0x9edc6d1 0x9edb6d1 0x7b9c74f 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f2e256 0x6f30572 0x73d4684 0x73d4709 0x73d480b 0x73d4903 0x749ff22 0x7246045 0x727559c 0x73bfa6b 0x740df81 0x72460cc 0x72757a3 0x756b847 0x73d4988 0x73d4a66 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:15:00.545 INFO:teuthology.orchestra.run.trial109.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1294d6 /lib64/libasan.so.8+0x1297ad /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x1138fc8 0xf4864f 0x9e1e492 0x9e1e78f 0x9e1ea02 0x9e20729 0x9e20c76 0x9e20e75 0x9f0d69d 0x9f0ddf8 0x9f18086 0x9f1c4ff 0x9edc6d1 0x9edb6d1 0x7b9c74f 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f2e256 0x6f30572 0x73d4684 0x73d4709 0x73d480b 0x73d4903 0x749ff22 0x7246045 0x727559c 0x73bfa6b 0x740df81 0x72460cc 0x72757a3 0x756b847 0x73d4988 0x73d4a66 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:15:00.545 INFO:teuthology.orchestra.run.trial109.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x129764 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x16507 0x9f1dac7 0x78bcf19 0x7b9c081 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f2e256 0x6f30572 0x73d4684 0x73d4709 0x73d480b 0x73d4903 0x749ff22 0x7246045 0x727559c 0x73bfa6b 0x740df81 0x72460cc 0x72757a3 0x756b847 0x73d4988 0x73d4a66 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:15:02.642 INFO:teuthology.orchestra.run.trial109.stderr:----------------------------------------------------- 2026-02-20T11:15:02.642 INFO:teuthology.orchestra.run.trial109.stderr:Suppressions used: 2026-02-20T11:15:02.642 INFO:teuthology.orchestra.run.trial109.stderr: count bytes template 2026-02-20T11:15:02.642 INFO:teuthology.orchestra.run.trial109.stderr: 1 8 MallocExtension::Register 2026-02-20T11:15:02.642 INFO:teuthology.orchestra.run.trial109.stderr:----------------------------------------------------- 2026-02-20T11:15:02.642 INFO:teuthology.orchestra.run.trial109.stderr: 2026-02-20T11:15:02.674 DEBUG:teuthology.orchestra.run.trial109:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2026-02-20T11:15:02.701 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-20T11:15:02.701 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-20T11:15:02.701 DEBUG:teuthology.orchestra.run.trial043:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-20T11:15:02.728 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-20T11:15:02.728 DEBUG:teuthology.orchestra.run.trial043:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-20T11:15:02.791 DEBUG:teuthology.orchestra.run.trial109:> set -ex 2026-02-20T11:15:02.791 DEBUG:teuthology.orchestra.run.trial109:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-20T11:15:02.817 DEBUG:teuthology.orchestra.run.trial109:> set -ex 2026-02-20T11:15:02.817 DEBUG:teuthology.orchestra.run.trial109:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-20T11:15:02.880 DEBUG:teuthology.orchestra.run.trial109:> set -ex 2026-02-20T11:15:02.880 DEBUG:teuthology.orchestra.run.trial109:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2026-02-20T11:15:02.945 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-20T11:15:02.945 DEBUG:teuthology.orchestra.run.trial043:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-20T11:15:02.963 DEBUG:teuthology.orchestra.run.trial109:> set -ex 2026-02-20T11:15:02.963 DEBUG:teuthology.orchestra.run.trial109:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2026-02-20T11:15:03.001 INFO:tasks.ceph:Adding keys to all mons... 2026-02-20T11:15:03.001 DEBUG:teuthology.orchestra.run.trial043:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-20T11:15:03.024 INFO:teuthology.orchestra.run.trial043.stdout:[osd.0] 2026-02-20T11:15:03.025 INFO:teuthology.orchestra.run.trial043.stdout: key = AQAjQphp8uxiORAAGwGCtzfhbtCmSw0xK0xRaQ== 2026-02-20T11:15:03.025 INFO:teuthology.orchestra.run.trial043.stdout:[osd.1] 2026-02-20T11:15:03.025 INFO:teuthology.orchestra.run.trial043.stdout: key = AQAoQphp2VORMRAAJPDO525v1gmwVfH4qp4H2Q== 2026-02-20T11:15:03.025 INFO:teuthology.orchestra.run.trial043.stdout:[mgr.x] 2026-02-20T11:15:03.025 INFO:teuthology.orchestra.run.trial043.stdout: key = AQAfQphpxhAcLBAAF+I8/NVgRxBEqxToJTXCtw== 2026-02-20T11:15:03.025 INFO:teuthology.orchestra.run.trial043.stdout:[osd.2] 2026-02-20T11:15:03.025 INFO:teuthology.orchestra.run.trial043.stdout: key = AQAvQphpV+MGFxAAcqrws0/yvkoCwzIp0dcLQw== 2026-02-20T11:15:03.025 INFO:teuthology.orchestra.run.trial043.stdout:[osd.3] 2026-02-20T11:15:03.025 INFO:teuthology.orchestra.run.trial043.stdout: key = AQA0Qphp55naEhAAbdZ1ptUbMtuUrKyQs/JO7g== 2026-02-20T11:15:03.025 INFO:teuthology.orchestra.run.trial043.stdout:[client.0] 2026-02-20T11:15:03.025 INFO:teuthology.orchestra.run.trial043.stdout: key = AQAfQphpDaMCLxAA/y/lNOoefGQUooL6ws1Qpg== 2026-02-20T11:15:03.026 INFO:teuthology.orchestra.run.trial043.stdout:[client.1] 2026-02-20T11:15:03.026 INFO:teuthology.orchestra.run.trial043.stdout: key = AQAfQphpme0fMhAA7gEZ61+fwRMFRjfrlOxolQ== 2026-02-20T11:15:03.027 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-20T11:15:03.115 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-20T11:15:03.163 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-20T11:15:03.212 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-20T11:15:03.261 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-20T11:15:03.310 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-20T11:15:03.360 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-20T11:15:03.411 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-20T11:15:03.411 DEBUG:teuthology.orchestra.run.trial043:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-20T11:15:03.435 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-20T11:15:03.520 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:03.520+0000 7fd24cebac00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:03.520 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:03.520+0000 7fd24cebac00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:03.532 DEBUG:teuthology.orchestra.run.trial043:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-20T11:15:03.558 DEBUG:teuthology.orchestra.run.trial043:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-20T11:15:03.614 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-20T11:15:03.614 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-20T11:15:03.614 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-20T11:15:03.656 INFO:tasks.ceph.mon.a:Started 2026-02-20T11:15:03.656 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-20T11:15:03.657 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-20T11:15:03.657 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-20T11:15:03.660 INFO:tasks.ceph.mgr.x:Started 2026-02-20T11:15:03.660 DEBUG:tasks.ceph:set 0 configs 2026-02-20T11:15:03.660 DEBUG:teuthology.orchestra.run.trial043:> sudo ceph --cluster ceph config dump 2026-02-20T11:15:03.716 INFO:tasks.ceph.mon.a.trial043.stderr:2026-02-20T11:15:03.715+0000 7fd0d862dc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:03.717 INFO:tasks.ceph.mon.a.trial043.stderr:2026-02-20T11:15:03.715+0000 7fd0d862dc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:03.717 INFO:tasks.ceph.mon.a.trial043.stderr:2026-02-20T11:15:03.715+0000 7fd0d862dc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:03.720 INFO:tasks.ceph.mgr.x.trial109.stderr:2026-02-20T11:15:03.718+0000 7f6de5c02fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:03.733 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:03.730+0000 7f25e8bdc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:03.736 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:03.736+0000 7f25e8bdc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:03.745 INFO:tasks.ceph.mgr.x.trial109.stderr:2026-02-20T11:15:03.743+0000 7f6de5c02fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:03.745 INFO:tasks.ceph.mgr.x.trial109.stderr:2026-02-20T11:15:03.743+0000 7f6de5c02fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:03.769 INFO:teuthology.orchestra.run.trial043.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-20T11:15:03.777 INFO:tasks.ceph:Setting crush tunables to default 2026-02-20T11:15:03.778 DEBUG:teuthology.orchestra.run.trial043:> sudo ceph --cluster ceph osd crush tunables default 2026-02-20T11:15:03.851 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:03.847+0000 7f898d006640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:03.853 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:03.852+0000 7f898d006640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:03.887 INFO:teuthology.orchestra.run.trial043.stderr:adjusted tunables profile to default 2026-02-20T11:15:03.895 INFO:tasks.ceph:check_enable_crimson: True 2026-02-20T11:15:03.895 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2026-02-20T11:15:03.896 DEBUG:teuthology.orchestra.run.trial043:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2026-02-20T11:15:03.969 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:03.965+0000 7f525e2cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:03.971 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:03.971+0000 7f525e2cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:04.010 INFO:tasks.ceph.mon.a.trial043.stderr:2026-02-20T11:15:04.009+0000 7fd0cf9f1640 -1 WARNING: experimental feature 'crimson' is enabled 2026-02-20T11:15:04.010 INFO:tasks.ceph.mon.a.trial043.stderr:Please be aware that this feature is experimental, untested, 2026-02-20T11:15:04.010 INFO:tasks.ceph.mon.a.trial043.stderr:unsupported, and may result in data corruption, data loss, 2026-02-20T11:15:04.011 INFO:tasks.ceph.mon.a.trial043.stderr:and/or irreparable damage to your cluster. Do not use 2026-02-20T11:15:04.011 INFO:tasks.ceph.mon.a.trial043.stderr:feature with important data. 2026-02-20T11:15:04.011 INFO:tasks.ceph.mon.a.trial043.stderr: 2026-02-20T11:15:04.579 INFO:tasks.ceph.mgr.x.trial109.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-20T11:15:04.579 INFO:tasks.ceph.mgr.x.trial109.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-20T11:15:04.579 INFO:tasks.ceph.mgr.x.trial109.stderr: from numpy import show_config as show_numpy_config 2026-02-20T11:15:04.899 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-20T11:15:04.900 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-20T11:15:04.900 DEBUG:teuthology.orchestra.run.trial043:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-20T11:15:04.925 DEBUG:teuthology.orchestra.run.trial043:> set -ex 2026-02-20T11:15:04.925 DEBUG:teuthology.orchestra.run.trial043:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-20T11:15:04.988 DEBUG:teuthology.orchestra.run.trial109:> set -ex 2026-02-20T11:15:04.988 DEBUG:teuthology.orchestra.run.trial109:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-20T11:15:05.015 DEBUG:teuthology.orchestra.run.trial109:> set -ex 2026-02-20T11:15:05.015 DEBUG:teuthology.orchestra.run.trial109:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2026-02-20T11:15:05.079 DEBUG:teuthology.orchestra.run.trial109:> sudo ceph --cluster ceph osd new a3a3446f-cb7d-471f-bcf9-06c7dd1baea1 0 2026-02-20T11:15:05.191 INFO:teuthology.orchestra.run.trial109.stderr:2026-02-20T11:15:05.186+0000 7fb67545a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:05.193 INFO:teuthology.orchestra.run.trial109.stderr:2026-02-20T11:15:05.192+0000 7fb67545a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:05.231 INFO:teuthology.orchestra.run.trial109.stdout:0 2026-02-20T11:15:05.240 DEBUG:teuthology.orchestra.run.trial109:> sudo ceph --cluster ceph osd new 8bb9e0b4-ad0f-4e8d-9815-71c91f5644c6 1 2026-02-20T11:15:05.314 INFO:teuthology.orchestra.run.trial109.stderr:2026-02-20T11:15:05.309+0000 7f91ce0cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:05.316 INFO:teuthology.orchestra.run.trial109.stderr:2026-02-20T11:15:05.314+0000 7f91ce0cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:05.353 INFO:teuthology.orchestra.run.trial109.stdout:1 2026-02-20T11:15:05.363 DEBUG:teuthology.orchestra.run.trial109:> sudo ceph --cluster ceph osd new 0c9b6f94-6c72-43b3-9987-12e16447a1df 2 2026-02-20T11:15:05.440 INFO:teuthology.orchestra.run.trial109.stderr:2026-02-20T11:15:05.434+0000 7ff467ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:05.442 INFO:teuthology.orchestra.run.trial109.stderr:2026-02-20T11:15:05.440+0000 7ff467ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:05.480 INFO:teuthology.orchestra.run.trial109.stdout:2 2026-02-20T11:15:05.488 DEBUG:teuthology.orchestra.run.trial109:> sudo ceph --cluster ceph osd new d9dfa6e5-fa90-49f6-a16c-74b687ecafef 3 2026-02-20T11:15:05.564 INFO:teuthology.orchestra.run.trial109.stderr:2026-02-20T11:15:05.558+0000 7f3f6b011640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:05.566 INFO:teuthology.orchestra.run.trial109.stderr:2026-02-20T11:15:05.564+0000 7f3f6b011640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:05.603 INFO:teuthology.orchestra.run.trial109.stdout:3 2026-02-20T11:15:05.611 DEBUG:teuthology.orchestra.run.trial043:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2026-02-20T11:15:05.641 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-20T11:15:05.642 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2026-02-20T11:15:05.684 INFO:tasks.ceph.osd.0:Started 2026-02-20T11:15:05.684 DEBUG:teuthology.orchestra.run.trial043:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2026-02-20T11:15:05.713 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-20T11:15:05.713 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2026-02-20T11:15:05.755 INFO:tasks.ceph.osd.1:Started 2026-02-20T11:15:05.756 DEBUG:teuthology.orchestra.run.trial109:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2026-02-20T11:15:05.784 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-20T11:15:05.784 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2026-02-20T11:15:05.827 INFO:tasks.ceph.osd.2:Started 2026-02-20T11:15:05.827 DEBUG:teuthology.orchestra.run.trial109:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.3.log 2026-02-20T11:15:05.857 INFO:tasks.ceph.osd.3:Restarting daemon 2026-02-20T11:15:05.857 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2026-02-20T11:15:05.901 INFO:tasks.ceph.osd.3:Started 2026-02-20T11:15:05.901 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-20T11:15:05.983 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:05.980+0000 7f1d0fd0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:05.985 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:05.985+0000 7f1d0fd0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:06.019 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:15:06.019 INFO:teuthology.orchestra.run.trial043.stdout:{"epoch":7,"fsid":"dfee6dd4-7cac-4c06-aaed-6868a4b5b9a0","created":"2026-02-20T11:15:03.729224+0000","modified":"2026-02-20T11:15:05.603559+0000","last_up_change":"0.000000","last_in_change":"2026-02-20T11:15:05.603559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"a3a3446f-cb7d-471f-bcf9-06c7dd1baea1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8bb9e0b4-ad0f-4e8d-9815-71c91f5644c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0c9b6f94-6c72-43b3-9987-12e16447a1df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d9dfa6e5-fa90-49f6-a16c-74b687ecafef","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-20T11:15:06.027 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-20T11:15:06.027 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-20T11:15:06.430 DEBUG:teuthology.orchestra.run.trial043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-20T11:15:06.497 INFO:teuthology.misc.health.trial043.stderr:2026-02-20T11:15:06.495+0000 7fa07696f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:06.500 INFO:teuthology.misc.health.trial043.stderr:2026-02-20T11:15:06.499+0000 7fa07696f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:06.533 INFO:teuthology.misc.health.trial043.stdout: 2026-02-20T11:15:06.533 INFO:teuthology.misc.health.trial043.stdout:{"epoch":7,"fsid":"dfee6dd4-7cac-4c06-aaed-6868a4b5b9a0","created":"2026-02-20T11:15:03.729224+0000","modified":"2026-02-20T11:15:05.603559+0000","last_up_change":"0.000000","last_in_change":"2026-02-20T11:15:05.603559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"a3a3446f-cb7d-471f-bcf9-06c7dd1baea1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8bb9e0b4-ad0f-4e8d-9815-71c91f5644c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0c9b6f94-6c72-43b3-9987-12e16447a1df","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d9dfa6e5-fa90-49f6-a16c-74b687ecafef","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-20T11:15:06.539 DEBUG:teuthology.misc:0 of 4 OSDs are up 2026-02-20T11:15:06.941 INFO:tasks.ceph.mgr.x.trial109.stderr:2026-02-20T11:15:06.939+0000 7f6d00bc1640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-20T11:15:12.943 DEBUG:teuthology.orchestra.run.trial043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-20T11:15:13.014 INFO:teuthology.misc.health.trial043.stderr:2026-02-20T11:15:13.011+0000 7f28420cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:13.016 INFO:teuthology.misc.health.trial043.stderr:2026-02-20T11:15:13.015+0000 7f28420cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:13.126 INFO:teuthology.misc.health.trial043.stdout: 2026-02-20T11:15:13.126 INFO:teuthology.misc.health.trial043.stdout:{"epoch":11,"fsid":"dfee6dd4-7cac-4c06-aaed-6868a4b5b9a0","created":"2026-02-20T11:15:03.729224+0000","modified":"2026-02-20T11:15:12.954498+0000","last_up_change":"2026-02-20T11:15:12.954498+0000","last_in_change":"2026-02-20T11:15:05.603559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"a3a3446f-cb7d-471f-bcf9-06c7dd1baea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.43:6803","nonce":1408480372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.43:6804","nonce":1408480372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.43:6809","nonce":1408480372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.43:6808","nonce":1408480372}]},"public_addr":"10.20.193.43:6803/1408480372","cluster_addr":"10.20.193.43:6804/1408480372","heartbeat_back_addr":"10.20.193.43:6809/1408480372","heartbeat_front_addr":"10.20.193.43:6808/1408480372","state":["exists","up"]},{"osd":1,"uuid":"8bb9e0b4-ad0f-4e8d-9815-71c91f5644c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.43:6800","nonce":18189571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.43:6801","nonce":18189571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.43:6805","nonce":18189571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.43:6802","nonce":18189571}]},"public_addr":"10.20.193.43:6800/18189571","cluster_addr":"10.20.193.43:6801/18189571","heartbeat_back_addr":"10.20.193.43:6805/18189571","heartbeat_front_addr":"10.20.193.43:6802/18189571","state":["exists","up"]},{"osd":2,"uuid":"0c9b6f94-6c72-43b3-9987-12e16447a1df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.109:6802","nonce":2255254232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.109:6805","nonce":2255254232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.109:6806","nonce":2255254232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.109:6808","nonce":2255254232}]},"public_addr":"10.20.193.109:6802/2255254232","cluster_addr":"10.20.193.109:6805/2255254232","heartbeat_back_addr":"10.20.193.109:6806/2255254232","heartbeat_front_addr":"10.20.193.109:6808/2255254232","state":["exists","up"]},{"osd":3,"uuid":"d9dfa6e5-fa90-49f6-a16c-74b687ecafef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.109:6804","nonce":829537078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.109:6803","nonce":829537078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.109:6809","nonce":829537078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.109:6807","nonce":829537078}]},"public_addr":"10.20.193.109:6804/829537078","cluster_addr":"10.20.193.109:6803/829537078","heartbeat_back_addr":"10.20.193.109:6809/829537078","heartbeat_front_addr":"10.20.193.109:6807/829537078","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-20T11:15:13.134 DEBUG:teuthology.misc:4 of 4 OSDs are up 2026-02-20T11:15:13.134 INFO:tasks.ceph:Creating RBD pool 2026-02-20T11:15:13.135 DEBUG:teuthology.orchestra.run.trial043:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-20T11:15:13.210 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:13.205+0000 7f127630b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:13.212 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:13.211+0000 7f127630b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:13.959 INFO:teuthology.orchestra.run.trial043.stderr:pool 'rbd' created 2026-02-20T11:15:13.971 DEBUG:teuthology.orchestra.run.trial043:> rbd --cluster ceph pool init rbd 2026-02-20T11:15:14.015 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:14.014+0000 7f0257595240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:14.015 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:14.014+0000 7f0257595240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:14.018 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:14.017+0000 7f0257595240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:16.999 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-20T11:15:17.000 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2026-02-20T11:15:17.001 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-20T11:15:17.116 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:17.112+0000 7f987b28b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:17.119 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:17.118+0000 7f987b28b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:17.228 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:15:17.237 INFO:teuthology.orchestra.run.trial043.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2026-02-20T11:15:17.238 INFO:tasks.ceph_manager:config epoch is 1 2026-02-20T11:15:17.238 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-20T11:15:17.238 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-20T11:15:17.238 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-20T11:15:17.313 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:17.310+0000 7f3fc381a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:17.316 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:17.315+0000 7f3fc381a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:17.457 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:15:17.466 INFO:teuthology.orchestra.run.trial043.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.109:6800","nonce":3205105008},{"type":"v1","addr":"10.20.193.109:6801","nonce":3205105008}]},"active_addr":"10.20.193.109:6801/3205105008","active_change":"2026-02-20T11:15:05.935691+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often the agent on each host will gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"The first TCP port the agent will try to bind to. Up to 1000 subsequent ports will be attempted if this port cannot be bound","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this option can facilitate debugging daemons that encounter runtime problems.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"Fraction of total system memory to divide among autotuned daemons. Converged systems hosting both Ceph and compute should set a lower value; systems with more than 10 GiB of memory per OSD may benefit from a slightly larger value so that the abundance of RAM can be fully utilized","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self-signed certificates generated and signed by the cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently Podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_alloy":{"name":"container_image_alloy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/alloy:latest","min":"","max":"","enum_allowed":[],"desc":"Alloy container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.6","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"Location of dashboards to include in Grafana","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether the hardware monitoring daemon be deployed on every host?","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at the debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of daemons per service per host: guard against mistakes and runaways","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of OSDs that will be drained simultaneously when OSDs are removed","long_desc":"","tags":[],"see_also":[]},"max_parallel_osd_upgrades":{"name":"max_parallel_osd_upgrades","type":"int","level":"advanced","flags":0,"default_value":"16","min":"","max":"","enum_allowed":[],"desc":"Maximum number of OSD daemons upgraded in parallel.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"Mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default IP address for RedFish API (OOB management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"Location of alerts to include in Prometheus Alertmanager","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry URL for login purposes. This is not the default registry.","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"Cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Customized SSH config file to distribute to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many sequential SSH connection liveness check failures before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often SSH connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"How frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use the cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest to ensure that all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":true,"error_string":"","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"healthcheck_history_max_entries":{"name":"healthcheck_history_max_entries","type":"int","level":"advanced","flags":1,"default_value":"1000","min":"","max":"","enum_allowed":[],"desc":"Maximum number of health check history entries to keep","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.109:0","nonce":1321458745}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.109:0","nonce":3398046911}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.109:0","nonce":172578844}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.109:0","nonce":2298002019}]}]} 2026-02-20T11:15:17.471 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-20T11:15:17.472 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-20T11:15:17.472 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-20T11:15:17.548 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:17.544+0000 7f1c6370b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:17.550 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:17.550+0000 7f1c6370b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:17.661 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:15:17.661 INFO:teuthology.orchestra.run.trial043.stdout:{"epoch":15,"fsid":"dfee6dd4-7cac-4c06-aaed-6868a4b5b9a0","created":"2026-02-20T11:15:03.729224+0000","modified":"2026-02-20T11:15:16.969194+0000","last_up_change":"2026-02-20T11:15:12.954498+0000","last_in_change":"2026-02-20T11:15:05.603559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-20T11:15:13.328113+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2026-02-20T11:15:13.956811+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3a3446f-cb7d-471f-bcf9-06c7dd1baea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.43:6803","nonce":1408480372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.43:6804","nonce":1408480372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.43:6809","nonce":1408480372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.43:6808","nonce":1408480372}]},"public_addr":"10.20.193.43:6803/1408480372","cluster_addr":"10.20.193.43:6804/1408480372","heartbeat_back_addr":"10.20.193.43:6809/1408480372","heartbeat_front_addr":"10.20.193.43:6808/1408480372","state":["exists","up"]},{"osd":1,"uuid":"8bb9e0b4-ad0f-4e8d-9815-71c91f5644c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.43:6800","nonce":18189571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.43:6801","nonce":18189571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.43:6805","nonce":18189571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.43:6802","nonce":18189571}]},"public_addr":"10.20.193.43:6800/18189571","cluster_addr":"10.20.193.43:6801/18189571","heartbeat_back_addr":"10.20.193.43:6805/18189571","heartbeat_front_addr":"10.20.193.43:6802/18189571","state":["exists","up"]},{"osd":2,"uuid":"0c9b6f94-6c72-43b3-9987-12e16447a1df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.109:6802","nonce":2255254232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.109:6805","nonce":2255254232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.109:6806","nonce":2255254232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.109:6808","nonce":2255254232}]},"public_addr":"10.20.193.109:6802/2255254232","cluster_addr":"10.20.193.109:6805/2255254232","heartbeat_back_addr":"10.20.193.109:6806/2255254232","heartbeat_front_addr":"10.20.193.109:6808/2255254232","state":["exists","up"]},{"osd":3,"uuid":"d9dfa6e5-fa90-49f6-a16c-74b687ecafef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.109:6804","nonce":829537078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.109:6803","nonce":829537078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.109:6809","nonce":829537078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.109:6807","nonce":829537078}]},"public_addr":"10.20.193.109:6804/829537078","cluster_addr":"10.20.193.109:6803/829537078","heartbeat_back_addr":"10.20.193.109:6809/829537078","heartbeat_front_addr":"10.20.193.109:6807/829537078","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-20T11:15:17.671 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-20T11:15:17.672 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-20T11:15:17.747 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:17.744+0000 7fc6f3bb8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:17.749 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:17.748+0000 7fc6f3bb8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:17.871 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:15:17.871 INFO:teuthology.orchestra.run.trial043.stdout:{"epoch":15,"fsid":"dfee6dd4-7cac-4c06-aaed-6868a4b5b9a0","created":"2026-02-20T11:15:03.729224+0000","modified":"2026-02-20T11:15:16.969194+0000","last_up_change":"2026-02-20T11:15:12.954498+0000","last_in_change":"2026-02-20T11:15:05.603559+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-20T11:15:13.328113+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2026-02-20T11:15:13.956811+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3a3446f-cb7d-471f-bcf9-06c7dd1baea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.43:6803","nonce":1408480372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.43:6804","nonce":1408480372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.43:6809","nonce":1408480372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.43:6808","nonce":1408480372}]},"public_addr":"10.20.193.43:6803/1408480372","cluster_addr":"10.20.193.43:6804/1408480372","heartbeat_back_addr":"10.20.193.43:6809/1408480372","heartbeat_front_addr":"10.20.193.43:6808/1408480372","state":["exists","up"]},{"osd":1,"uuid":"8bb9e0b4-ad0f-4e8d-9815-71c91f5644c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.43:6800","nonce":18189571}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.43:6801","nonce":18189571}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.43:6805","nonce":18189571}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.43:6802","nonce":18189571}]},"public_addr":"10.20.193.43:6800/18189571","cluster_addr":"10.20.193.43:6801/18189571","heartbeat_back_addr":"10.20.193.43:6805/18189571","heartbeat_front_addr":"10.20.193.43:6802/18189571","state":["exists","up"]},{"osd":2,"uuid":"0c9b6f94-6c72-43b3-9987-12e16447a1df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.109:6802","nonce":2255254232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.109:6805","nonce":2255254232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.109:6806","nonce":2255254232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.109:6808","nonce":2255254232}]},"public_addr":"10.20.193.109:6802/2255254232","cluster_addr":"10.20.193.109:6805/2255254232","heartbeat_back_addr":"10.20.193.109:6806/2255254232","heartbeat_front_addr":"10.20.193.109:6808/2255254232","state":["exists","up"]},{"osd":3,"uuid":"d9dfa6e5-fa90-49f6-a16c-74b687ecafef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.109:6804","nonce":829537078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.109:6803","nonce":829537078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.109:6809","nonce":829537078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.109:6807","nonce":829537078}]},"public_addr":"10.20.193.109:6804/829537078","cluster_addr":"10.20.193.109:6803/829537078","heartbeat_back_addr":"10.20.193.109:6809/829537078","heartbeat_front_addr":"10.20.193.109:6807/829537078","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-20T11:15:17.881 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-20T11:15:17.881 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-20T11:15:17.881 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-20T11:15:17.881 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-20T11:15:17.964 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:17.961+0000 7fb83450b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:17.964 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:17.960+0000 7f95dc25a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:17.964 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:17.961+0000 7f1a9a4cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:17.964 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:17.961+0000 7f1d7005a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:17.966 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:17.966+0000 7fb83450b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:17.966 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:17.966+0000 7f95dc25a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:17.967 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:17.966+0000 7f1d7005a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:17.967 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:17.966+0000 7f1a9a4cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:17.986 INFO:teuthology.orchestra.run.trial043.stdout:42949672966 2026-02-20T11:15:17.986 INFO:teuthology.orchestra.run.trial043.stdout:47244640261 2026-02-20T11:15:17.986 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-20T11:15:17.987 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-20T11:15:17.987 INFO:teuthology.orchestra.run.trial043.stdout:47244640261 2026-02-20T11:15:17.987 INFO:teuthology.orchestra.run.trial043.stdout:47244640261 2026-02-20T11:15:17.987 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-20T11:15:17.987 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-20T11:15:18.108 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:18.105+0000 7f5611b93640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:18.109 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:18.105+0000 7fa15743d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:18.109 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:18.105+0000 7f35e205a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:18.109 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:18.105+0000 7f2021acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:18.111 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:18.110+0000 7f5611b93640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:18.112 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:18.110+0000 7fa15743d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:18.112 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:18.111+0000 7f2021acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:18.113 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:18.111+0000 7f35e205a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:18.222 INFO:teuthology.orchestra.run.trial043.stdout:47244640261 2026-02-20T11:15:18.229 INFO:teuthology.orchestra.run.trial043.stdout:47244640261 2026-02-20T11:15:18.229 INFO:teuthology.orchestra.run.trial043.stdout:47244640260 2026-02-20T11:15:18.233 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.0 2026-02-20T11:15:18.233 DEBUG:teuthology.parallel:result is None 2026-02-20T11:15:18.234 INFO:teuthology.orchestra.run.trial043.stdout:42949672966 2026-02-20T11:15:18.241 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.2 2026-02-20T11:15:18.242 DEBUG:teuthology.parallel:result is None 2026-02-20T11:15:18.242 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640260 for osd.3 2026-02-20T11:15:18.245 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672966 for osd.1 2026-02-20T11:15:18.245 DEBUG:teuthology.parallel:result is None 2026-02-20T11:15:19.243 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-20T11:15:19.322 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:19.318+0000 7fd9e5574640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:19.324 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:19.324+0000 7fd9e5574640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:19.436 INFO:teuthology.orchestra.run.trial043.stdout:47244640260 2026-02-20T11:15:19.445 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640260 for osd.3 2026-02-20T11:15:20.446 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-20T11:15:20.525 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:20.521+0000 7f249f4cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:20.527 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:20.527+0000 7f249f4cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:20.638 INFO:teuthology.orchestra.run.trial043.stdout:47244640261 2026-02-20T11:15:20.648 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.3 2026-02-20T11:15:20.649 DEBUG:teuthology.parallel:result is None 2026-02-20T11:15:20.649 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-20T11:15:20.649 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-20T11:15:20.724 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:20.721+0000 7f1b4f10b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:20.726 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:20.726+0000 7f1b4f10b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:20.835 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:15:20.836 INFO:teuthology.orchestra.run.trial043.stderr:dumped all 2026-02-20T11:15:20.845 INFO:teuthology.orchestra.run.trial043.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2026-02-20T11:15:19.938135+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209715136,"kb_used":83890120,"kb_used_data":83890120,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":125825016,"statfs":{"total":214748299264,"available":128844816384,"internally_reserved":0,"allocated":85903482880,"data_stored":85903482880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.960781"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.983722+0000","last_change":"2026-02-20T11:15:16.986570+0000","last_active":"2026-02-20T11:15:16.983722+0000","last_peered":"2026-02-20T11:15:16.983722+0000","last_clean":"2026-02-20T11:15:16.983722+0000","last_became_active":"2026-02-20T11:15:15.057391+0000","last_became_peered":"2026-02-20T11:15:15.057391+0000","last_unstale":"2026-02-20T11:15:16.983722+0000","last_undegraded":"2026-02-20T11:15:16.983722+0000","last_fullsized":"2026-02-20T11:15:16.983722+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.982620+0000","last_change":"2026-02-20T11:15:16.986873+0000","last_active":"2026-02-20T11:15:16.982620+0000","last_peered":"2026-02-20T11:15:16.982620+0000","last_clean":"2026-02-20T11:15:16.982620+0000","last_became_active":"2026-02-20T11:15:15.018131+0000","last_became_peered":"2026-02-20T11:15:15.018131+0000","last_unstale":"2026-02-20T11:15:16.982620+0000","last_undegraded":"2026-02-20T11:15:16.982620+0000","last_fullsized":"2026-02-20T11:15:16.982620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.983812+0000","last_change":"2026-02-20T11:15:16.985671+0000","last_active":"2026-02-20T11:15:16.983812+0000","last_peered":"2026-02-20T11:15:16.983812+0000","last_clean":"2026-02-20T11:15:16.983812+0000","last_became_active":"2026-02-20T11:15:15.003549+0000","last_became_peered":"2026-02-20T11:15:15.003549+0000","last_unstale":"2026-02-20T11:15:16.983812+0000","last_undegraded":"2026-02-20T11:15:16.983812+0000","last_fullsized":"2026-02-20T11:15:16.983812+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.982312+0000","last_change":"2026-02-20T11:15:16.984208+0000","last_active":"2026-02-20T11:15:16.982312+0000","last_peered":"2026-02-20T11:15:16.982312+0000","last_clean":"2026-02-20T11:15:16.982312+0000","last_became_active":"2026-02-20T11:15:15.027750+0000","last_became_peered":"2026-02-20T11:15:15.027750+0000","last_unstale":"2026-02-20T11:15:16.982312+0000","last_undegraded":"2026-02-20T11:15:16.982312+0000","last_fullsized":"2026-02-20T11:15:16.982312+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'52","reported_seq":82,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.980870+0000","last_change":"2026-02-20T11:15:15.035494+0000","last_active":"2026-02-20T11:15:16.980870+0000","last_peered":"2026-02-20T11:15:16.980870+0000","last_clean":"2026-02-20T11:15:16.980870+0000","last_became_active":"2026-02-20T11:15:15.012645+0000","last_became_peered":"2026-02-20T11:15:15.012645+0000","last_unstale":"2026-02-20T11:15:16.980870+0000","last_undegraded":"2026-02-20T11:15:16.980870+0000","last_fullsized":"2026-02-20T11:15:16.980870+0000","mapping_epoch":12,"log_start":"13'48","ondisk_log_start":"13'48","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":48,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.980370+0000","last_change":"2026-02-20T11:15:16.983291+0000","last_active":"2026-02-20T11:15:16.980370+0000","last_peered":"2026-02-20T11:15:16.980370+0000","last_clean":"2026-02-20T11:15:16.980370+0000","last_became_active":"2026-02-20T11:15:15.009983+0000","last_became_peered":"2026-02-20T11:15:15.009983+0000","last_unstale":"2026-02-20T11:15:16.980370+0000","last_undegraded":"2026-02-20T11:15:16.980370+0000","last_fullsized":"2026-02-20T11:15:16.980370+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"14'1","reported_seq":18,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.983646+0000","last_change":"2026-02-20T11:15:16.983646+0000","last_active":"2026-02-20T11:15:16.983646+0000","last_peered":"2026-02-20T11:15:16.983646+0000","last_clean":"2026-02-20T11:15:16.983646+0000","last_became_active":"2026-02-20T11:15:15.012729+0000","last_became_peered":"2026-02-20T11:15:15.012729+0000","last_unstale":"2026-02-20T11:15:16.983646+0000","last_undegraded":"2026-02-20T11:15:16.983646+0000","last_fullsized":"2026-02-20T11:15:16.983646+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.983008+0000","last_change":"2026-02-20T11:15:16.985758+0000","last_active":"2026-02-20T11:15:16.983008+0000","last_peered":"2026-02-20T11:15:16.983008+0000","last_clean":"2026-02-20T11:15:16.983008+0000","last_became_active":"2026-02-20T11:15:15.019055+0000","last_became_peered":"2026-02-20T11:15:15.019055+0000","last_unstale":"2026-02-20T11:15:16.983008+0000","last_undegraded":"2026-02-20T11:15:16.983008+0000","last_fullsized":"2026-02-20T11:15:16.983008+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.981411+0000","last_change":"2026-02-20T11:15:16.984535+0000","last_active":"2026-02-20T11:15:16.981411+0000","last_peered":"2026-02-20T11:15:16.981411+0000","last_clean":"2026-02-20T11:15:16.981411+0000","last_became_active":"2026-02-20T11:15:15.006596+0000","last_became_peered":"2026-02-20T11:15:15.006596+0000","last_unstale":"2026-02-20T11:15:16.981411+0000","last_undegraded":"2026-02-20T11:15:16.981411+0000","last_fullsized":"2026-02-20T11:15:16.981411+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32210939904,"internally_reserved":0,"allocated":21476134912,"data_stored":21476134912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96633778176,"internally_reserved":0,"allocated":64427446272,"data_stored":64427446272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":24,"acting":24,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972788,"kb_used_data":20972788,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31455996,"statfs":{"total":53687074816,"available":32210939904,"internally_reserved":0,"allocated":21476134912,"data_stored":21476134912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20971984,"kb_used_data":20971984,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456800,"statfs":{"total":53687074816,"available":32211763200,"internally_reserved":0,"allocated":21475311616,"data_stored":21475311616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972656,"kb_used_data":20972656,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456128,"statfs":{"total":53687074816,"available":32211075072,"internally_reserved":0,"allocated":21475999744,"data_stored":21475999744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972692,"kb_used_data":20972692,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456092,"statfs":{"total":53687074816,"available":32211038208,"internally_reserved":0,"allocated":21476036608,"data_stored":21476036608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32211075072,"internally_reserved":0,"allocated":21475999744,"data_stored":21475999744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53687074816,"available":32211763200,"internally_reserved":0,"allocated":21475311616,"data_stored":21475311616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":53687074816,"available":32210939904,"internally_reserved":0,"allocated":21476134912,"data_stored":21476134912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53687074816,"available":32210939904,"internally_reserved":0,"allocated":21476134912,"data_stored":21476134912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-20T11:15:20.846 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-20T11:15:20.924 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:20.920+0000 7f3b8a85a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:20.926 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:20.925+0000 7f3b8a85a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:21.036 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:15:21.037 INFO:teuthology.orchestra.run.trial043.stderr:dumped all 2026-02-20T11:15:21.046 INFO:teuthology.orchestra.run.trial043.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2026-02-20T11:15:19.938135+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209715136,"kb_used":83890120,"kb_used_data":83890120,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":125825016,"statfs":{"total":214748299264,"available":128844816384,"internally_reserved":0,"allocated":85903482880,"data_stored":85903482880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.960781"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.983722+0000","last_change":"2026-02-20T11:15:16.986570+0000","last_active":"2026-02-20T11:15:16.983722+0000","last_peered":"2026-02-20T11:15:16.983722+0000","last_clean":"2026-02-20T11:15:16.983722+0000","last_became_active":"2026-02-20T11:15:15.057391+0000","last_became_peered":"2026-02-20T11:15:15.057391+0000","last_unstale":"2026-02-20T11:15:16.983722+0000","last_undegraded":"2026-02-20T11:15:16.983722+0000","last_fullsized":"2026-02-20T11:15:16.983722+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.982620+0000","last_change":"2026-02-20T11:15:16.986873+0000","last_active":"2026-02-20T11:15:16.982620+0000","last_peered":"2026-02-20T11:15:16.982620+0000","last_clean":"2026-02-20T11:15:16.982620+0000","last_became_active":"2026-02-20T11:15:15.018131+0000","last_became_peered":"2026-02-20T11:15:15.018131+0000","last_unstale":"2026-02-20T11:15:16.982620+0000","last_undegraded":"2026-02-20T11:15:16.982620+0000","last_fullsized":"2026-02-20T11:15:16.982620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.983812+0000","last_change":"2026-02-20T11:15:16.985671+0000","last_active":"2026-02-20T11:15:16.983812+0000","last_peered":"2026-02-20T11:15:16.983812+0000","last_clean":"2026-02-20T11:15:16.983812+0000","last_became_active":"2026-02-20T11:15:15.003549+0000","last_became_peered":"2026-02-20T11:15:15.003549+0000","last_unstale":"2026-02-20T11:15:16.983812+0000","last_undegraded":"2026-02-20T11:15:16.983812+0000","last_fullsized":"2026-02-20T11:15:16.983812+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.982312+0000","last_change":"2026-02-20T11:15:16.984208+0000","last_active":"2026-02-20T11:15:16.982312+0000","last_peered":"2026-02-20T11:15:16.982312+0000","last_clean":"2026-02-20T11:15:16.982312+0000","last_became_active":"2026-02-20T11:15:15.027750+0000","last_became_peered":"2026-02-20T11:15:15.027750+0000","last_unstale":"2026-02-20T11:15:16.982312+0000","last_undegraded":"2026-02-20T11:15:16.982312+0000","last_fullsized":"2026-02-20T11:15:16.982312+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'52","reported_seq":82,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.980870+0000","last_change":"2026-02-20T11:15:15.035494+0000","last_active":"2026-02-20T11:15:16.980870+0000","last_peered":"2026-02-20T11:15:16.980870+0000","last_clean":"2026-02-20T11:15:16.980870+0000","last_became_active":"2026-02-20T11:15:15.012645+0000","last_became_peered":"2026-02-20T11:15:15.012645+0000","last_unstale":"2026-02-20T11:15:16.980870+0000","last_undegraded":"2026-02-20T11:15:16.980870+0000","last_fullsized":"2026-02-20T11:15:16.980870+0000","mapping_epoch":12,"log_start":"13'48","ondisk_log_start":"13'48","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":48,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.980370+0000","last_change":"2026-02-20T11:15:16.983291+0000","last_active":"2026-02-20T11:15:16.980370+0000","last_peered":"2026-02-20T11:15:16.980370+0000","last_clean":"2026-02-20T11:15:16.980370+0000","last_became_active":"2026-02-20T11:15:15.009983+0000","last_became_peered":"2026-02-20T11:15:15.009983+0000","last_unstale":"2026-02-20T11:15:16.980370+0000","last_undegraded":"2026-02-20T11:15:16.980370+0000","last_fullsized":"2026-02-20T11:15:16.980370+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"14'1","reported_seq":18,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.983646+0000","last_change":"2026-02-20T11:15:16.983646+0000","last_active":"2026-02-20T11:15:16.983646+0000","last_peered":"2026-02-20T11:15:16.983646+0000","last_clean":"2026-02-20T11:15:16.983646+0000","last_became_active":"2026-02-20T11:15:15.012729+0000","last_became_peered":"2026-02-20T11:15:15.012729+0000","last_unstale":"2026-02-20T11:15:16.983646+0000","last_undegraded":"2026-02-20T11:15:16.983646+0000","last_fullsized":"2026-02-20T11:15:16.983646+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.983008+0000","last_change":"2026-02-20T11:15:16.985758+0000","last_active":"2026-02-20T11:15:16.983008+0000","last_peered":"2026-02-20T11:15:16.983008+0000","last_clean":"2026-02-20T11:15:16.983008+0000","last_became_active":"2026-02-20T11:15:15.019055+0000","last_became_peered":"2026-02-20T11:15:15.019055+0000","last_unstale":"2026-02-20T11:15:16.983008+0000","last_undegraded":"2026-02-20T11:15:16.983008+0000","last_fullsized":"2026-02-20T11:15:16.983008+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.981411+0000","last_change":"2026-02-20T11:15:16.984535+0000","last_active":"2026-02-20T11:15:16.981411+0000","last_peered":"2026-02-20T11:15:16.981411+0000","last_clean":"2026-02-20T11:15:16.981411+0000","last_became_active":"2026-02-20T11:15:15.006596+0000","last_became_peered":"2026-02-20T11:15:15.006596+0000","last_unstale":"2026-02-20T11:15:16.981411+0000","last_undegraded":"2026-02-20T11:15:16.981411+0000","last_fullsized":"2026-02-20T11:15:16.981411+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32210939904,"internally_reserved":0,"allocated":21476134912,"data_stored":21476134912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96633778176,"internally_reserved":0,"allocated":64427446272,"data_stored":64427446272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":24,"acting":24,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972788,"kb_used_data":20972788,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31455996,"statfs":{"total":53687074816,"available":32210939904,"internally_reserved":0,"allocated":21476134912,"data_stored":21476134912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20971984,"kb_used_data":20971984,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456800,"statfs":{"total":53687074816,"available":32211763200,"internally_reserved":0,"allocated":21475311616,"data_stored":21475311616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972656,"kb_used_data":20972656,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456128,"statfs":{"total":53687074816,"available":32211075072,"internally_reserved":0,"allocated":21475999744,"data_stored":21475999744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972692,"kb_used_data":20972692,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456092,"statfs":{"total":53687074816,"available":32211038208,"internally_reserved":0,"allocated":21476036608,"data_stored":21476036608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32211075072,"internally_reserved":0,"allocated":21475999744,"data_stored":21475999744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53687074816,"available":32211763200,"internally_reserved":0,"allocated":21475311616,"data_stored":21475311616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":53687074816,"available":32210939904,"internally_reserved":0,"allocated":21476134912,"data_stored":21476134912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53687074816,"available":32210939904,"internally_reserved":0,"allocated":21476134912,"data_stored":21476134912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-20T11:15:21.048 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-20T11:15:21.048 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-20T11:15:21.048 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-20T11:15:21.048 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-20T11:15:21.124 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:21.122+0000 7f65929f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:21.126 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:21.126+0000 7f65929f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:21.266 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:15:21.266 INFO:teuthology.orchestra.run.trial043.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-20T11:15:21.277 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-20T11:15:21.277 INFO:teuthology.run_tasks:Running task thrashosds... 2026-02-20T11:15:21.293 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_trim_stale_osdmaps': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2026-02-20T11:15:21.293 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-perezjos-crimson-only-19-02-2026-PR64439'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd target transaction size': 5, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Degraded data redundancy', 'is down', 'osds down', 'pg .*? is .*?degraded.*?, acting', 'pg .*? is stuck', 'pg degraded', 'PG_BACKFILL_FULL', 'Low space hindering backfill .*? backfill_toofull', 'OSD_ROOT_DOWN', 'pgs degraded', 'pgs undersized', 'is active.*backfill_toofull.*'], 'sha1': '0c3148ca19c827ca2a7a9b92c2554d99507fd1f4'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'conf': {'osd': {'osd deep scrub update digest min age': 0}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': '0c3148ca19c827ca2a7a9b92c2554d99507fd1f4'}}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '445d5c9ba18a4040e70e25303230c36c77773acb'}} 2026-02-20T11:15:21.293 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_trim_stale_osdmaps': 0, 'dump_ops_enable': False, 'min_in': 4, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2026-02-20T11:15:21.294 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2026-02-20T11:15:21.372 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:21.367+0000 7fe4184c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:21.374 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:21.373+0000 7fe4184c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:21.498 INFO:tasks.thrashosds:Beginning thrashosds... 2026-02-20T11:15:21.499 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-20T11:15:21.499 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-20T11:15:21.574 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:21.571+0000 7ff31b70b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:21.576 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:21.576+0000 7ff31b70b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:21.687 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:15:21.687 INFO:teuthology.orchestra.run.trial043.stderr:dumped all 2026-02-20T11:15:21.696 INFO:teuthology.orchestra.run.trial043.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2026-02-20T11:15:19.938135+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209715136,"kb_used":83890120,"kb_used_data":83890120,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":125825016,"statfs":{"total":214748299264,"available":128844816384,"internally_reserved":0,"allocated":85903482880,"data_stored":85903482880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.960781"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.983722+0000","last_change":"2026-02-20T11:15:16.986570+0000","last_active":"2026-02-20T11:15:16.983722+0000","last_peered":"2026-02-20T11:15:16.983722+0000","last_clean":"2026-02-20T11:15:16.983722+0000","last_became_active":"2026-02-20T11:15:15.057391+0000","last_became_peered":"2026-02-20T11:15:15.057391+0000","last_unstale":"2026-02-20T11:15:16.983722+0000","last_undegraded":"2026-02-20T11:15:16.983722+0000","last_fullsized":"2026-02-20T11:15:16.983722+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.982620+0000","last_change":"2026-02-20T11:15:16.986873+0000","last_active":"2026-02-20T11:15:16.982620+0000","last_peered":"2026-02-20T11:15:16.982620+0000","last_clean":"2026-02-20T11:15:16.982620+0000","last_became_active":"2026-02-20T11:15:15.018131+0000","last_became_peered":"2026-02-20T11:15:15.018131+0000","last_unstale":"2026-02-20T11:15:16.982620+0000","last_undegraded":"2026-02-20T11:15:16.982620+0000","last_fullsized":"2026-02-20T11:15:16.982620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.983812+0000","last_change":"2026-02-20T11:15:16.985671+0000","last_active":"2026-02-20T11:15:16.983812+0000","last_peered":"2026-02-20T11:15:16.983812+0000","last_clean":"2026-02-20T11:15:16.983812+0000","last_became_active":"2026-02-20T11:15:15.003549+0000","last_became_peered":"2026-02-20T11:15:15.003549+0000","last_unstale":"2026-02-20T11:15:16.983812+0000","last_undegraded":"2026-02-20T11:15:16.983812+0000","last_fullsized":"2026-02-20T11:15:16.983812+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.982312+0000","last_change":"2026-02-20T11:15:16.984208+0000","last_active":"2026-02-20T11:15:16.982312+0000","last_peered":"2026-02-20T11:15:16.982312+0000","last_clean":"2026-02-20T11:15:16.982312+0000","last_became_active":"2026-02-20T11:15:15.027750+0000","last_became_peered":"2026-02-20T11:15:15.027750+0000","last_unstale":"2026-02-20T11:15:16.982312+0000","last_undegraded":"2026-02-20T11:15:16.982312+0000","last_fullsized":"2026-02-20T11:15:16.982312+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'52","reported_seq":82,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.980870+0000","last_change":"2026-02-20T11:15:15.035494+0000","last_active":"2026-02-20T11:15:16.980870+0000","last_peered":"2026-02-20T11:15:16.980870+0000","last_clean":"2026-02-20T11:15:16.980870+0000","last_became_active":"2026-02-20T11:15:15.012645+0000","last_became_peered":"2026-02-20T11:15:15.012645+0000","last_unstale":"2026-02-20T11:15:16.980870+0000","last_undegraded":"2026-02-20T11:15:16.980870+0000","last_fullsized":"2026-02-20T11:15:16.980870+0000","mapping_epoch":12,"log_start":"13'48","ondisk_log_start":"13'48","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":48,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.980370+0000","last_change":"2026-02-20T11:15:16.983291+0000","last_active":"2026-02-20T11:15:16.980370+0000","last_peered":"2026-02-20T11:15:16.980370+0000","last_clean":"2026-02-20T11:15:16.980370+0000","last_became_active":"2026-02-20T11:15:15.009983+0000","last_became_peered":"2026-02-20T11:15:15.009983+0000","last_unstale":"2026-02-20T11:15:16.980370+0000","last_undegraded":"2026-02-20T11:15:16.980370+0000","last_fullsized":"2026-02-20T11:15:16.980370+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"14'1","reported_seq":18,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.983646+0000","last_change":"2026-02-20T11:15:16.983646+0000","last_active":"2026-02-20T11:15:16.983646+0000","last_peered":"2026-02-20T11:15:16.983646+0000","last_clean":"2026-02-20T11:15:16.983646+0000","last_became_active":"2026-02-20T11:15:15.012729+0000","last_became_peered":"2026-02-20T11:15:15.012729+0000","last_unstale":"2026-02-20T11:15:16.983646+0000","last_undegraded":"2026-02-20T11:15:16.983646+0000","last_fullsized":"2026-02-20T11:15:16.983646+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.983008+0000","last_change":"2026-02-20T11:15:16.985758+0000","last_active":"2026-02-20T11:15:16.983008+0000","last_peered":"2026-02-20T11:15:16.983008+0000","last_clean":"2026-02-20T11:15:16.983008+0000","last_became_active":"2026-02-20T11:15:15.019055+0000","last_became_peered":"2026-02-20T11:15:15.019055+0000","last_unstale":"2026-02-20T11:15:16.983008+0000","last_undegraded":"2026-02-20T11:15:16.983008+0000","last_fullsized":"2026-02-20T11:15:16.983008+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.981411+0000","last_change":"2026-02-20T11:15:16.984535+0000","last_active":"2026-02-20T11:15:16.981411+0000","last_peered":"2026-02-20T11:15:16.981411+0000","last_clean":"2026-02-20T11:15:16.981411+0000","last_became_active":"2026-02-20T11:15:15.006596+0000","last_became_peered":"2026-02-20T11:15:15.006596+0000","last_unstale":"2026-02-20T11:15:16.981411+0000","last_undegraded":"2026-02-20T11:15:16.981411+0000","last_fullsized":"2026-02-20T11:15:16.981411+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32210939904,"internally_reserved":0,"allocated":21476134912,"data_stored":21476134912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96633778176,"internally_reserved":0,"allocated":64427446272,"data_stored":64427446272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":24,"acting":24,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972788,"kb_used_data":20972788,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31455996,"statfs":{"total":53687074816,"available":32210939904,"internally_reserved":0,"allocated":21476134912,"data_stored":21476134912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20971984,"kb_used_data":20971984,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456800,"statfs":{"total":53687074816,"available":32211763200,"internally_reserved":0,"allocated":21475311616,"data_stored":21475311616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972656,"kb_used_data":20972656,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456128,"statfs":{"total":53687074816,"available":32211075072,"internally_reserved":0,"allocated":21475999744,"data_stored":21475999744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972692,"kb_used_data":20972692,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456092,"statfs":{"total":53687074816,"available":32211038208,"internally_reserved":0,"allocated":21476036608,"data_stored":21476036608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32211075072,"internally_reserved":0,"allocated":21475999744,"data_stored":21475999744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53687074816,"available":32211763200,"internally_reserved":0,"allocated":21475311616,"data_stored":21475311616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":53687074816,"available":32210939904,"internally_reserved":0,"allocated":21476134912,"data_stored":21476134912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53687074816,"available":32210939904,"internally_reserved":0,"allocated":21476134912,"data_stored":21476134912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-20T11:15:21.698 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-20T11:15:21.772 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:21.769+0000 7fb28cf9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:21.775 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:21.774+0000 7fb28cf9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:21.883 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:15:21.883 INFO:teuthology.orchestra.run.trial043.stderr:dumped all 2026-02-20T11:15:21.892 INFO:teuthology.orchestra.run.trial043.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2026-02-20T11:15:19.938135+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209715136,"kb_used":83890120,"kb_used_data":83890120,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":125825016,"statfs":{"total":214748299264,"available":128844816384,"internally_reserved":0,"allocated":85903482880,"data_stored":85903482880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.960781"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.983722+0000","last_change":"2026-02-20T11:15:16.986570+0000","last_active":"2026-02-20T11:15:16.983722+0000","last_peered":"2026-02-20T11:15:16.983722+0000","last_clean":"2026-02-20T11:15:16.983722+0000","last_became_active":"2026-02-20T11:15:15.057391+0000","last_became_peered":"2026-02-20T11:15:15.057391+0000","last_unstale":"2026-02-20T11:15:16.983722+0000","last_undegraded":"2026-02-20T11:15:16.983722+0000","last_fullsized":"2026-02-20T11:15:16.983722+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.982620+0000","last_change":"2026-02-20T11:15:16.986873+0000","last_active":"2026-02-20T11:15:16.982620+0000","last_peered":"2026-02-20T11:15:16.982620+0000","last_clean":"2026-02-20T11:15:16.982620+0000","last_became_active":"2026-02-20T11:15:15.018131+0000","last_became_peered":"2026-02-20T11:15:15.018131+0000","last_unstale":"2026-02-20T11:15:16.982620+0000","last_undegraded":"2026-02-20T11:15:16.982620+0000","last_fullsized":"2026-02-20T11:15:16.982620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.983812+0000","last_change":"2026-02-20T11:15:16.985671+0000","last_active":"2026-02-20T11:15:16.983812+0000","last_peered":"2026-02-20T11:15:16.983812+0000","last_clean":"2026-02-20T11:15:16.983812+0000","last_became_active":"2026-02-20T11:15:15.003549+0000","last_became_peered":"2026-02-20T11:15:15.003549+0000","last_unstale":"2026-02-20T11:15:16.983812+0000","last_undegraded":"2026-02-20T11:15:16.983812+0000","last_fullsized":"2026-02-20T11:15:16.983812+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.982312+0000","last_change":"2026-02-20T11:15:16.984208+0000","last_active":"2026-02-20T11:15:16.982312+0000","last_peered":"2026-02-20T11:15:16.982312+0000","last_clean":"2026-02-20T11:15:16.982312+0000","last_became_active":"2026-02-20T11:15:15.027750+0000","last_became_peered":"2026-02-20T11:15:15.027750+0000","last_unstale":"2026-02-20T11:15:16.982312+0000","last_undegraded":"2026-02-20T11:15:16.982312+0000","last_fullsized":"2026-02-20T11:15:16.982312+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'52","reported_seq":82,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.980870+0000","last_change":"2026-02-20T11:15:15.035494+0000","last_active":"2026-02-20T11:15:16.980870+0000","last_peered":"2026-02-20T11:15:16.980870+0000","last_clean":"2026-02-20T11:15:16.980870+0000","last_became_active":"2026-02-20T11:15:15.012645+0000","last_became_peered":"2026-02-20T11:15:15.012645+0000","last_unstale":"2026-02-20T11:15:16.980870+0000","last_undegraded":"2026-02-20T11:15:16.980870+0000","last_fullsized":"2026-02-20T11:15:16.980870+0000","mapping_epoch":12,"log_start":"13'48","ondisk_log_start":"13'48","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":48,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.980370+0000","last_change":"2026-02-20T11:15:16.983291+0000","last_active":"2026-02-20T11:15:16.980370+0000","last_peered":"2026-02-20T11:15:16.980370+0000","last_clean":"2026-02-20T11:15:16.980370+0000","last_became_active":"2026-02-20T11:15:15.009983+0000","last_became_peered":"2026-02-20T11:15:15.009983+0000","last_unstale":"2026-02-20T11:15:16.980370+0000","last_undegraded":"2026-02-20T11:15:16.980370+0000","last_fullsized":"2026-02-20T11:15:16.980370+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"14'1","reported_seq":18,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.983646+0000","last_change":"2026-02-20T11:15:16.983646+0000","last_active":"2026-02-20T11:15:16.983646+0000","last_peered":"2026-02-20T11:15:16.983646+0000","last_clean":"2026-02-20T11:15:16.983646+0000","last_became_active":"2026-02-20T11:15:15.012729+0000","last_became_peered":"2026-02-20T11:15:15.012729+0000","last_unstale":"2026-02-20T11:15:16.983646+0000","last_undegraded":"2026-02-20T11:15:16.983646+0000","last_fullsized":"2026-02-20T11:15:16.983646+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.983008+0000","last_change":"2026-02-20T11:15:16.985758+0000","last_active":"2026-02-20T11:15:16.983008+0000","last_peered":"2026-02-20T11:15:16.983008+0000","last_clean":"2026-02-20T11:15:16.983008+0000","last_became_active":"2026-02-20T11:15:15.019055+0000","last_became_peered":"2026-02-20T11:15:15.019055+0000","last_unstale":"2026-02-20T11:15:16.983008+0000","last_undegraded":"2026-02-20T11:15:16.983008+0000","last_fullsized":"2026-02-20T11:15:16.983008+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.981411+0000","last_change":"2026-02-20T11:15:16.984535+0000","last_active":"2026-02-20T11:15:16.981411+0000","last_peered":"2026-02-20T11:15:16.981411+0000","last_clean":"2026-02-20T11:15:16.981411+0000","last_became_active":"2026-02-20T11:15:15.006596+0000","last_became_peered":"2026-02-20T11:15:15.006596+0000","last_unstale":"2026-02-20T11:15:16.981411+0000","last_undegraded":"2026-02-20T11:15:16.981411+0000","last_fullsized":"2026-02-20T11:15:16.981411+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32210939904,"internally_reserved":0,"allocated":21476134912,"data_stored":21476134912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96633778176,"internally_reserved":0,"allocated":64427446272,"data_stored":64427446272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":24,"acting":24,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972788,"kb_used_data":20972788,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31455996,"statfs":{"total":53687074816,"available":32210939904,"internally_reserved":0,"allocated":21476134912,"data_stored":21476134912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20971984,"kb_used_data":20971984,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456800,"statfs":{"total":53687074816,"available":32211763200,"internally_reserved":0,"allocated":21475311616,"data_stored":21475311616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972656,"kb_used_data":20972656,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456128,"statfs":{"total":53687074816,"available":32211075072,"internally_reserved":0,"allocated":21475999744,"data_stored":21475999744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972692,"kb_used_data":20972692,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456092,"statfs":{"total":53687074816,"available":32211038208,"internally_reserved":0,"allocated":21476036608,"data_stored":21476036608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32211075072,"internally_reserved":0,"allocated":21475999744,"data_stored":21475999744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53687074816,"available":32211763200,"internally_reserved":0,"allocated":21475311616,"data_stored":21475311616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":53687074816,"available":32210939904,"internally_reserved":0,"allocated":21476134912,"data_stored":21476134912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53687074816,"available":32210939904,"internally_reserved":0,"allocated":21476134912,"data_stored":21476134912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-20T11:15:21.894 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-20T11:15:21.894 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2026-02-20T11:15:21.969 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:21.966+0000 7f5055b9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:21.971 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:21.970+0000 7f5055b9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:22.083 INFO:teuthology.orchestra.run.trial043.stdout:epoch 15 2026-02-20T11:15:22.083 INFO:teuthology.orchestra.run.trial043.stdout:fsid dfee6dd4-7cac-4c06-aaed-6868a4b5b9a0 2026-02-20T11:15:22.084 INFO:teuthology.orchestra.run.trial043.stdout:created 2026-02-20T11:15:03.729224+0000 2026-02-20T11:15:22.084 INFO:teuthology.orchestra.run.trial043.stdout:modified 2026-02-20T11:15:16.969194+0000 2026-02-20T11:15:22.084 INFO:teuthology.orchestra.run.trial043.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2026-02-20T11:15:22.084 INFO:teuthology.orchestra.run.trial043.stdout:crush_version 5 2026-02-20T11:15:22.084 INFO:teuthology.orchestra.run.trial043.stdout:full_ratio 0.95 2026-02-20T11:15:22.084 INFO:teuthology.orchestra.run.trial043.stdout:backfillfull_ratio 0.9 2026-02-20T11:15:22.084 INFO:teuthology.orchestra.run.trial043.stdout:nearfull_ratio 0.85 2026-02-20T11:15:22.084 INFO:teuthology.orchestra.run.trial043.stdout:require_min_compat_client luminous 2026-02-20T11:15:22.084 INFO:teuthology.orchestra.run.trial043.stdout:min_compat_client jewel 2026-02-20T11:15:22.084 INFO:teuthology.orchestra.run.trial043.stdout:require_osd_release tentacle 2026-02-20T11:15:22.084 INFO:teuthology.orchestra.run.trial043.stdout:stretch_mode_enabled false 2026-02-20T11:15:22.085 INFO:teuthology.orchestra.run.trial043.stdout:allow_crimson=true 2026-02-20T11:15:22.085 INFO:teuthology.orchestra.run.trial043.stdout:pool 1 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 15 flags hashpspool,selfmanaged_snaps,crimson stripe_width 0 application rbd read_balance_score 2.00 2026-02-20T11:15:22.085 INFO:teuthology.orchestra.run.trial043.stdout: removed_snaps_queue [2~1] 2026-02-20T11:15:22.085 INFO:teuthology.orchestra.run.trial043.stdout:pool 2 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 14 flags hashpspool,crimson stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 4.00 2026-02-20T11:15:22.085 INFO:teuthology.orchestra.run.trial043.stdout:max_osd 4 2026-02-20T11:15:22.085 INFO:teuthology.orchestra.run.trial043.stdout:osd.0 up in weight 1 up_from 11 up_thru 0 down_at 0 last_clean_interval [0,0) v2:10.20.193.43:6803/1408480372 v2:10.20.193.43:6804/1408480372 exists,up a3a3446f-cb7d-471f-bcf9-06c7dd1baea1 2026-02-20T11:15:22.085 INFO:teuthology.orchestra.run.trial043.stdout:osd.1 up in weight 1 up_from 10 up_thru 12 down_at 0 last_clean_interval [0,0) v2:10.20.193.43:6800/18189571 v2:10.20.193.43:6801/18189571 exists,up 8bb9e0b4-ad0f-4e8d-9815-71c91f5644c6 2026-02-20T11:15:22.085 INFO:teuthology.orchestra.run.trial043.stdout:osd.2 up in weight 1 up_from 11 up_thru 12 down_at 0 last_clean_interval [0,0) v2:10.20.193.109:6802/2255254232 v2:10.20.193.109:6805/2255254232 exists,up 0c9b6f94-6c72-43b3-9987-12e16447a1df 2026-02-20T11:15:22.085 INFO:teuthology.orchestra.run.trial043.stdout:osd.3 up in weight 1 up_from 11 up_thru 12 down_at 0 last_clean_interval [0,0) v2:10.20.193.109:6804/829537078 v2:10.20.193.109:6803/829537078 exists,up d9dfa6e5-fa90-49f6-a16c-74b687ecafef 2026-02-20T11:15:22.093 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 11 up_thru 0 down_at 0 last_clean_interval [0,0) v2:10.20.193.43:6803/1408480372 v2:10.20.193.43:6804/1408480372 exists,up a3a3446f-cb7d-471f-bcf9-06c7dd1baea1', 'osd.1 up in weight 1 up_from 10 up_thru 12 down_at 0 last_clean_interval [0,0) v2:10.20.193.43:6800/18189571 v2:10.20.193.43:6801/18189571 exists,up 8bb9e0b4-ad0f-4e8d-9815-71c91f5644c6', 'osd.2 up in weight 1 up_from 11 up_thru 12 down_at 0 last_clean_interval [0,0) v2:10.20.193.109:6802/2255254232 v2:10.20.193.109:6805/2255254232 exists,up 0c9b6f94-6c72-43b3-9987-12e16447a1df', 'osd.3 up in weight 1 up_from 11 up_thru 12 down_at 0 last_clean_interval [0,0) v2:10.20.193.109:6804/829537078 v2:10.20.193.109:6803/829537078 exists,up d9dfa6e5-fa90-49f6-a16c-74b687ecafef'] 2026-02-20T11:15:22.093 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2026-02-20T11:15:22.168 INFO:teuthology.orchestra.run.trial043.stdout:{ 2026-02-20T11:15:22.168 INFO:teuthology.orchestra.run.trial043.stdout: "name": "mon.a", 2026-02-20T11:15:22.168 INFO:teuthology.orchestra.run.trial043.stdout: "cluster": "ceph", 2026-02-20T11:15:22.169 INFO:teuthology.orchestra.run.trial043.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2026-02-20T11:15:22.169 INFO:teuthology.orchestra.run.trial043.stdout: "admin_socket_mode": "", 2026-02-20T11:15:22.169 INFO:teuthology.orchestra.run.trial043.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2026-02-20T11:15:22.169 INFO:teuthology.orchestra.run.trial043.stdout: "auth_client_required": "cephx, none", 2026-02-20T11:15:22.169 INFO:teuthology.orchestra.run.trial043.stdout: "auth_cluster_required": "cephx", 2026-02-20T11:15:22.169 INFO:teuthology.orchestra.run.trial043.stdout: "auth_debug": "true", 2026-02-20T11:15:22.169 INFO:teuthology.orchestra.run.trial043.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2026-02-20T11:15:22.169 INFO:teuthology.orchestra.run.trial043.stdout: "auth_mon_ticket_ttl": "660.000000", 2026-02-20T11:15:22.169 INFO:teuthology.orchestra.run.trial043.stdout: "auth_service_required": "cephx", 2026-02-20T11:15:22.169 INFO:teuthology.orchestra.run.trial043.stdout: "auth_service_ticket_ttl": "240.000000", 2026-02-20T11:15:22.169 INFO:teuthology.orchestra.run.trial043.stdout: "auth_supported": "cephx", 2026-02-20T11:15:22.170 INFO:teuthology.orchestra.run.trial043.stdout: "bdev_aio": "true", 2026-02-20T11:15:22.170 INFO:teuthology.orchestra.run.trial043.stdout: "bdev_aio_max_queue_depth": "1024", 2026-02-20T11:15:22.170 INFO:teuthology.orchestra.run.trial043.stdout: "bdev_aio_poll_ms": "250", 2026-02-20T11:15:22.170 INFO:teuthology.orchestra.run.trial043.stdout: "bdev_aio_reap_max": "16", 2026-02-20T11:15:22.170 INFO:teuthology.orchestra.run.trial043.stdout: "bdev_aio_submit_retry_initial_delay_us": "125", 2026-02-20T11:15:22.170 INFO:teuthology.orchestra.run.trial043.stdout: "bdev_aio_submit_retry_max": "16", 2026-02-20T11:15:22.170 INFO:teuthology.orchestra.run.trial043.stdout: "bdev_async_discard": "false", 2026-02-20T11:15:22.170 INFO:teuthology.orchestra.run.trial043.stdout: "bdev_async_discard_max_pending": "1000000", 2026-02-20T11:15:22.170 INFO:teuthology.orchestra.run.trial043.stdout: "bdev_async_discard_threads": "0", 2026-02-20T11:15:22.170 INFO:teuthology.orchestra.run.trial043.stdout: "bdev_block_size": "4096", 2026-02-20T11:15:22.170 INFO:teuthology.orchestra.run.trial043.stdout: "bdev_debug_aio": "false", 2026-02-20T11:15:22.170 INFO:teuthology.orchestra.run.trial043.stdout: "bdev_debug_aio_log_age": "5.000000", 2026-02-20T11:15:22.171 INFO:teuthology.orchestra.run.trial043.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2026-02-20T11:15:22.171 INFO:teuthology.orchestra.run.trial043.stdout: "bdev_debug_discard_sleep": "0", 2026-02-20T11:15:22.171 INFO:teuthology.orchestra.run.trial043.stdout: "bdev_debug_inflight_ios": "false", 2026-02-20T11:15:22.171 INFO:teuthology.orchestra.run.trial043.stdout: "bdev_discard_max_bytes": "10737418240", 2026-02-20T11:15:22.171 INFO:teuthology.orchestra.run.trial043.stdout: "bdev_enable_discard": "false", 2026-02-20T11:15:22.171 INFO:teuthology.orchestra.run.trial043.stdout: "bdev_flock_retry": "3", 2026-02-20T11:15:22.171 INFO:teuthology.orchestra.run.trial043.stdout: "bdev_flock_retry_interval": "0.100000", 2026-02-20T11:15:22.171 INFO:teuthology.orchestra.run.trial043.stdout: "bdev_inject_crash": "0", 2026-02-20T11:15:22.171 INFO:teuthology.orchestra.run.trial043.stdout: "bdev_inject_crash_flush_delay": "2", 2026-02-20T11:15:22.171 INFO:teuthology.orchestra.run.trial043.stdout: "bdev_ioring": "false", 2026-02-20T11:15:22.171 INFO:teuthology.orchestra.run.trial043.stdout: "bdev_ioring_hipri": "false", 2026-02-20T11:15:22.171 INFO:teuthology.orchestra.run.trial043.stdout: "bdev_ioring_sqthread_poll": "false", 2026-02-20T11:15:22.171 INFO:teuthology.orchestra.run.trial043.stdout: "bdev_max_discard_length": "2147483648", 2026-02-20T11:15:22.172 INFO:teuthology.orchestra.run.trial043.stdout: "bdev_nvme_unbind_from_kernel": "false", 2026-02-20T11:15:22.172 INFO:teuthology.orchestra.run.trial043.stdout: "bdev_read_buffer_alignment": "4096", 2026-02-20T11:15:22.172 INFO:teuthology.orchestra.run.trial043.stdout: "bdev_read_preallocated_huge_buffers": "", 2026-02-20T11:15:22.172 INFO:teuthology.orchestra.run.trial043.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2026-02-20T11:15:22.172 INFO:teuthology.orchestra.run.trial043.stdout: "bdev_stalled_read_warn_threshold": "1", 2026-02-20T11:15:22.172 INFO:teuthology.orchestra.run.trial043.stdout: "bdev_type": "", 2026-02-20T11:15:22.172 INFO:teuthology.orchestra.run.trial043.stdout: "bluefs_alloc_size": "1048576", 2026-02-20T11:15:22.172 INFO:teuthology.orchestra.run.trial043.stdout: "bluefs_allocator": "hybrid", 2026-02-20T11:15:22.172 INFO:teuthology.orchestra.run.trial043.stdout: "bluefs_buffered_io": "true", 2026-02-20T11:15:22.172 INFO:teuthology.orchestra.run.trial043.stdout: "bluefs_check_for_zeros": "false", 2026-02-20T11:15:22.172 INFO:teuthology.orchestra.run.trial043.stdout: "bluefs_check_volume_selector_often": "false", 2026-02-20T11:15:22.172 INFO:teuthology.orchestra.run.trial043.stdout: "bluefs_check_volume_selector_on_umount": "false", 2026-02-20T11:15:22.173 INFO:teuthology.orchestra.run.trial043.stdout: "bluefs_compact_log_sync": "false", 2026-02-20T11:15:22.173 INFO:teuthology.orchestra.run.trial043.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2026-02-20T11:15:22.173 INFO:teuthology.orchestra.run.trial043.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2026-02-20T11:15:22.173 INFO:teuthology.orchestra.run.trial043.stdout: "bluefs_log_compact_min_size": "16777216", 2026-02-20T11:15:22.173 INFO:teuthology.orchestra.run.trial043.stdout: "bluefs_log_replay_check_allocations": "true", 2026-02-20T11:15:22.173 INFO:teuthology.orchestra.run.trial043.stdout: "bluefs_max_log_runway": "4194304", 2026-02-20T11:15:22.173 INFO:teuthology.orchestra.run.trial043.stdout: "bluefs_max_prefetch": "1048576", 2026-02-20T11:15:22.174 INFO:teuthology.orchestra.run.trial043.stdout: "bluefs_min_flush_size": "524288", 2026-02-20T11:15:22.174 INFO:teuthology.orchestra.run.trial043.stdout: "bluefs_min_log_runway": "1048576", 2026-02-20T11:15:22.174 INFO:teuthology.orchestra.run.trial043.stdout: "bluefs_replay_recovery": "false", 2026-02-20T11:15:22.174 INFO:teuthology.orchestra.run.trial043.stdout: "bluefs_replay_recovery_disable_compact": "false", 2026-02-20T11:15:22.174 INFO:teuthology.orchestra.run.trial043.stdout: "bluefs_shared_alloc_size": "65536", 2026-02-20T11:15:22.174 INFO:teuthology.orchestra.run.trial043.stdout: "bluefs_sync_write": "false", 2026-02-20T11:15:22.175 INFO:teuthology.orchestra.run.trial043.stdout: "bluefs_wal_envelope_mode": "true", 2026-02-20T11:15:22.175 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2026-02-20T11:15:22.175 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2026-02-20T11:15:22.175 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2026-02-20T11:15:22.175 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_allocation_from_file": "true", 2026-02-20T11:15:22.175 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_allocator": "hybrid", 2026-02-20T11:15:22.176 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_allocator_lookup_policy": "auto", 2026-02-20T11:15:22.176 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_async_db_compaction": "true", 2026-02-20T11:15:22.176 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2026-02-20T11:15:22.176 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2026-02-20T11:15:22.176 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2026-02-20T11:15:22.176 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2026-02-20T11:15:22.176 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_bdev_label_multi": "true", 2026-02-20T11:15:22.177 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2026-02-20T11:15:22.177 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_bdev_label_require_all": "true", 2026-02-20T11:15:22.177 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2026-02-20T11:15:22.177 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_bitmapallocator_span_size": "1024", 2026-02-20T11:15:22.177 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_blobid_prealloc": "10240", 2026-02-20T11:15:22.177 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_block_create": "true", 2026-02-20T11:15:22.177 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_block_db_create": "false", 2026-02-20T11:15:22.177 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_block_db_path": "", 2026-02-20T11:15:22.177 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_block_db_size": "0", 2026-02-20T11:15:22.177 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_block_path": "", 2026-02-20T11:15:22.177 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_block_preallocate_file": "false", 2026-02-20T11:15:22.178 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_block_size": "107374182400", 2026-02-20T11:15:22.178 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_block_wal_create": "false", 2026-02-20T11:15:22.178 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_block_wal_path": "", 2026-02-20T11:15:22.178 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_block_wal_size": "100663296", 2026-02-20T11:15:22.178 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_bluefs": "true", 2026-02-20T11:15:22.178 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2026-02-20T11:15:22.178 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_bluefs_env_mirror": "false", 2026-02-20T11:15:22.178 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_bluefs_max_free": "10737418240", 2026-02-20T11:15:22.178 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2026-02-20T11:15:22.178 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2026-02-20T11:15:22.178 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2026-02-20T11:15:22.178 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2026-02-20T11:15:22.178 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2026-02-20T11:15:22.179 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2026-02-20T11:15:22.179 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_cache_autotune": "true", 2026-02-20T11:15:22.179 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_cache_autotune_interval": "5.000000", 2026-02-20T11:15:22.179 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2026-02-20T11:15:22.179 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_cache_kv_ratio": "0.450000", 2026-02-20T11:15:22.179 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_cache_meta_ratio": "0.450000", 2026-02-20T11:15:22.179 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_cache_size": "0", 2026-02-20T11:15:22.179 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_cache_size_hdd": "1073741824", 2026-02-20T11:15:22.179 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_cache_size_ssd": "3221225472", 2026-02-20T11:15:22.179 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_cache_trim_interval": "0.050000", 2026-02-20T11:15:22.179 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2026-02-20T11:15:22.179 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_cache_type": "2q", 2026-02-20T11:15:22.179 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2026-02-20T11:15:22.179 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_clone_cow": "true", 2026-02-20T11:15:22.180 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_compression_algorithm": "snappy", 2026-02-20T11:15:22.180 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_compression_max_blob_size": "0", 2026-02-20T11:15:22.180 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2026-02-20T11:15:22.180 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2026-02-20T11:15:22.180 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_compression_min_blob_size": "0", 2026-02-20T11:15:22.180 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2026-02-20T11:15:22.180 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2026-02-20T11:15:22.180 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_compression_mode": "none", 2026-02-20T11:15:22.180 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_compression_required_ratio": "0.875000", 2026-02-20T11:15:22.180 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_csum_type": "crc32c", 2026-02-20T11:15:22.180 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2026-02-20T11:15:22.180 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_debug_enforce_settings": "default", 2026-02-20T11:15:22.180 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_debug_extent_map_encode_check": "false", 2026-02-20T11:15:22.181 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_debug_freelist": "false", 2026-02-20T11:15:22.181 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_debug_fsck_abort": "false", 2026-02-20T11:15:22.181 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2026-02-20T11:15:22.181 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2026-02-20T11:15:22.181 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_debug_inject_read_err": "false", 2026-02-20T11:15:22.181 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_debug_legacy_omap": "false", 2026-02-20T11:15:22.181 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_debug_no_reuse_blocks": "false", 2026-02-20T11:15:22.182 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_debug_omit_block_device_write": "false", 2026-02-20T11:15:22.182 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_debug_omit_kv_commit": "false", 2026-02-20T11:15:22.182 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_debug_onode_segmentation_random": "false", 2026-02-20T11:15:22.182 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2026-02-20T11:15:22.182 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_debug_prefragment_max": "1048576", 2026-02-20T11:15:22.182 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_debug_random_read_err": "0.000000", 2026-02-20T11:15:22.182 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2026-02-20T11:15:22.182 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_debug_small_allocations": "0", 2026-02-20T11:15:22.183 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2026-02-20T11:15:22.183 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_default_buffered_read": "true", 2026-02-20T11:15:22.183 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_default_buffered_write": "false", 2026-02-20T11:15:22.183 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_deferred_batch_ops": "0", 2026-02-20T11:15:22.183 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2026-02-20T11:15:22.183 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2026-02-20T11:15:22.183 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_discard_on_mkfs": "true", 2026-02-20T11:15:22.183 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_elastic_shared_blobs": "true", 2026-02-20T11:15:22.183 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2026-02-20T11:15:22.183 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_extent_map_shard_max_size": "1200", 2026-02-20T11:15:22.184 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_extent_map_shard_min_size": "150", 2026-02-20T11:15:22.184 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_extent_map_shard_target_size": "500", 2026-02-20T11:15:22.184 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2026-02-20T11:15:22.184 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_fail_eio": "false", 2026-02-20T11:15:22.184 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_fragmentation_check_period": "3600", 2026-02-20T11:15:22.184 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_freelist_blocks_per_key": "128", 2026-02-20T11:15:22.184 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2026-02-20T11:15:22.184 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2026-02-20T11:15:22.184 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2026-02-20T11:15:22.184 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_fsck_on_mkfs": "true", 2026-02-20T11:15:22.184 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2026-02-20T11:15:22.184 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_fsck_on_mount": "false", 2026-02-20T11:15:22.184 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_fsck_on_mount_deep": "false", 2026-02-20T11:15:22.184 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_fsck_on_umount": "false", 2026-02-20T11:15:22.184 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_fsck_on_umount_deep": "false", 2026-02-20T11:15:22.184 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2026-02-20T11:15:22.184 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_fsck_quick_fix_threads": "2", 2026-02-20T11:15:22.185 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2026-02-20T11:15:22.185 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2026-02-20T11:15:22.185 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_gc_enable_blob_threshold": "0", 2026-02-20T11:15:22.185 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_gc_enable_total_threshold": "0", 2026-02-20T11:15:22.185 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2026-02-20T11:15:22.185 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_ignore_data_csum": "false", 2026-02-20T11:15:22.185 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2026-02-20T11:15:22.185 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_kvbackend": "rocksdb", 2026-02-20T11:15:22.185 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_log_collection_list_age": "60.000000", 2026-02-20T11:15:22.185 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2026-02-20T11:15:22.185 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_log_op_age": "5.000000", 2026-02-20T11:15:22.185 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_max_alloc_size": "0", 2026-02-20T11:15:22.185 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_max_blob_size": "0", 2026-02-20T11:15:22.185 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_max_blob_size_hdd": "65536", 2026-02-20T11:15:22.185 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_max_blob_size_ssd": "65536", 2026-02-20T11:15:22.185 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_max_defer_interval": "3.000000", 2026-02-20T11:15:22.185 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_max_deferred_txc": "32", 2026-02-20T11:15:22.186 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_min_alloc_size": "0", 2026-02-20T11:15:22.186 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_min_alloc_size_hdd": "4096", 2026-02-20T11:15:22.186 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_min_alloc_size_ssd": "4096", 2026-02-20T11:15:22.186 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_nid_prealloc": "1024", 2026-02-20T11:15:22.186 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_onode_segment_size": "0", 2026-02-20T11:15:22.186 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_prefer_deferred_size": "0", 2026-02-20T11:15:22.186 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2026-02-20T11:15:22.186 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2026-02-20T11:15:22.186 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_qfsck_on_mount": "true", 2026-02-20T11:15:22.186 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_recompression_min_gain": "1.200000", 2026-02-20T11:15:22.186 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_retry_disk_reads": "3", 2026-02-20T11:15:22.186 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_rocksdb_cf": "true", 2026-02-20T11:15:22.186 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2026-02-20T11:15:22.186 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2026-02-20T11:15:22.186 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_rocksdb_options_annex": "", 2026-02-20T11:15:22.186 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2026-02-20T11:15:22.186 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_slow_ops_warn_threshold": "1", 2026-02-20T11:15:22.186 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_spdk_coremask": "0x1", 2026-02-20T11:15:22.187 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_spdk_io_sleep": "5", 2026-02-20T11:15:22.187 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_spdk_max_io_completion": "0", 2026-02-20T11:15:22.187 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_spdk_mem": "512", 2026-02-20T11:15:22.187 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_sync_submit_transaction": "false", 2026-02-20T11:15:22.187 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_throttle_bytes": "67108864", 2026-02-20T11:15:22.187 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_throttle_cost_per_io": "0", 2026-02-20T11:15:22.187 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2026-02-20T11:15:22.187 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2026-02-20T11:15:22.187 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2026-02-20T11:15:22.187 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_throttle_trace_rate": "0.000000", 2026-02-20T11:15:22.187 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_tracing": "false", 2026-02-20T11:15:22.187 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2026-02-20T11:15:22.187 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2026-02-20T11:15:22.187 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_volume_selection_reserved": "0", 2026-02-20T11:15:22.187 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2026-02-20T11:15:22.187 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2026-02-20T11:15:22.187 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_warn_on_free_fragmentation": "0.800000", 2026-02-20T11:15:22.187 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_warn_on_legacy_statfs": "true", 2026-02-20T11:15:22.187 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2026-02-20T11:15:22.188 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2026-02-20T11:15:22.188 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2026-02-20T11:15:22.188 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_write_v2": "false", 2026-02-20T11:15:22.188 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_write_v2_random": "false", 2026-02-20T11:15:22.188 INFO:teuthology.orchestra.run.trial043.stdout: "bluestore_zero_block_detection": "false", 2026-02-20T11:15:22.188 INFO:teuthology.orchestra.run.trial043.stdout: "breakpad": "false", 2026-02-20T11:15:22.188 INFO:teuthology.orchestra.run.trial043.stdout: "ceph_assert_supresssions": "", 2026-02-20T11:15:22.188 INFO:teuthology.orchestra.run.trial043.stdout: "cephadm_path": "/usr/sbin/cephadm", 2026-02-20T11:15:22.188 INFO:teuthology.orchestra.run.trial043.stdout: "cephfs_mirror_action_update_interval": "2", 2026-02-20T11:15:22.188 INFO:teuthology.orchestra.run.trial043.stdout: "cephfs_mirror_directory_scan_interval": "10", 2026-02-20T11:15:22.188 INFO:teuthology.orchestra.run.trial043.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2026-02-20T11:15:22.188 INFO:teuthology.orchestra.run.trial043.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2026-02-20T11:15:22.188 INFO:teuthology.orchestra.run.trial043.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2026-02-20T11:15:22.188 INFO:teuthology.orchestra.run.trial043.stdout: "cephfs_mirror_mount_timeout": "10", 2026-02-20T11:15:22.188 INFO:teuthology.orchestra.run.trial043.stdout: "cephfs_mirror_perf_stats_prio": "5", 2026-02-20T11:15:22.188 INFO:teuthology.orchestra.run.trial043.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2026-02-20T11:15:22.188 INFO:teuthology.orchestra.run.trial043.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2026-02-20T11:15:22.188 INFO:teuthology.orchestra.run.trial043.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2026-02-20T11:15:22.189 INFO:teuthology.orchestra.run.trial043.stdout: "cephsqlite_blocklist_dead_locker": "true", 2026-02-20T11:15:22.189 INFO:teuthology.orchestra.run.trial043.stdout: "cephsqlite_lock_renewal_interval": "2000", 2026-02-20T11:15:22.189 INFO:teuthology.orchestra.run.trial043.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2026-02-20T11:15:22.189 INFO:teuthology.orchestra.run.trial043.stdout: "cephx_cluster_require_signatures": "false", 2026-02-20T11:15:22.189 INFO:teuthology.orchestra.run.trial043.stdout: "cephx_cluster_require_version": "2", 2026-02-20T11:15:22.189 INFO:teuthology.orchestra.run.trial043.stdout: "cephx_require_signatures": "false", 2026-02-20T11:15:22.189 INFO:teuthology.orchestra.run.trial043.stdout: "cephx_require_version": "2", 2026-02-20T11:15:22.189 INFO:teuthology.orchestra.run.trial043.stdout: "cephx_service_require_signatures": "false", 2026-02-20T11:15:22.189 INFO:teuthology.orchestra.run.trial043.stdout: "cephx_service_require_version": "2", 2026-02-20T11:15:22.189 INFO:teuthology.orchestra.run.trial043.stdout: "cephx_sign_messages": "true", 2026-02-20T11:15:22.189 INFO:teuthology.orchestra.run.trial043.stdout: "chdir": "", 2026-02-20T11:15:22.189 INFO:teuthology.orchestra.run.trial043.stdout: "client_acl_type": "", 2026-02-20T11:15:22.189 INFO:teuthology.orchestra.run.trial043.stdout: "client_asio_thread_count": "2", 2026-02-20T11:15:22.189 INFO:teuthology.orchestra.run.trial043.stdout: "client_cache_mid": "0.750000", 2026-02-20T11:15:22.189 INFO:teuthology.orchestra.run.trial043.stdout: "client_cache_size": "16384", 2026-02-20T11:15:22.189 INFO:teuthology.orchestra.run.trial043.stdout: "client_caps_release_delay": "5", 2026-02-20T11:15:22.189 INFO:teuthology.orchestra.run.trial043.stdout: "client_check_pool_perm": "true", 2026-02-20T11:15:22.189 INFO:teuthology.orchestra.run.trial043.stdout: "client_collect_and_send_global_metrics": "false", 2026-02-20T11:15:22.189 INFO:teuthology.orchestra.run.trial043.stdout: "client_debug_force_sync_read": "false", 2026-02-20T11:15:22.190 INFO:teuthology.orchestra.run.trial043.stdout: "client_debug_getattr_caps": "false", 2026-02-20T11:15:22.190 INFO:teuthology.orchestra.run.trial043.stdout: "client_debug_inject_features": "", 2026-02-20T11:15:22.190 INFO:teuthology.orchestra.run.trial043.stdout: "client_debug_inject_tick_delay": "0", 2026-02-20T11:15:22.190 INFO:teuthology.orchestra.run.trial043.stdout: "client_die_on_failed_dentry_invalidate": "true", 2026-02-20T11:15:22.190 INFO:teuthology.orchestra.run.trial043.stdout: "client_die_on_failed_remount": "false", 2026-02-20T11:15:22.190 INFO:teuthology.orchestra.run.trial043.stdout: "client_dirsize_rbytes": "true", 2026-02-20T11:15:22.190 INFO:teuthology.orchestra.run.trial043.stdout: "client_file_blockdiff_max_concurrent_object_scans": "16", 2026-02-20T11:15:22.190 INFO:teuthology.orchestra.run.trial043.stdout: "client_force_lazyio": "false", 2026-02-20T11:15:22.190 INFO:teuthology.orchestra.run.trial043.stdout: "client_fs": "", 2026-02-20T11:15:22.190 INFO:teuthology.orchestra.run.trial043.stdout: "client_fscrypt_as": "true", 2026-02-20T11:15:22.190 INFO:teuthology.orchestra.run.trial043.stdout: "client_fscrypt_dummy_encryption": "false", 2026-02-20T11:15:22.190 INFO:teuthology.orchestra.run.trial043.stdout: "client_inject_fixed_oldest_tid": "false", 2026-02-20T11:15:22.190 INFO:teuthology.orchestra.run.trial043.stdout: "client_inject_release_failure": "false", 2026-02-20T11:15:22.190 INFO:teuthology.orchestra.run.trial043.stdout: "client_inject_write_delay_secs": "0", 2026-02-20T11:15:22.190 INFO:teuthology.orchestra.run.trial043.stdout: "client_max_inline_size": "4096", 2026-02-20T11:15:22.190 INFO:teuthology.orchestra.run.trial043.stdout: "client_max_retries_on_remount_failure": "5", 2026-02-20T11:15:22.190 INFO:teuthology.orchestra.run.trial043.stdout: "client_mds_namespace": "", 2026-02-20T11:15:22.190 INFO:teuthology.orchestra.run.trial043.stdout: "client_metadata": "", 2026-02-20T11:15:22.191 INFO:teuthology.orchestra.run.trial043.stdout: "client_mount_gid": "-1", 2026-02-20T11:15:22.191 INFO:teuthology.orchestra.run.trial043.stdout: "client_mount_timeout": "300", 2026-02-20T11:15:22.191 INFO:teuthology.orchestra.run.trial043.stdout: "client_mount_uid": "-1", 2026-02-20T11:15:22.191 INFO:teuthology.orchestra.run.trial043.stdout: "client_mountpoint": "/", 2026-02-20T11:15:22.191 INFO:teuthology.orchestra.run.trial043.stdout: "client_notify_timeout": "10", 2026-02-20T11:15:22.191 INFO:teuthology.orchestra.run.trial043.stdout: "client_oc": "true", 2026-02-20T11:15:22.191 INFO:teuthology.orchestra.run.trial043.stdout: "client_oc_max_dirty": "104857600", 2026-02-20T11:15:22.191 INFO:teuthology.orchestra.run.trial043.stdout: "client_oc_max_dirty_age": "5.000000", 2026-02-20T11:15:22.191 INFO:teuthology.orchestra.run.trial043.stdout: "client_oc_max_objects": "1000", 2026-02-20T11:15:22.191 INFO:teuthology.orchestra.run.trial043.stdout: "client_oc_size": "209715200", 2026-02-20T11:15:22.191 INFO:teuthology.orchestra.run.trial043.stdout: "client_oc_target_dirty": "8388608", 2026-02-20T11:15:22.191 INFO:teuthology.orchestra.run.trial043.stdout: "client_permissions": "true", 2026-02-20T11:15:22.191 INFO:teuthology.orchestra.run.trial043.stdout: "client_quota": "true", 2026-02-20T11:15:22.191 INFO:teuthology.orchestra.run.trial043.stdout: "client_quota_df": "true", 2026-02-20T11:15:22.191 INFO:teuthology.orchestra.run.trial043.stdout: "client_readahead_max_bytes": "0", 2026-02-20T11:15:22.191 INFO:teuthology.orchestra.run.trial043.stdout: "client_readahead_max_periods": "4", 2026-02-20T11:15:22.192 INFO:teuthology.orchestra.run.trial043.stdout: "client_readahead_min": "131072", 2026-02-20T11:15:22.192 INFO:teuthology.orchestra.run.trial043.stdout: "client_reconnect_stale": "false", 2026-02-20T11:15:22.192 INFO:teuthology.orchestra.run.trial043.stdout: "client_respect_subvolume_snapshot_visibility": "false", 2026-02-20T11:15:22.192 INFO:teuthology.orchestra.run.trial043.stdout: "client_shutdown_timeout": "30", 2026-02-20T11:15:22.192 INFO:teuthology.orchestra.run.trial043.stdout: "client_snapdir": ".snap", 2026-02-20T11:15:22.192 INFO:teuthology.orchestra.run.trial043.stdout: "client_tick_interval": "1", 2026-02-20T11:15:22.192 INFO:teuthology.orchestra.run.trial043.stdout: "client_trace": "", 2026-02-20T11:15:22.192 INFO:teuthology.orchestra.run.trial043.stdout: "client_try_dentry_invalidate": "false", 2026-02-20T11:15:22.192 INFO:teuthology.orchestra.run.trial043.stdout: "client_use_faked_inos": "false", 2026-02-20T11:15:22.192 INFO:teuthology.orchestra.run.trial043.stdout: "client_use_random_mds": "false", 2026-02-20T11:15:22.192 INFO:teuthology.orchestra.run.trial043.stdout: "clog_to_graylog": "false", 2026-02-20T11:15:22.192 INFO:teuthology.orchestra.run.trial043.stdout: "clog_to_graylog_host": "127.0.0.1", 2026-02-20T11:15:22.192 INFO:teuthology.orchestra.run.trial043.stdout: "clog_to_graylog_port": "12201", 2026-02-20T11:15:22.192 INFO:teuthology.orchestra.run.trial043.stdout: "clog_to_monitors": "default=true", 2026-02-20T11:15:22.192 INFO:teuthology.orchestra.run.trial043.stdout: "clog_to_syslog": "false", 2026-02-20T11:15:22.192 INFO:teuthology.orchestra.run.trial043.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2026-02-20T11:15:22.192 INFO:teuthology.orchestra.run.trial043.stdout: "clog_to_syslog_level": "info", 2026-02-20T11:15:22.192 INFO:teuthology.orchestra.run.trial043.stdout: "cluster_addr": "-", 2026-02-20T11:15:22.192 INFO:teuthology.orchestra.run.trial043.stdout: "cluster_network": "", 2026-02-20T11:15:22.192 INFO:teuthology.orchestra.run.trial043.stdout: "cluster_network_interface": "", 2026-02-20T11:15:22.192 INFO:teuthology.orchestra.run.trial043.stdout: "compressor_zlib_isal": "false", 2026-02-20T11:15:22.193 INFO:teuthology.orchestra.run.trial043.stdout: "compressor_zlib_level": "5", 2026-02-20T11:15:22.193 INFO:teuthology.orchestra.run.trial043.stdout: "compressor_zlib_winsize": "-15", 2026-02-20T11:15:22.193 INFO:teuthology.orchestra.run.trial043.stdout: "compressor_zstd_level": "1", 2026-02-20T11:15:22.193 INFO:teuthology.orchestra.run.trial043.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2026-02-20T11:15:22.193 INFO:teuthology.orchestra.run.trial043.stdout: "crash_dir": "/var/lib/ceph/crash", 2026-02-20T11:15:22.193 INFO:teuthology.orchestra.run.trial043.stdout: "crimson_allow_pg_split": "true", 2026-02-20T11:15:22.193 INFO:teuthology.orchestra.run.trial043.stdout: "crimson_bluestore_cpu_set": "", 2026-02-20T11:15:22.193 INFO:teuthology.orchestra.run.trial043.stdout: "crimson_bluestore_num_threads": "6", 2026-02-20T11:15:22.193 INFO:teuthology.orchestra.run.trial043.stdout: "crimson_cpu_num": "0", 2026-02-20T11:15:22.193 INFO:teuthology.orchestra.run.trial043.stdout: "crimson_cpu_set": "", 2026-02-20T11:15:22.193 INFO:teuthology.orchestra.run.trial043.stdout: "crimson_osd_obc_lru_size": "512", 2026-02-20T11:15:22.193 INFO:teuthology.orchestra.run.trial043.stdout: "crimson_osd_scheduler_concurrency": "0", 2026-02-20T11:15:22.193 INFO:teuthology.orchestra.run.trial043.stdout: "crimson_osd_stat_interval": "0", 2026-02-20T11:15:22.193 INFO:teuthology.orchestra.run.trial043.stdout: "crimson_poll_mode": "false", 2026-02-20T11:15:22.193 INFO:teuthology.orchestra.run.trial043.stdout: "crimson_reactor_backend": "", 2026-02-20T11:15:22.193 INFO:teuthology.orchestra.run.trial043.stdout: "crimson_reactor_idle_poll_time_us": "200", 2026-02-20T11:15:22.193 INFO:teuthology.orchestra.run.trial043.stdout: "crimson_reactor_io_latency_goal_ms": "0.000000", 2026-02-20T11:15:22.193 INFO:teuthology.orchestra.run.trial043.stdout: "crimson_reactor_task_quota_ms": "0.500000", 2026-02-20T11:15:22.194 INFO:teuthology.orchestra.run.trial043.stdout: "crush_location": "", 2026-02-20T11:15:22.194 INFO:teuthology.orchestra.run.trial043.stdout: "crush_location_hook": "", 2026-02-20T11:15:22.194 INFO:teuthology.orchestra.run.trial043.stdout: "crush_location_hook_timeout": "10", 2026-02-20T11:15:22.194 INFO:teuthology.orchestra.run.trial043.stdout: "d4n_writecache_enabled": "false", 2026-02-20T11:15:22.194 INFO:teuthology.orchestra.run.trial043.stdout: "daemonize": "false", 2026-02-20T11:15:22.194 INFO:teuthology.orchestra.run.trial043.stdout: "daos_pool": "tank", 2026-02-20T11:15:22.194 INFO:teuthology.orchestra.run.trial043.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2026-02-20T11:15:22.194 INFO:teuthology.orchestra.run.trial043.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2026-02-20T11:15:22.194 INFO:teuthology.orchestra.run.trial043.stdout: "dbstore_db_name_prefix": "dbstore", 2026-02-20T11:15:22.194 INFO:teuthology.orchestra.run.trial043.stdout: "debug_alienstore": "0/5", 2026-02-20T11:15:22.194 INFO:teuthology.orchestra.run.trial043.stdout: "debug_allow_any_pool_priority": "false", 2026-02-20T11:15:22.194 INFO:teuthology.orchestra.run.trial043.stdout: "debug_asok": "1/5", 2026-02-20T11:15:22.194 INFO:teuthology.orchestra.run.trial043.stdout: "debug_asok_assert_abort": "false", 2026-02-20T11:15:22.194 INFO:teuthology.orchestra.run.trial043.stdout: "debug_asserts_on_shutdown": "true", 2026-02-20T11:15:22.194 INFO:teuthology.orchestra.run.trial043.stdout: "debug_auth": "20/20", 2026-02-20T11:15:22.194 INFO:teuthology.orchestra.run.trial043.stdout: "debug_bdev": "1/3", 2026-02-20T11:15:22.194 INFO:teuthology.orchestra.run.trial043.stdout: "debug_bluefs": "1/5", 2026-02-20T11:15:22.194 INFO:teuthology.orchestra.run.trial043.stdout: "debug_bluestore": "1/5", 2026-02-20T11:15:22.194 INFO:teuthology.orchestra.run.trial043.stdout: "debug_bluestore_compression": "1/5", 2026-02-20T11:15:22.194 INFO:teuthology.orchestra.run.trial043.stdout: "debug_buffer": "0/1", 2026-02-20T11:15:22.194 INFO:teuthology.orchestra.run.trial043.stdout: "debug_ceph_dedup": "0/5", 2026-02-20T11:15:22.194 INFO:teuthology.orchestra.run.trial043.stdout: "debug_ceph_exporter": "1/5", 2026-02-20T11:15:22.194 INFO:teuthology.orchestra.run.trial043.stdout: "debug_cephfs_mirror": "0/5", 2026-02-20T11:15:22.195 INFO:teuthology.orchestra.run.trial043.stdout: "debug_cephsqlite": "0/5", 2026-02-20T11:15:22.195 INFO:teuthology.orchestra.run.trial043.stdout: "debug_client": "0/5", 2026-02-20T11:15:22.195 INFO:teuthology.orchestra.run.trial043.stdout: "debug_compressor": "1/5", 2026-02-20T11:15:22.195 INFO:teuthology.orchestra.run.trial043.stdout: "debug_context": "0/1", 2026-02-20T11:15:22.195 INFO:teuthology.orchestra.run.trial043.stdout: "debug_crimson_interrupt": "0/5", 2026-02-20T11:15:22.195 INFO:teuthology.orchestra.run.trial043.stdout: "debug_crush": "1/1", 2026-02-20T11:15:22.195 INFO:teuthology.orchestra.run.trial043.stdout: "debug_crypto": "1/5", 2026-02-20T11:15:22.195 INFO:teuthology.orchestra.run.trial043.stdout: "debug_cyanstore": "0/5", 2026-02-20T11:15:22.195 INFO:teuthology.orchestra.run.trial043.stdout: "debug_deliberately_leak_memory": "false", 2026-02-20T11:15:22.195 INFO:teuthology.orchestra.run.trial043.stdout: "debug_disable_randomized_ping": "false", 2026-02-20T11:15:22.195 INFO:teuthology.orchestra.run.trial043.stdout: "debug_dpdk": "1/5", 2026-02-20T11:15:22.195 INFO:teuthology.orchestra.run.trial043.stdout: "debug_eventtrace": "1/5", 2026-02-20T11:15:22.195 INFO:teuthology.orchestra.run.trial043.stdout: "debug_filer": "0/1", 2026-02-20T11:15:22.195 INFO:teuthology.orchestra.run.trial043.stdout: "debug_filestore": "1/3", 2026-02-20T11:15:22.195 INFO:teuthology.orchestra.run.trial043.stdout: "debug_finisher": "1/1", 2026-02-20T11:15:22.195 INFO:teuthology.orchestra.run.trial043.stdout: "debug_fuse": "1/5", 2026-02-20T11:15:22.195 INFO:teuthology.orchestra.run.trial043.stdout: "debug_heartbeat_testing_span": "0", 2026-02-20T11:15:22.195 INFO:teuthology.orchestra.run.trial043.stdout: "debug_heartbeatmap": "1/5", 2026-02-20T11:15:22.195 INFO:teuthology.orchestra.run.trial043.stdout: "debug_immutable_obj_cache": "0/5", 2026-02-20T11:15:22.195 INFO:teuthology.orchestra.run.trial043.stdout: "debug_javaclient": "1/5", 2026-02-20T11:15:22.195 INFO:teuthology.orchestra.run.trial043.stdout: "debug_journal": "1/3", 2026-02-20T11:15:22.195 INFO:teuthology.orchestra.run.trial043.stdout: "debug_journaler": "0/5", 2026-02-20T11:15:22.196 INFO:teuthology.orchestra.run.trial043.stdout: "debug_kstore": "1/5", 2026-02-20T11:15:22.196 INFO:teuthology.orchestra.run.trial043.stdout: "debug_lockdep": "0/1", 2026-02-20T11:15:22.196 INFO:teuthology.orchestra.run.trial043.stdout: "debug_mclock": "1/5", 2026-02-20T11:15:22.196 INFO:teuthology.orchestra.run.trial043.stdout: "debug_mds": "1/5", 2026-02-20T11:15:22.196 INFO:teuthology.orchestra.run.trial043.stdout: "debug_mds_balancer": "1/5", 2026-02-20T11:15:22.196 INFO:teuthology.orchestra.run.trial043.stdout: "debug_mds_locker": "1/5", 2026-02-20T11:15:22.196 INFO:teuthology.orchestra.run.trial043.stdout: "debug_mds_log": "1/5", 2026-02-20T11:15:22.196 INFO:teuthology.orchestra.run.trial043.stdout: "debug_mds_log_expire": "1/5", 2026-02-20T11:15:22.196 INFO:teuthology.orchestra.run.trial043.stdout: "debug_mds_migrator": "1/5", 2026-02-20T11:15:22.196 INFO:teuthology.orchestra.run.trial043.stdout: "debug_mds_quiesce": "3/5", 2026-02-20T11:15:22.196 INFO:teuthology.orchestra.run.trial043.stdout: "debug_memstore": "1/5", 2026-02-20T11:15:22.196 INFO:teuthology.orchestra.run.trial043.stdout: "debug_mgr": "2/5", 2026-02-20T11:15:22.196 INFO:teuthology.orchestra.run.trial043.stdout: "debug_mgrc": "1/5", 2026-02-20T11:15:22.196 INFO:teuthology.orchestra.run.trial043.stdout: "debug_mon": "20/20", 2026-02-20T11:15:22.196 INFO:teuthology.orchestra.run.trial043.stdout: "debug_monc": "0/10", 2026-02-20T11:15:22.196 INFO:teuthology.orchestra.run.trial043.stdout: "debug_ms": "1/1", 2026-02-20T11:15:22.196 INFO:teuthology.orchestra.run.trial043.stdout: "debug_none": "0/5", 2026-02-20T11:15:22.196 INFO:teuthology.orchestra.run.trial043.stdout: "debug_objclass": "0/5", 2026-02-20T11:15:22.196 INFO:teuthology.orchestra.run.trial043.stdout: "debug_objectcacher": "0/5", 2026-02-20T11:15:22.196 INFO:teuthology.orchestra.run.trial043.stdout: "debug_objecter": "0/1", 2026-02-20T11:15:22.196 INFO:teuthology.orchestra.run.trial043.stdout: "debug_optracker": "0/5", 2026-02-20T11:15:22.196 INFO:teuthology.orchestra.run.trial043.stdout: "debug_osd": "1/5", 2026-02-20T11:15:22.197 INFO:teuthology.orchestra.run.trial043.stdout: "debug_paxos": "20/20", 2026-02-20T11:15:22.197 INFO:teuthology.orchestra.run.trial043.stdout: "debug_perfcounter": "1/5", 2026-02-20T11:15:22.197 INFO:teuthology.orchestra.run.trial043.stdout: "debug_prioritycache": "1/5", 2026-02-20T11:15:22.197 INFO:teuthology.orchestra.run.trial043.stdout: "debug_rados": "0/5", 2026-02-20T11:15:22.197 INFO:teuthology.orchestra.run.trial043.stdout: "debug_rbd": "0/5", 2026-02-20T11:15:22.197 INFO:teuthology.orchestra.run.trial043.stdout: "debug_rbd_mirror": "0/5", 2026-02-20T11:15:22.197 INFO:teuthology.orchestra.run.trial043.stdout: "debug_rbd_pwl": "0/5", 2026-02-20T11:15:22.197 INFO:teuthology.orchestra.run.trial043.stdout: "debug_rbd_replay": "0/5", 2026-02-20T11:15:22.197 INFO:teuthology.orchestra.run.trial043.stdout: "debug_refs": "0/0", 2026-02-20T11:15:22.197 INFO:teuthology.orchestra.run.trial043.stdout: "debug_reserver": "1/1", 2026-02-20T11:15:22.197 INFO:teuthology.orchestra.run.trial043.stdout: "debug_rgw": "1/5", 2026-02-20T11:15:22.197 INFO:teuthology.orchestra.run.trial043.stdout: "debug_rgw_access": "1/5", 2026-02-20T11:15:22.197 INFO:teuthology.orchestra.run.trial043.stdout: "debug_rgw_bucket_logging": "1/5", 2026-02-20T11:15:22.197 INFO:teuthology.orchestra.run.trial043.stdout: "debug_rgw_datacache": "1/5", 2026-02-20T11:15:22.197 INFO:teuthology.orchestra.run.trial043.stdout: "debug_rgw_dbstore": "1/5", 2026-02-20T11:15:22.197 INFO:teuthology.orchestra.run.trial043.stdout: "debug_rgw_dedup": "1/5", 2026-02-20T11:15:22.197 INFO:teuthology.orchestra.run.trial043.stdout: "debug_rgw_flight": "1/5", 2026-02-20T11:15:22.197 INFO:teuthology.orchestra.run.trial043.stdout: "debug_rgw_lifecycle": "1/5", 2026-02-20T11:15:22.197 INFO:teuthology.orchestra.run.trial043.stdout: "debug_rgw_notification": "1/5", 2026-02-20T11:15:22.197 INFO:teuthology.orchestra.run.trial043.stdout: "debug_rgw_restore": "1/5", 2026-02-20T11:15:22.197 INFO:teuthology.orchestra.run.trial043.stdout: "debug_rgw_sync": "1/5", 2026-02-20T11:15:22.197 INFO:teuthology.orchestra.run.trial043.stdout: "debug_rocksdb": "4/5", 2026-02-20T11:15:22.197 INFO:teuthology.orchestra.run.trial043.stdout: "debug_seastore": "0/5", 2026-02-20T11:15:22.197 INFO:teuthology.orchestra.run.trial043.stdout: "debug_seastore_backref": "0/5", 2026-02-20T11:15:22.198 INFO:teuthology.orchestra.run.trial043.stdout: "debug_seastore_cache": "0/5", 2026-02-20T11:15:22.198 INFO:teuthology.orchestra.run.trial043.stdout: "debug_seastore_cleaner": "0/5", 2026-02-20T11:15:22.198 INFO:teuthology.orchestra.run.trial043.stdout: "debug_seastore_device": "0/5", 2026-02-20T11:15:22.198 INFO:teuthology.orchestra.run.trial043.stdout: "debug_seastore_epm": "0/5", 2026-02-20T11:15:22.198 INFO:teuthology.orchestra.run.trial043.stdout: "debug_seastore_fixedkv_tree": "0/5", 2026-02-20T11:15:22.198 INFO:teuthology.orchestra.run.trial043.stdout: "debug_seastore_journal": "0/5", 2026-02-20T11:15:22.198 INFO:teuthology.orchestra.run.trial043.stdout: "debug_seastore_lba": "0/5", 2026-02-20T11:15:22.198 INFO:teuthology.orchestra.run.trial043.stdout: "debug_seastore_odata": "0/5", 2026-02-20T11:15:22.198 INFO:teuthology.orchestra.run.trial043.stdout: "debug_seastore_omap": "0/5", 2026-02-20T11:15:22.198 INFO:teuthology.orchestra.run.trial043.stdout: "debug_seastore_onode": "0/5", 2026-02-20T11:15:22.198 INFO:teuthology.orchestra.run.trial043.stdout: "debug_seastore_t": "0/5", 2026-02-20T11:15:22.198 INFO:teuthology.orchestra.run.trial043.stdout: "debug_seastore_tm": "0/5", 2026-02-20T11:15:22.198 INFO:teuthology.orchestra.run.trial043.stdout: "debug_striper": "0/1", 2026-02-20T11:15:22.198 INFO:teuthology.orchestra.run.trial043.stdout: "debug_test": "0/5", 2026-02-20T11:15:22.198 INFO:teuthology.orchestra.run.trial043.stdout: "debug_throttle": "1/1", 2026-02-20T11:15:22.198 INFO:teuthology.orchestra.run.trial043.stdout: "debug_timer": "0/1", 2026-02-20T11:15:22.198 INFO:teuthology.orchestra.run.trial043.stdout: "debug_tp": "0/5", 2026-02-20T11:15:22.198 INFO:teuthology.orchestra.run.trial043.stdout: "debug_trace": "1/5", 2026-02-20T11:15:22.198 INFO:teuthology.orchestra.run.trial043.stdout: "defer_client_eviction_on_laggy_osds": "false", 2026-02-20T11:15:22.198 INFO:teuthology.orchestra.run.trial043.stdout: "device_failure_prediction_mode": "none", 2026-02-20T11:15:22.198 INFO:teuthology.orchestra.run.trial043.stdout: "ec_extent_cache_size": "10485760", 2026-02-20T11:15:22.198 INFO:teuthology.orchestra.run.trial043.stdout: "ec_pdw_write_mode": "0", 2026-02-20T11:15:22.198 INFO:teuthology.orchestra.run.trial043.stdout: "enable_availability_tracking": "false", 2026-02-20T11:15:22.198 INFO:teuthology.orchestra.run.trial043.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "crimson", 2026-02-20T11:15:22.199 INFO:teuthology.orchestra.run.trial043.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2026-02-20T11:15:22.199 INFO:teuthology.orchestra.run.trial043.stdout: "err_to_graylog": "false", 2026-02-20T11:15:22.199 INFO:teuthology.orchestra.run.trial043.stdout: "err_to_journald": "false", 2026-02-20T11:15:22.199 INFO:teuthology.orchestra.run.trial043.stdout: "err_to_stderr": "true", 2026-02-20T11:15:22.199 INFO:teuthology.orchestra.run.trial043.stdout: "err_to_syslog": "false", 2026-02-20T11:15:22.199 INFO:teuthology.orchestra.run.trial043.stdout: "event_tracing": "false", 2026-02-20T11:15:22.199 INFO:teuthology.orchestra.run.trial043.stdout: "exporter_addr": "0.0.0.0", 2026-02-20T11:15:22.199 INFO:teuthology.orchestra.run.trial043.stdout: "exporter_cert_file": "", 2026-02-20T11:15:22.199 INFO:teuthology.orchestra.run.trial043.stdout: "exporter_http_port": "9926", 2026-02-20T11:15:22.199 INFO:teuthology.orchestra.run.trial043.stdout: "exporter_key_file": "", 2026-02-20T11:15:22.199 INFO:teuthology.orchestra.run.trial043.stdout: "exporter_prio_limit": "5", 2026-02-20T11:15:22.199 INFO:teuthology.orchestra.run.trial043.stdout: "exporter_sock_dir": "/var/run/ceph/", 2026-02-20T11:15:22.199 INFO:teuthology.orchestra.run.trial043.stdout: "exporter_sort_metrics": "true", 2026-02-20T11:15:22.199 INFO:teuthology.orchestra.run.trial043.stdout: "exporter_stats_period": "5", 2026-02-20T11:15:22.199 INFO:teuthology.orchestra.run.trial043.stdout: "fake_statfs_for_testing": "0", 2026-02-20T11:15:22.199 INFO:teuthology.orchestra.run.trial043.stdout: "fatal_signal_handlers": "true", 2026-02-20T11:15:22.199 INFO:teuthology.orchestra.run.trial043.stdout: "filer_max_purge_ops": "10", 2026-02-20T11:15:22.199 INFO:teuthology.orchestra.run.trial043.stdout: "filer_max_truncate_ops": "128", 2026-02-20T11:15:22.199 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_apply_finisher_threads": "1", 2026-02-20T11:15:22.199 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_blackhole": "false", 2026-02-20T11:15:22.199 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_btrfs_clone_range": "true", 2026-02-20T11:15:22.199 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_btrfs_snap": "true", 2026-02-20T11:15:22.199 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_caller_concurrency": "10", 2026-02-20T11:15:22.199 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_collect_device_partition_information": "true", 2026-02-20T11:15:22.199 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_commit_timeout": "600.000000", 2026-02-20T11:15:22.199 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_debug_inject_read_err": "false", 2026-02-20T11:15:22.199 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_debug_omap_check": "false", 2026-02-20T11:15:22.200 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_debug_verify_split": "false", 2026-02-20T11:15:22.200 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_dump_file": "", 2026-02-20T11:15:22.200 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2026-02-20T11:15:22.200 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_expected_throughput_ops": "200.000000", 2026-02-20T11:15:22.200 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_fadvise": "true", 2026-02-20T11:15:22.200 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_fail_eio": "true", 2026-02-20T11:15:22.200 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_fd_cache_shards": "16", 2026-02-20T11:15:22.200 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_fd_cache_size": "128", 2026-02-20T11:15:22.200 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_fiemap": "false", 2026-02-20T11:15:22.200 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_fiemap_threshold": "4096", 2026-02-20T11:15:22.200 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_fsync_flushes_journal_data": "false", 2026-02-20T11:15:22.200 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_index_retry_probability": "0.000000", 2026-02-20T11:15:22.200 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_inject_stall": "0", 2026-02-20T11:15:22.200 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_journal_parallel": "false", 2026-02-20T11:15:22.200 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_journal_trailing": "false", 2026-02-20T11:15:22.200 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_journal_writeahead": "false", 2026-02-20T11:15:22.200 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_kill_at": "0", 2026-02-20T11:15:22.200 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_max_alloc_hint_size": "1048576", 2026-02-20T11:15:22.200 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_max_inline_xattr_size": "0", 2026-02-20T11:15:22.200 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2026-02-20T11:15:22.200 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_max_inline_xattr_size_other": "512", 2026-02-20T11:15:22.200 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2026-02-20T11:15:22.200 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_max_inline_xattrs": "0", 2026-02-20T11:15:22.200 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2026-02-20T11:15:22.200 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_max_inline_xattrs_other": "2", 2026-02-20T11:15:22.201 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_max_inline_xattrs_xfs": "10", 2026-02-20T11:15:22.201 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_max_sync_interval": "5.000000", 2026-02-20T11:15:22.201 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_max_xattr_value_size": "0", 2026-02-20T11:15:22.201 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2026-02-20T11:15:22.201 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_max_xattr_value_size_other": "1024", 2026-02-20T11:15:22.201 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2026-02-20T11:15:22.201 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_merge_threshold": "-10", 2026-02-20T11:15:22.201 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_min_sync_interval": "0.010000", 2026-02-20T11:15:22.201 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_odsync_write": "false", 2026-02-20T11:15:22.201 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_omap_backend": "rocksdb", 2026-02-20T11:15:22.201 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_omap_backend_path": "", 2026-02-20T11:15:22.201 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_omap_header_cache_size": "1024", 2026-02-20T11:15:22.201 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_ondisk_finisher_threads": "1", 2026-02-20T11:15:22.201 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_op_thread_suicide_timeout": "180", 2026-02-20T11:15:22.201 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_op_thread_timeout": "60", 2026-02-20T11:15:22.201 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_op_threads": "2", 2026-02-20T11:15:22.201 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_punch_hole": "false", 2026-02-20T11:15:22.201 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2026-02-20T11:15:22.201 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2026-02-20T11:15:22.201 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2026-02-20T11:15:22.201 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_queue_high_threshhold": "0.900000", 2026-02-20T11:15:22.201 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_queue_low_threshhold": "0.300000", 2026-02-20T11:15:22.201 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_queue_max_bytes": "104857600", 2026-02-20T11:15:22.201 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2026-02-20T11:15:22.201 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2026-02-20T11:15:22.201 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2026-02-20T11:15:22.201 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_queue_max_ops": "50", 2026-02-20T11:15:22.202 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2026-02-20T11:15:22.202 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_seek_data_hole": "false", 2026-02-20T11:15:22.202 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_sloppy_crc": "false", 2026-02-20T11:15:22.202 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_sloppy_crc_block_size": "65536", 2026-02-20T11:15:22.202 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_splice": "false", 2026-02-20T11:15:22.202 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_split_multiple": "2", 2026-02-20T11:15:22.202 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_split_rand_factor": "20", 2026-02-20T11:15:22.202 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_update_to": "1000", 2026-02-20T11:15:22.202 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2026-02-20T11:15:22.202 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2026-02-20T11:15:22.202 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2026-02-20T11:15:22.202 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2026-02-20T11:15:22.202 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2026-02-20T11:15:22.202 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2026-02-20T11:15:22.202 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_wbthrottle_enable": "true", 2026-02-20T11:15:22.203 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2026-02-20T11:15:22.203 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2026-02-20T11:15:22.203 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2026-02-20T11:15:22.203 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2026-02-20T11:15:22.203 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2026-02-20T11:15:22.203 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2026-02-20T11:15:22.203 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_xfs_extsize": "false", 2026-02-20T11:15:22.203 INFO:teuthology.orchestra.run.trial043.stdout: "filestore_zfs_snap": "false", 2026-02-20T11:15:22.203 INFO:teuthology.orchestra.run.trial043.stdout: "fio_dir": "/tmp/fio", 2026-02-20T11:15:22.203 INFO:teuthology.orchestra.run.trial043.stdout: "fsid": "dfee6dd4-7cac-4c06-aaed-6868a4b5b9a0", 2026-02-20T11:15:22.203 INFO:teuthology.orchestra.run.trial043.stdout: "fuse_allow_other": "true", 2026-02-20T11:15:22.203 INFO:teuthology.orchestra.run.trial043.stdout: "fuse_atomic_o_trunc": "true", 2026-02-20T11:15:22.203 INFO:teuthology.orchestra.run.trial043.stdout: "fuse_big_writes": "true", 2026-02-20T11:15:22.204 INFO:teuthology.orchestra.run.trial043.stdout: "fuse_debug": "false", 2026-02-20T11:15:22.204 INFO:teuthology.orchestra.run.trial043.stdout: "fuse_default_permissions": "false", 2026-02-20T11:15:22.204 INFO:teuthology.orchestra.run.trial043.stdout: "fuse_disable_pagecache": "false", 2026-02-20T11:15:22.204 INFO:teuthology.orchestra.run.trial043.stdout: "fuse_max_write": "0", 2026-02-20T11:15:22.204 INFO:teuthology.orchestra.run.trial043.stdout: "fuse_multithreaded": "true", 2026-02-20T11:15:22.204 INFO:teuthology.orchestra.run.trial043.stdout: "fuse_require_active_mds": "true", 2026-02-20T11:15:22.204 INFO:teuthology.orchestra.run.trial043.stdout: "fuse_set_user_groups": "true", 2026-02-20T11:15:22.204 INFO:teuthology.orchestra.run.trial043.stdout: "fuse_splice_move": "true", 2026-02-20T11:15:22.204 INFO:teuthology.orchestra.run.trial043.stdout: "fuse_splice_read": "true", 2026-02-20T11:15:22.204 INFO:teuthology.orchestra.run.trial043.stdout: "fuse_splice_write": "true", 2026-02-20T11:15:22.204 INFO:teuthology.orchestra.run.trial043.stdout: "fuse_syncfs_on_mksnap": "true", 2026-02-20T11:15:22.204 INFO:teuthology.orchestra.run.trial043.stdout: "fuse_use_invalidate_cb": "true", 2026-02-20T11:15:22.204 INFO:teuthology.orchestra.run.trial043.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2026-02-20T11:15:22.204 INFO:teuthology.orchestra.run.trial043.stdout: "gss_target_name": "ceph", 2026-02-20T11:15:22.205 INFO:teuthology.orchestra.run.trial043.stdout: "heartbeat_file": "", 2026-02-20T11:15:22.205 INFO:teuthology.orchestra.run.trial043.stdout: "heartbeat_inject_failure": "0", 2026-02-20T11:15:22.205 INFO:teuthology.orchestra.run.trial043.stdout: "heartbeat_interval": "5", 2026-02-20T11:15:22.205 INFO:teuthology.orchestra.run.trial043.stdout: "host": "trial043", 2026-02-20T11:15:22.205 INFO:teuthology.orchestra.run.trial043.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2026-02-20T11:15:22.205 INFO:teuthology.orchestra.run.trial043.stdout: "immutable_object_cache_max_inflight_ops": "128", 2026-02-20T11:15:22.205 INFO:teuthology.orchestra.run.trial043.stdout: "immutable_object_cache_max_size": "1073741824", 2026-02-20T11:15:22.205 INFO:teuthology.orchestra.run.trial043.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2026-02-20T11:15:22.205 INFO:teuthology.orchestra.run.trial043.stdout: "immutable_object_cache_qos_bps_burst": "0", 2026-02-20T11:15:22.205 INFO:teuthology.orchestra.run.trial043.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2026-02-20T11:15:22.205 INFO:teuthology.orchestra.run.trial043.stdout: "immutable_object_cache_qos_bps_limit": "0", 2026-02-20T11:15:22.205 INFO:teuthology.orchestra.run.trial043.stdout: "immutable_object_cache_qos_iops_burst": "0", 2026-02-20T11:15:22.205 INFO:teuthology.orchestra.run.trial043.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2026-02-20T11:15:22.205 INFO:teuthology.orchestra.run.trial043.stdout: "immutable_object_cache_qos_iops_limit": "0", 2026-02-20T11:15:22.205 INFO:teuthology.orchestra.run.trial043.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2026-02-20T11:15:22.205 INFO:teuthology.orchestra.run.trial043.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2026-02-20T11:15:22.205 INFO:teuthology.orchestra.run.trial043.stdout: "immutable_object_cache_watermark": "0.900000", 2026-02-20T11:15:22.205 INFO:teuthology.orchestra.run.trial043.stdout: "inject_early_sigterm": "false", 2026-02-20T11:15:22.205 INFO:teuthology.orchestra.run.trial043.stdout: "jaeger_agent_port": "6799", 2026-02-20T11:15:22.205 INFO:teuthology.orchestra.run.trial043.stdout: "jaeger_tracing_enable": "false", 2026-02-20T11:15:22.205 INFO:teuthology.orchestra.run.trial043.stdout: "journal_aio": "true", 2026-02-20T11:15:22.205 INFO:teuthology.orchestra.run.trial043.stdout: "journal_align_min_size": "65536", 2026-02-20T11:15:22.205 INFO:teuthology.orchestra.run.trial043.stdout: "journal_block_align": "true", 2026-02-20T11:15:22.205 INFO:teuthology.orchestra.run.trial043.stdout: "journal_block_size": "4096", 2026-02-20T11:15:22.205 INFO:teuthology.orchestra.run.trial043.stdout: "journal_dio": "true", 2026-02-20T11:15:22.205 INFO:teuthology.orchestra.run.trial043.stdout: "journal_discard": "false", 2026-02-20T11:15:22.206 INFO:teuthology.orchestra.run.trial043.stdout: "journal_force_aio": "false", 2026-02-20T11:15:22.206 INFO:teuthology.orchestra.run.trial043.stdout: "journal_ignore_corruption": "false", 2026-02-20T11:15:22.206 INFO:teuthology.orchestra.run.trial043.stdout: "journal_max_write_bytes": "10485760", 2026-02-20T11:15:22.206 INFO:teuthology.orchestra.run.trial043.stdout: "journal_max_write_entries": "100", 2026-02-20T11:15:22.206 INFO:teuthology.orchestra.run.trial043.stdout: "journal_replay_from": "0", 2026-02-20T11:15:22.206 INFO:teuthology.orchestra.run.trial043.stdout: "journal_throttle_high_multiple": "0.000000", 2026-02-20T11:15:22.206 INFO:teuthology.orchestra.run.trial043.stdout: "journal_throttle_high_threshhold": "0.900000", 2026-02-20T11:15:22.206 INFO:teuthology.orchestra.run.trial043.stdout: "journal_throttle_low_threshhold": "0.600000", 2026-02-20T11:15:22.206 INFO:teuthology.orchestra.run.trial043.stdout: "journal_throttle_max_multiple": "0.000000", 2026-02-20T11:15:22.206 INFO:teuthology.orchestra.run.trial043.stdout: "journal_write_header_frequency": "0", 2026-02-20T11:15:22.206 INFO:teuthology.orchestra.run.trial043.stdout: "journal_zero_on_create": "false", 2026-02-20T11:15:22.206 INFO:teuthology.orchestra.run.trial043.stdout: "journaler_prefetch_periods": "10", 2026-02-20T11:15:22.206 INFO:teuthology.orchestra.run.trial043.stdout: "journaler_prezero_periods": "5", 2026-02-20T11:15:22.206 INFO:teuthology.orchestra.run.trial043.stdout: "journaler_write_head_interval": "15", 2026-02-20T11:15:22.206 INFO:teuthology.orchestra.run.trial043.stdout: "key": "", 2026-02-20T11:15:22.206 INFO:teuthology.orchestra.run.trial043.stdout: "keyfile": "", 2026-02-20T11:15:22.206 INFO:teuthology.orchestra.run.trial043.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2026-02-20T11:15:22.206 INFO:teuthology.orchestra.run.trial043.stdout: "librados_thread_count": "2", 2026-02-20T11:15:22.206 INFO:teuthology.orchestra.run.trial043.stdout: "lockdep": "false", 2026-02-20T11:15:22.206 INFO:teuthology.orchestra.run.trial043.stdout: "lockdep_force_backtrace": "false", 2026-02-20T11:15:22.206 INFO:teuthology.orchestra.run.trial043.stdout: "log_coarse_timestamps": "true", 2026-02-20T11:15:22.206 INFO:teuthology.orchestra.run.trial043.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2026-02-20T11:15:22.206 INFO:teuthology.orchestra.run.trial043.stdout: "log_flush_on_exit": "false", 2026-02-20T11:15:22.206 INFO:teuthology.orchestra.run.trial043.stdout: "log_graylog_host": "127.0.0.1", 2026-02-20T11:15:22.206 INFO:teuthology.orchestra.run.trial043.stdout: "log_graylog_port": "12201", 2026-02-20T11:15:22.206 INFO:teuthology.orchestra.run.trial043.stdout: "log_max_new": "1000", 2026-02-20T11:15:22.206 INFO:teuthology.orchestra.run.trial043.stdout: "log_max_recent": "10000", 2026-02-20T11:15:22.206 INFO:teuthology.orchestra.run.trial043.stdout: "log_stderr_prefix": "", 2026-02-20T11:15:22.207 INFO:teuthology.orchestra.run.trial043.stdout: "log_stop_at_utilization": "0.970000", 2026-02-20T11:15:22.207 INFO:teuthology.orchestra.run.trial043.stdout: "log_to_file": "true", 2026-02-20T11:15:22.207 INFO:teuthology.orchestra.run.trial043.stdout: "log_to_graylog": "false", 2026-02-20T11:15:22.207 INFO:teuthology.orchestra.run.trial043.stdout: "log_to_journald": "false", 2026-02-20T11:15:22.207 INFO:teuthology.orchestra.run.trial043.stdout: "log_to_stderr": "false", 2026-02-20T11:15:22.207 INFO:teuthology.orchestra.run.trial043.stdout: "log_to_syslog": "false", 2026-02-20T11:15:22.207 INFO:teuthology.orchestra.run.trial043.stdout: "max_rotating_auth_attempts": "10", 2026-02-20T11:15:22.207 INFO:teuthology.orchestra.run.trial043.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2026-02-20T11:15:22.207 INFO:teuthology.orchestra.run.trial043.stdout: "mds_action_on_write_error": "1", 2026-02-20T11:15:22.207 INFO:teuthology.orchestra.run.trial043.stdout: "mds_allow_async_dirops": "true", 2026-02-20T11:15:22.207 INFO:teuthology.orchestra.run.trial043.stdout: "mds_allow_batched_ops": "true", 2026-02-20T11:15:22.207 INFO:teuthology.orchestra.run.trial043.stdout: "mds_alternate_name_max": "8192", 2026-02-20T11:15:22.207 INFO:teuthology.orchestra.run.trial043.stdout: "mds_asio_thread_count": "2", 2026-02-20T11:15:22.207 INFO:teuthology.orchestra.run.trial043.stdout: "mds_bal_export_pin": "true", 2026-02-20T11:15:22.207 INFO:teuthology.orchestra.run.trial043.stdout: "mds_bal_fragment_dirs": "true", 2026-02-20T11:15:22.207 INFO:teuthology.orchestra.run.trial043.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2026-02-20T11:15:22.207 INFO:teuthology.orchestra.run.trial043.stdout: "mds_bal_fragment_interval": "5", 2026-02-20T11:15:22.207 INFO:teuthology.orchestra.run.trial043.stdout: "mds_bal_fragment_size_max": "100000", 2026-02-20T11:15:22.207 INFO:teuthology.orchestra.run.trial043.stdout: "mds_bal_idle_threshold": "0.000000", 2026-02-20T11:15:22.207 INFO:teuthology.orchestra.run.trial043.stdout: "mds_bal_interval": "10", 2026-02-20T11:15:22.207 INFO:teuthology.orchestra.run.trial043.stdout: "mds_bal_max": "-1", 2026-02-20T11:15:22.207 INFO:teuthology.orchestra.run.trial043.stdout: "mds_bal_max_until": "-1", 2026-02-20T11:15:22.207 INFO:teuthology.orchestra.run.trial043.stdout: "mds_bal_merge_size": "50", 2026-02-20T11:15:22.207 INFO:teuthology.orchestra.run.trial043.stdout: "mds_bal_midchunk": "0.300000", 2026-02-20T11:15:22.207 INFO:teuthology.orchestra.run.trial043.stdout: "mds_bal_min_rebalance": "0.100000", 2026-02-20T11:15:22.207 INFO:teuthology.orchestra.run.trial043.stdout: "mds_bal_min_start": "0.200000", 2026-02-20T11:15:22.207 INFO:teuthology.orchestra.run.trial043.stdout: "mds_bal_minchunk": "0.001000", 2026-02-20T11:15:22.207 INFO:teuthology.orchestra.run.trial043.stdout: "mds_bal_mode": "0", 2026-02-20T11:15:22.207 INFO:teuthology.orchestra.run.trial043.stdout: "mds_bal_need_max": "1.200000", 2026-02-20T11:15:22.208 INFO:teuthology.orchestra.run.trial043.stdout: "mds_bal_need_min": "0.800000", 2026-02-20T11:15:22.208 INFO:teuthology.orchestra.run.trial043.stdout: "mds_bal_overload_epochs": "2", 2026-02-20T11:15:22.208 INFO:teuthology.orchestra.run.trial043.stdout: "mds_bal_replicate_threshold": "8000.000000", 2026-02-20T11:15:22.208 INFO:teuthology.orchestra.run.trial043.stdout: "mds_bal_sample_interval": "3.000000", 2026-02-20T11:15:22.208 INFO:teuthology.orchestra.run.trial043.stdout: "mds_bal_split_bits": "3", 2026-02-20T11:15:22.208 INFO:teuthology.orchestra.run.trial043.stdout: "mds_bal_split_rd": "25000.000000", 2026-02-20T11:15:22.208 INFO:teuthology.orchestra.run.trial043.stdout: "mds_bal_split_size": "10000", 2026-02-20T11:15:22.208 INFO:teuthology.orchestra.run.trial043.stdout: "mds_bal_split_wr": "10000.000000", 2026-02-20T11:15:22.208 INFO:teuthology.orchestra.run.trial043.stdout: "mds_bal_target_decay": "10.000000", 2026-02-20T11:15:22.208 INFO:teuthology.orchestra.run.trial043.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2026-02-20T11:15:22.208 INFO:teuthology.orchestra.run.trial043.stdout: "mds_beacon_grace": "15.000000", 2026-02-20T11:15:22.208 INFO:teuthology.orchestra.run.trial043.stdout: "mds_beacon_interval": "4.000000", 2026-02-20T11:15:22.208 INFO:teuthology.orchestra.run.trial043.stdout: "mds_beacon_mon_down_grace": "60", 2026-02-20T11:15:22.208 INFO:teuthology.orchestra.run.trial043.stdout: "mds_cache_memory_limit": "4294967296", 2026-02-20T11:15:22.208 INFO:teuthology.orchestra.run.trial043.stdout: "mds_cache_mid": "0.700000", 2026-02-20T11:15:22.208 INFO:teuthology.orchestra.run.trial043.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2026-02-20T11:15:22.208 INFO:teuthology.orchestra.run.trial043.stdout: "mds_cache_quiesce_delay": "0", 2026-02-20T11:15:22.208 INFO:teuthology.orchestra.run.trial043.stdout: "mds_cache_quiesce_sleep": "200", 2026-02-20T11:15:22.208 INFO:teuthology.orchestra.run.trial043.stdout: "mds_cache_quiesce_splitauth": "true", 2026-02-20T11:15:22.208 INFO:teuthology.orchestra.run.trial043.stdout: "mds_cache_quiesce_threshold": "524288", 2026-02-20T11:15:22.208 INFO:teuthology.orchestra.run.trial043.stdout: "mds_cache_release_free_interval": "10", 2026-02-20T11:15:22.208 INFO:teuthology.orchestra.run.trial043.stdout: "mds_cache_reservation": "0.050000", 2026-02-20T11:15:22.208 INFO:teuthology.orchestra.run.trial043.stdout: "mds_cache_trim_decay_rate": "1.000000", 2026-02-20T11:15:22.208 INFO:teuthology.orchestra.run.trial043.stdout: "mds_cache_trim_interval": "1", 2026-02-20T11:15:22.208 INFO:teuthology.orchestra.run.trial043.stdout: "mds_cache_trim_threshold": "262144", 2026-02-20T11:15:22.208 INFO:teuthology.orchestra.run.trial043.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2026-02-20T11:15:22.208 INFO:teuthology.orchestra.run.trial043.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2026-02-20T11:15:22.208 INFO:teuthology.orchestra.run.trial043.stdout: "mds_client_delegate_inos_pct": "50", 2026-02-20T11:15:22.208 INFO:teuthology.orchestra.run.trial043.stdout: "mds_client_prealloc_inos": "1000", 2026-02-20T11:15:22.209 INFO:teuthology.orchestra.run.trial043.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2026-02-20T11:15:22.209 INFO:teuthology.orchestra.run.trial043.stdout: "mds_connect_bootstrapping": "false", 2026-02-20T11:15:22.209 INFO:teuthology.orchestra.run.trial043.stdout: "mds_damage_table_max_entries": "10000", 2026-02-20T11:15:22.209 INFO:teuthology.orchestra.run.trial043.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2026-02-20T11:15:22.209 INFO:teuthology.orchestra.run.trial043.stdout: "mds_debug_auth_pins": "false", 2026-02-20T11:15:22.209 INFO:teuthology.orchestra.run.trial043.stdout: "mds_debug_frag": "false", 2026-02-20T11:15:22.209 INFO:teuthology.orchestra.run.trial043.stdout: "mds_debug_scatterstat": "false", 2026-02-20T11:15:22.209 INFO:teuthology.orchestra.run.trial043.stdout: "mds_debug_subtrees": "false", 2026-02-20T11:15:22.209 INFO:teuthology.orchestra.run.trial043.stdout: "mds_decay_halflife": "5.000000", 2026-02-20T11:15:22.209 INFO:teuthology.orchestra.run.trial043.stdout: "mds_default_dir_hash": "2", 2026-02-20T11:15:22.209 INFO:teuthology.orchestra.run.trial043.stdout: "mds_defer_session_stale": "true", 2026-02-20T11:15:22.209 INFO:teuthology.orchestra.run.trial043.stdout: "mds_delay_journal_replay_for_testing": "0", 2026-02-20T11:15:22.209 INFO:teuthology.orchestra.run.trial043.stdout: "mds_deny_all_reconnect": "false", 2026-02-20T11:15:22.209 INFO:teuthology.orchestra.run.trial043.stdout: "mds_dir_keys_per_op": "16384", 2026-02-20T11:15:22.209 INFO:teuthology.orchestra.run.trial043.stdout: "mds_dir_max_commit_size": "10", 2026-02-20T11:15:22.209 INFO:teuthology.orchestra.run.trial043.stdout: "mds_dir_max_entries": "0", 2026-02-20T11:15:22.209 INFO:teuthology.orchestra.run.trial043.stdout: "mds_dir_prefetch": "true", 2026-02-20T11:15:22.209 INFO:teuthology.orchestra.run.trial043.stdout: "mds_dirstat_min_interval": "1.000000", 2026-02-20T11:15:22.209 INFO:teuthology.orchestra.run.trial043.stdout: "mds_dump_cache_after_rejoin": "false", 2026-02-20T11:15:22.209 INFO:teuthology.orchestra.run.trial043.stdout: "mds_dump_cache_on_map": "false", 2026-02-20T11:15:22.209 INFO:teuthology.orchestra.run.trial043.stdout: "mds_dump_cache_threshold_file": "0", 2026-02-20T11:15:22.209 INFO:teuthology.orchestra.run.trial043.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2026-02-20T11:15:22.209 INFO:teuthology.orchestra.run.trial043.stdout: "mds_early_reply": "true", 2026-02-20T11:15:22.209 INFO:teuthology.orchestra.run.trial043.stdout: "mds_enable_op_tracker": "true", 2026-02-20T11:15:22.209 INFO:teuthology.orchestra.run.trial043.stdout: "mds_enforce_unique_name": "true", 2026-02-20T11:15:22.209 INFO:teuthology.orchestra.run.trial043.stdout: "mds_export_ephemeral_distributed": "true", 2026-02-20T11:15:22.209 INFO:teuthology.orchestra.run.trial043.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2026-02-20T11:15:22.209 INFO:teuthology.orchestra.run.trial043.stdout: "mds_export_ephemeral_random": "true", 2026-02-20T11:15:22.209 INFO:teuthology.orchestra.run.trial043.stdout: "mds_export_ephemeral_random_max": "0.010000", 2026-02-20T11:15:22.210 INFO:teuthology.orchestra.run.trial043.stdout: "mds_extraordinary_events_dump_interval": "0", 2026-02-20T11:15:22.210 INFO:teuthology.orchestra.run.trial043.stdout: "mds_file_blockdiff_max_concurrent_object_scans": "16", 2026-02-20T11:15:22.210 INFO:teuthology.orchestra.run.trial043.stdout: "mds_forward_all_requests_to_auth": "false", 2026-02-20T11:15:22.210 INFO:teuthology.orchestra.run.trial043.stdout: "mds_freeze_tree_timeout": "30.000000", 2026-02-20T11:15:22.210 INFO:teuthology.orchestra.run.trial043.stdout: "mds_fscrypt_last_block_max_size": "4096", 2026-02-20T11:15:22.210 INFO:teuthology.orchestra.run.trial043.stdout: "mds_go_bad_corrupt_dentry": "true", 2026-02-20T11:15:22.210 INFO:teuthology.orchestra.run.trial043.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2026-02-20T11:15:22.210 INFO:teuthology.orchestra.run.trial043.stdout: "mds_health_cache_threshold": "1.500000", 2026-02-20T11:15:22.210 INFO:teuthology.orchestra.run.trial043.stdout: "mds_health_summarize_threshold": "10", 2026-02-20T11:15:22.210 INFO:teuthology.orchestra.run.trial043.stdout: "mds_heartbeat_grace": "15.000000", 2026-02-20T11:15:22.210 INFO:teuthology.orchestra.run.trial043.stdout: "mds_heartbeat_reset_grace": "1000", 2026-02-20T11:15:22.210 INFO:teuthology.orchestra.run.trial043.stdout: "mds_inject_health_dummy": "false", 2026-02-20T11:15:22.210 INFO:teuthology.orchestra.run.trial043.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2026-02-20T11:15:22.210 INFO:teuthology.orchestra.run.trial043.stdout: "mds_inject_migrator_session_race": "false", 2026-02-20T11:15:22.210 INFO:teuthology.orchestra.run.trial043.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2026-02-20T11:15:22.210 INFO:teuthology.orchestra.run.trial043.stdout: "mds_inject_skip_replaying_inotable": "false", 2026-02-20T11:15:22.210 INFO:teuthology.orchestra.run.trial043.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2026-02-20T11:15:22.210 INFO:teuthology.orchestra.run.trial043.stdout: "mds_join_fs": "", 2026-02-20T11:15:22.210 INFO:teuthology.orchestra.run.trial043.stdout: "mds_journal_format": "1", 2026-02-20T11:15:22.210 INFO:teuthology.orchestra.run.trial043.stdout: "mds_kill_after_journal_logs_flushed": "false", 2026-02-20T11:15:22.210 INFO:teuthology.orchestra.run.trial043.stdout: "mds_kill_create_at": "0", 2026-02-20T11:15:22.210 INFO:teuthology.orchestra.run.trial043.stdout: "mds_kill_dirfrag_at": "0", 2026-02-20T11:15:22.210 INFO:teuthology.orchestra.run.trial043.stdout: "mds_kill_export_at": "0", 2026-02-20T11:15:22.210 INFO:teuthology.orchestra.run.trial043.stdout: "mds_kill_import_at": "0", 2026-02-20T11:15:22.210 INFO:teuthology.orchestra.run.trial043.stdout: "mds_kill_journal_at": "0", 2026-02-20T11:15:22.210 INFO:teuthology.orchestra.run.trial043.stdout: "mds_kill_journal_expire_at": "0", 2026-02-20T11:15:22.210 INFO:teuthology.orchestra.run.trial043.stdout: "mds_kill_journal_replay_at": "0", 2026-02-20T11:15:22.210 INFO:teuthology.orchestra.run.trial043.stdout: "mds_kill_link_at": "0", 2026-02-20T11:15:22.210 INFO:teuthology.orchestra.run.trial043.stdout: "mds_kill_mdstable_at": "0", 2026-02-20T11:15:22.210 INFO:teuthology.orchestra.run.trial043.stdout: "mds_kill_openc_at": "0", 2026-02-20T11:15:22.210 INFO:teuthology.orchestra.run.trial043.stdout: "mds_kill_rename_at": "0", 2026-02-20T11:15:22.210 INFO:teuthology.orchestra.run.trial043.stdout: "mds_kill_shutdown_at": "0", 2026-02-20T11:15:22.210 INFO:teuthology.orchestra.run.trial043.stdout: "mds_log_event_large_threshold": "524288", 2026-02-20T11:15:22.211 INFO:teuthology.orchestra.run.trial043.stdout: "mds_log_events_per_segment": "1024", 2026-02-20T11:15:22.211 INFO:teuthology.orchestra.run.trial043.stdout: "mds_log_max_events": "-1", 2026-02-20T11:15:22.211 INFO:teuthology.orchestra.run.trial043.stdout: "mds_log_max_segments": "128", 2026-02-20T11:15:22.211 INFO:teuthology.orchestra.run.trial043.stdout: "mds_log_minor_segments_per_major_segment": "16", 2026-02-20T11:15:22.211 INFO:teuthology.orchestra.run.trial043.stdout: "mds_log_pause": "false", 2026-02-20T11:15:22.211 INFO:teuthology.orchestra.run.trial043.stdout: "mds_log_segment_size": "0", 2026-02-20T11:15:22.211 INFO:teuthology.orchestra.run.trial043.stdout: "mds_log_skip_corrupt_events": "false", 2026-02-20T11:15:22.211 INFO:teuthology.orchestra.run.trial043.stdout: "mds_log_skip_unbounded_events": "false", 2026-02-20T11:15:22.211 INFO:teuthology.orchestra.run.trial043.stdout: "mds_log_trim_decay_rate": "1.000000", 2026-02-20T11:15:22.211 INFO:teuthology.orchestra.run.trial043.stdout: "mds_log_trim_threshold": "128", 2026-02-20T11:15:22.211 INFO:teuthology.orchestra.run.trial043.stdout: "mds_log_trim_upkeep_interval": "1000", 2026-02-20T11:15:22.211 INFO:teuthology.orchestra.run.trial043.stdout: "mds_log_warn_factor": "2.000000", 2026-02-20T11:15:22.211 INFO:teuthology.orchestra.run.trial043.stdout: "mds_max_caps_per_client": "1048576", 2026-02-20T11:15:22.211 INFO:teuthology.orchestra.run.trial043.stdout: "mds_max_completed_flushes": "100000", 2026-02-20T11:15:22.211 INFO:teuthology.orchestra.run.trial043.stdout: "mds_max_completed_requests": "100000", 2026-02-20T11:15:22.211 INFO:teuthology.orchestra.run.trial043.stdout: "mds_max_export_size": "20971520", 2026-02-20T11:15:22.211 INFO:teuthology.orchestra.run.trial043.stdout: "mds_max_file_recover": "32", 2026-02-20T11:15:22.211 INFO:teuthology.orchestra.run.trial043.stdout: "mds_max_purge_files": "64", 2026-02-20T11:15:22.211 INFO:teuthology.orchestra.run.trial043.stdout: "mds_max_purge_ops": "8192", 2026-02-20T11:15:22.211 INFO:teuthology.orchestra.run.trial043.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2026-02-20T11:15:22.211 INFO:teuthology.orchestra.run.trial043.stdout: "mds_max_scrub_ops_in_progress": "5", 2026-02-20T11:15:22.211 INFO:teuthology.orchestra.run.trial043.stdout: "mds_max_snaps_per_dir": "100", 2026-02-20T11:15:22.211 INFO:teuthology.orchestra.run.trial043.stdout: "mds_metrics_update_interval": "2", 2026-02-20T11:15:22.211 INFO:teuthology.orchestra.run.trial043.stdout: "mds_min_caps_per_client": "100", 2026-02-20T11:15:22.211 INFO:teuthology.orchestra.run.trial043.stdout: "mds_min_caps_working_set": "10000", 2026-02-20T11:15:22.211 INFO:teuthology.orchestra.run.trial043.stdout: "mds_mon_shutdown_timeout": "5.000000", 2026-02-20T11:15:22.211 INFO:teuthology.orchestra.run.trial043.stdout: "mds_numa_node": "-1", 2026-02-20T11:15:22.211 INFO:teuthology.orchestra.run.trial043.stdout: "mds_oft_prefetch_dirfrags": "false", 2026-02-20T11:15:22.211 INFO:teuthology.orchestra.run.trial043.stdout: "mds_op_complaint_time": "30.000000", 2026-02-20T11:15:22.211 INFO:teuthology.orchestra.run.trial043.stdout: "mds_op_history_duration": "600", 2026-02-20T11:15:22.211 INFO:teuthology.orchestra.run.trial043.stdout: "mds_op_history_size": "20", 2026-02-20T11:15:22.211 INFO:teuthology.orchestra.run.trial043.stdout: "mds_op_history_slow_op_size": "20", 2026-02-20T11:15:22.211 INFO:teuthology.orchestra.run.trial043.stdout: "mds_op_history_slow_op_threshold": "10", 2026-02-20T11:15:22.212 INFO:teuthology.orchestra.run.trial043.stdout: "mds_op_log_threshold": "5", 2026-02-20T11:15:22.212 INFO:teuthology.orchestra.run.trial043.stdout: "mds_ping_grace": "15", 2026-02-20T11:15:22.212 INFO:teuthology.orchestra.run.trial043.stdout: "mds_ping_interval": "5", 2026-02-20T11:15:22.212 INFO:teuthology.orchestra.run.trial043.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2026-02-20T11:15:22.212 INFO:teuthology.orchestra.run.trial043.stdout: "mds_recall_global_max_decay_threshold": "131072", 2026-02-20T11:15:22.212 INFO:teuthology.orchestra.run.trial043.stdout: "mds_recall_max_caps": "30000", 2026-02-20T11:15:22.212 INFO:teuthology.orchestra.run.trial043.stdout: "mds_recall_max_decay_rate": "1.500000", 2026-02-20T11:15:22.212 INFO:teuthology.orchestra.run.trial043.stdout: "mds_recall_max_decay_threshold": "131072", 2026-02-20T11:15:22.212 INFO:teuthology.orchestra.run.trial043.stdout: "mds_recall_warning_decay_rate": "60.000000", 2026-02-20T11:15:22.212 INFO:teuthology.orchestra.run.trial043.stdout: "mds_recall_warning_threshold": "262144", 2026-02-20T11:15:22.212 INFO:teuthology.orchestra.run.trial043.stdout: "mds_reconnect_timeout": "45.000000", 2026-02-20T11:15:22.212 INFO:teuthology.orchestra.run.trial043.stdout: "mds_replay_interval": "1.000000", 2026-02-20T11:15:22.212 INFO:teuthology.orchestra.run.trial043.stdout: "mds_replay_unsafe_with_closed_session": "false", 2026-02-20T11:15:22.212 INFO:teuthology.orchestra.run.trial043.stdout: "mds_request_load_average_decay_rate": "60.000000", 2026-02-20T11:15:22.212 INFO:teuthology.orchestra.run.trial043.stdout: "mds_root_ino_gid": "0", 2026-02-20T11:15:22.212 INFO:teuthology.orchestra.run.trial043.stdout: "mds_root_ino_uid": "0", 2026-02-20T11:15:22.212 INFO:teuthology.orchestra.run.trial043.stdout: "mds_scatter_nudge_interval": "5.000000", 2026-02-20T11:15:22.212 INFO:teuthology.orchestra.run.trial043.stdout: "mds_scrub_stats_review_period": "1", 2026-02-20T11:15:22.212 INFO:teuthology.orchestra.run.trial043.stdout: "mds_server_dispatch_client_request_delay": "0", 2026-02-20T11:15:22.212 INFO:teuthology.orchestra.run.trial043.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2026-02-20T11:15:22.212 INFO:teuthology.orchestra.run.trial043.stdout: "mds_session_blocklist_on_evict": "true", 2026-02-20T11:15:22.212 INFO:teuthology.orchestra.run.trial043.stdout: "mds_session_blocklist_on_timeout": "true", 2026-02-20T11:15:22.212 INFO:teuthology.orchestra.run.trial043.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2026-02-20T11:15:22.212 INFO:teuthology.orchestra.run.trial043.stdout: "mds_session_cache_liveness_magnitude": "10", 2026-02-20T11:15:22.212 INFO:teuthology.orchestra.run.trial043.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2026-02-20T11:15:22.212 INFO:teuthology.orchestra.run.trial043.stdout: "mds_session_cap_acquisition_throttle": "100000", 2026-02-20T11:15:22.212 INFO:teuthology.orchestra.run.trial043.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2026-02-20T11:15:22.212 INFO:teuthology.orchestra.run.trial043.stdout: "mds_session_metadata_threshold": "16777216", 2026-02-20T11:15:22.212 INFO:teuthology.orchestra.run.trial043.stdout: "mds_sessionmap_keys_per_op": "1024", 2026-02-20T11:15:22.212 INFO:teuthology.orchestra.run.trial043.stdout: "mds_shutdown_check": "0", 2026-02-20T11:15:22.213 INFO:teuthology.orchestra.run.trial043.stdout: "mds_skip_ino": "0", 2026-02-20T11:15:22.213 INFO:teuthology.orchestra.run.trial043.stdout: "mds_sleep_rank_change": "0.000000", 2026-02-20T11:15:22.213 INFO:teuthology.orchestra.run.trial043.stdout: "mds_snap_max_uid": "4294967294", 2026-02-20T11:15:22.213 INFO:teuthology.orchestra.run.trial043.stdout: "mds_snap_min_uid": "0", 2026-02-20T11:15:22.213 INFO:teuthology.orchestra.run.trial043.stdout: "mds_snap_rstat": "false", 2026-02-20T11:15:22.213 INFO:teuthology.orchestra.run.trial043.stdout: "mds_standby_replay_damaged": "false", 2026-02-20T11:15:22.213 INFO:teuthology.orchestra.run.trial043.stdout: "mds_symlink_recovery": "true", 2026-02-20T11:15:22.213 INFO:teuthology.orchestra.run.trial043.stdout: "mds_task_status_update_interval": "2.000000", 2026-02-20T11:15:22.213 INFO:teuthology.orchestra.run.trial043.stdout: "mds_thrash_exports": "0", 2026-02-20T11:15:22.213 INFO:teuthology.orchestra.run.trial043.stdout: "mds_thrash_fragments": "0", 2026-02-20T11:15:22.213 INFO:teuthology.orchestra.run.trial043.stdout: "mds_tick_interval": "5.000000", 2026-02-20T11:15:22.213 INFO:teuthology.orchestra.run.trial043.stdout: "mds_use_global_snaprealm_seq_for_subvol": "true", 2026-02-20T11:15:22.213 INFO:teuthology.orchestra.run.trial043.stdout: "mds_valgrind_exit": "false", 2026-02-20T11:15:22.213 INFO:teuthology.orchestra.run.trial043.stdout: "mds_verify_backtrace": "1", 2026-02-20T11:15:22.213 INFO:teuthology.orchestra.run.trial043.stdout: "mds_verify_scatter": "false", 2026-02-20T11:15:22.213 INFO:teuthology.orchestra.run.trial043.stdout: "mds_wipe_ino_prealloc": "false", 2026-02-20T11:15:22.213 INFO:teuthology.orchestra.run.trial043.stdout: "mds_wipe_sessions": "false", 2026-02-20T11:15:22.213 INFO:teuthology.orchestra.run.trial043.stdout: "mempool_debug": "false", 2026-02-20T11:15:22.213 INFO:teuthology.orchestra.run.trial043.stdout: "memstore_debug_omit_block_device_write": "false", 2026-02-20T11:15:22.213 INFO:teuthology.orchestra.run.trial043.stdout: "memstore_device_bytes": "1073741824", 2026-02-20T11:15:22.213 INFO:teuthology.orchestra.run.trial043.stdout: "memstore_page_set": "false", 2026-02-20T11:15:22.213 INFO:teuthology.orchestra.run.trial043.stdout: "memstore_page_size": "65536", 2026-02-20T11:15:22.214 INFO:teuthology.orchestra.run.trial043.stdout: "mgr_client_bytes": "134217728", 2026-02-20T11:15:22.214 INFO:teuthology.orchestra.run.trial043.stdout: "mgr_client_messages": "512", 2026-02-20T11:15:22.214 INFO:teuthology.orchestra.run.trial043.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2026-02-20T11:15:22.214 INFO:teuthology.orchestra.run.trial043.stdout: "mgr_connect_retry_interval": "1.000000", 2026-02-20T11:15:22.214 INFO:teuthology.orchestra.run.trial043.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2026-02-20T11:15:22.214 INFO:teuthology.orchestra.run.trial043.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2026-02-20T11:15:22.214 INFO:teuthology.orchestra.run.trial043.stdout: "mgr_disabled_modules": "", 2026-02-20T11:15:22.214 INFO:teuthology.orchestra.run.trial043.stdout: "mgr_enable_op_tracker": "true", 2026-02-20T11:15:22.214 INFO:teuthology.orchestra.run.trial043.stdout: "mgr_initial_modules": "iostat nfs", 2026-02-20T11:15:22.214 INFO:teuthology.orchestra.run.trial043.stdout: "mgr_max_pg_creating": "1024", 2026-02-20T11:15:22.214 INFO:teuthology.orchestra.run.trial043.stdout: "mgr_max_pg_num_change": "128", 2026-02-20T11:15:22.214 INFO:teuthology.orchestra.run.trial043.stdout: "mgr_mds_bytes": "134217728", 2026-02-20T11:15:22.214 INFO:teuthology.orchestra.run.trial043.stdout: "mgr_mds_messages": "128", 2026-02-20T11:15:22.214 INFO:teuthology.orchestra.run.trial043.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2026-02-20T11:15:22.214 INFO:teuthology.orchestra.run.trial043.stdout: "mgr_mon_bytes": "134217728", 2026-02-20T11:15:22.214 INFO:teuthology.orchestra.run.trial043.stdout: "mgr_mon_messages": "128", 2026-02-20T11:15:22.214 INFO:teuthology.orchestra.run.trial043.stdout: "mgr_num_op_tracker_shard": "32", 2026-02-20T11:15:22.215 INFO:teuthology.orchestra.run.trial043.stdout: "mgr_op_complaint_time": "30.000000", 2026-02-20T11:15:22.215 INFO:teuthology.orchestra.run.trial043.stdout: "mgr_op_history_duration": "600", 2026-02-20T11:15:22.215 INFO:teuthology.orchestra.run.trial043.stdout: "mgr_op_history_size": "20", 2026-02-20T11:15:22.215 INFO:teuthology.orchestra.run.trial043.stdout: "mgr_op_history_slow_op_size": "20", 2026-02-20T11:15:22.215 INFO:teuthology.orchestra.run.trial043.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2026-02-20T11:15:22.215 INFO:teuthology.orchestra.run.trial043.stdout: "mgr_op_log_threshold": "5", 2026-02-20T11:15:22.215 INFO:teuthology.orchestra.run.trial043.stdout: "mgr_osd_bytes": "536870912", 2026-02-20T11:15:22.215 INFO:teuthology.orchestra.run.trial043.stdout: "mgr_osd_messages": "8192", 2026-02-20T11:15:22.215 INFO:teuthology.orchestra.run.trial043.stdout: "mgr_pool": "true", 2026-02-20T11:15:22.215 INFO:teuthology.orchestra.run.trial043.stdout: "mgr_service_beacon_grace": "60.000000", 2026-02-20T11:15:22.215 INFO:teuthology.orchestra.run.trial043.stdout: "mgr_standby_modules": "true", 2026-02-20T11:15:22.215 INFO:teuthology.orchestra.run.trial043.stdout: "mgr_stats_period": "5", 2026-02-20T11:15:22.215 INFO:teuthology.orchestra.run.trial043.stdout: "mgr_stats_threshold": "5", 2026-02-20T11:15:22.215 INFO:teuthology.orchestra.run.trial043.stdout: "mgr_tick_period": "2", 2026-02-20T11:15:22.215 INFO:teuthology.orchestra.run.trial043.stdout: "mgr_ttl_cache_expire_seconds": "0", 2026-02-20T11:15:22.215 INFO:teuthology.orchestra.run.trial043.stdout: "mon_accept_timeout_factor": "2.000000", 2026-02-20T11:15:22.215 INFO:teuthology.orchestra.run.trial043.stdout: "mon_allow_pool_delete": "true", 2026-02-20T11:15:22.215 INFO:teuthology.orchestra.run.trial043.stdout: "mon_allow_pool_size_one": "true", 2026-02-20T11:15:22.215 INFO:teuthology.orchestra.run.trial043.stdout: "mon_auth_validate_all_caps": "true", 2026-02-20T11:15:22.215 INFO:teuthology.orchestra.run.trial043.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2026-02-20T11:15:22.215 INFO:teuthology.orchestra.run.trial043.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2026-02-20T11:15:22.215 INFO:teuthology.orchestra.run.trial043.stdout: "mon_client_bytes": "104857600", 2026-02-20T11:15:22.215 INFO:teuthology.orchestra.run.trial043.stdout: "mon_client_directed_command_retry": "2", 2026-02-20T11:15:22.216 INFO:teuthology.orchestra.run.trial043.stdout: "mon_client_hunt_interval": "3.000000", 2026-02-20T11:15:22.216 INFO:teuthology.orchestra.run.trial043.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2026-02-20T11:15:22.216 INFO:teuthology.orchestra.run.trial043.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2026-02-20T11:15:22.216 INFO:teuthology.orchestra.run.trial043.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2026-02-20T11:15:22.216 INFO:teuthology.orchestra.run.trial043.stdout: "mon_client_hunt_on_resend": "true", 2026-02-20T11:15:22.216 INFO:teuthology.orchestra.run.trial043.stdout: "mon_client_hunt_parallel": "3", 2026-02-20T11:15:22.216 INFO:teuthology.orchestra.run.trial043.stdout: "mon_client_log_interval": "1.000000", 2026-02-20T11:15:22.216 INFO:teuthology.orchestra.run.trial043.stdout: "mon_client_max_log_entries_per_message": "1000", 2026-02-20T11:15:22.216 INFO:teuthology.orchestra.run.trial043.stdout: "mon_client_ping_interval": "10.000000", 2026-02-20T11:15:22.216 INFO:teuthology.orchestra.run.trial043.stdout: "mon_client_ping_timeout": "30.000000", 2026-02-20T11:15:22.216 INFO:teuthology.orchestra.run.trial043.stdout: "mon_client_target_rank": "-1", 2026-02-20T11:15:22.216 INFO:teuthology.orchestra.run.trial043.stdout: "mon_clock_drift_allowed": "1.000000", 2026-02-20T11:15:22.216 INFO:teuthology.orchestra.run.trial043.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2026-02-20T11:15:22.216 INFO:teuthology.orchestra.run.trial043.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2026-02-20T11:15:22.216 INFO:teuthology.orchestra.run.trial043.stdout: "mon_cluster_log_level": "debug", 2026-02-20T11:15:22.216 INFO:teuthology.orchestra.run.trial043.stdout: "mon_cluster_log_to_file": "true", 2026-02-20T11:15:22.216 INFO:teuthology.orchestra.run.trial043.stdout: "mon_cluster_log_to_graylog": "false", 2026-02-20T11:15:22.216 INFO:teuthology.orchestra.run.trial043.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2026-02-20T11:15:22.216 INFO:teuthology.orchestra.run.trial043.stdout: "mon_cluster_log_to_graylog_port": "12201", 2026-02-20T11:15:22.216 INFO:teuthology.orchestra.run.trial043.stdout: "mon_cluster_log_to_journald": "false", 2026-02-20T11:15:22.216 INFO:teuthology.orchestra.run.trial043.stdout: "mon_cluster_log_to_stderr": "false", 2026-02-20T11:15:22.216 INFO:teuthology.orchestra.run.trial043.stdout: "mon_cluster_log_to_syslog": "default=false", 2026-02-20T11:15:22.216 INFO:teuthology.orchestra.run.trial043.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2026-02-20T11:15:22.216 INFO:teuthology.orchestra.run.trial043.stdout: "mon_compact_on_bootstrap": "false", 2026-02-20T11:15:22.216 INFO:teuthology.orchestra.run.trial043.stdout: "mon_compact_on_start": "false", 2026-02-20T11:15:22.216 INFO:teuthology.orchestra.run.trial043.stdout: "mon_compact_on_trim": "true", 2026-02-20T11:15:22.216 INFO:teuthology.orchestra.run.trial043.stdout: "mon_con_tracker_persist_interval": "10", 2026-02-20T11:15:22.217 INFO:teuthology.orchestra.run.trial043.stdout: "mon_con_tracker_score_halflife": "43200", 2026-02-20T11:15:22.217 INFO:teuthology.orchestra.run.trial043.stdout: "mon_config_key_max_entry_size": "65536", 2026-02-20T11:15:22.217 INFO:teuthology.orchestra.run.trial043.stdout: "mon_cpu_threads": "4", 2026-02-20T11:15:22.217 INFO:teuthology.orchestra.run.trial043.stdout: "mon_crush_min_required_version": "hammer", 2026-02-20T11:15:22.217 INFO:teuthology.orchestra.run.trial043.stdout: "mon_daemon_bytes": "419430400", 2026-02-20T11:15:22.217 INFO:teuthology.orchestra.run.trial043.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2026-02-20T11:15:22.217 INFO:teuthology.orchestra.run.trial043.stdout: "mon_data_avail_crit": "5", 2026-02-20T11:15:22.217 INFO:teuthology.orchestra.run.trial043.stdout: "mon_data_avail_warn": "5", 2026-02-20T11:15:22.217 INFO:teuthology.orchestra.run.trial043.stdout: "mon_data_size_warn": "16106127360", 2026-02-20T11:15:22.217 INFO:teuthology.orchestra.run.trial043.stdout: "mon_debug_block_osdmap_trim": "false", 2026-02-20T11:15:22.217 INFO:teuthology.orchestra.run.trial043.stdout: "mon_debug_deprecated_as_obsolete": "false", 2026-02-20T11:15:22.217 INFO:teuthology.orchestra.run.trial043.stdout: "mon_debug_dump_json": "false", 2026-02-20T11:15:22.217 INFO:teuthology.orchestra.run.trial043.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2026-02-20T11:15:22.217 INFO:teuthology.orchestra.run.trial043.stdout: "mon_debug_dump_transactions": "false", 2026-02-20T11:15:22.217 INFO:teuthology.orchestra.run.trial043.stdout: "mon_debug_extra_checks": "false", 2026-02-20T11:15:22.217 INFO:teuthology.orchestra.run.trial043.stdout: "mon_debug_no_initial_persistent_features": "false", 2026-02-20T11:15:22.217 INFO:teuthology.orchestra.run.trial043.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2026-02-20T11:15:22.217 INFO:teuthology.orchestra.run.trial043.stdout: "mon_debug_no_require_squid": "false", 2026-02-20T11:15:22.217 INFO:teuthology.orchestra.run.trial043.stdout: "mon_debug_no_require_tentacle": "false", 2026-02-20T11:15:22.217 INFO:teuthology.orchestra.run.trial043.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2026-02-20T11:15:22.217 INFO:teuthology.orchestra.run.trial043.stdout: "mon_delta_reset_interval": "10.000000", 2026-02-20T11:15:22.217 INFO:teuthology.orchestra.run.trial043.stdout: "mon_dns_srv_name": "ceph-mon", 2026-02-20T11:15:22.217 INFO:teuthology.orchestra.run.trial043.stdout: "mon_down_added_grace": "180", 2026-02-20T11:15:22.217 INFO:teuthology.orchestra.run.trial043.stdout: "mon_down_mkfs_grace": "120", 2026-02-20T11:15:22.217 INFO:teuthology.orchestra.run.trial043.stdout: "mon_down_uptime_grace": "60", 2026-02-20T11:15:22.218 INFO:teuthology.orchestra.run.trial043.stdout: "mon_election_default_strategy": "1", 2026-02-20T11:15:22.218 INFO:teuthology.orchestra.run.trial043.stdout: "mon_election_timeout": "5.000000", 2026-02-20T11:15:22.218 INFO:teuthology.orchestra.run.trial043.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2026-02-20T11:15:22.218 INFO:teuthology.orchestra.run.trial043.stdout: "mon_elector_ping_divisor": "2", 2026-02-20T11:15:22.218 INFO:teuthology.orchestra.run.trial043.stdout: "mon_elector_ping_timeout": "2.000000", 2026-02-20T11:15:22.218 INFO:teuthology.orchestra.run.trial043.stdout: "mon_enable_op_tracker": "true", 2026-02-20T11:15:22.218 INFO:teuthology.orchestra.run.trial043.stdout: "mon_fake_pool_delete": "false", 2026-02-20T11:15:22.218 INFO:teuthology.orchestra.run.trial043.stdout: "mon_force_quorum_join": "false", 2026-02-20T11:15:22.218 INFO:teuthology.orchestra.run.trial043.stdout: "mon_fsmap_prune_threshold": "300", 2026-02-20T11:15:22.218 INFO:teuthology.orchestra.run.trial043.stdout: "mon_globalid_prealloc": "10000", 2026-02-20T11:15:22.218 INFO:teuthology.orchestra.run.trial043.stdout: "mon_health_detail_to_clog": "false", 2026-02-20T11:15:22.218 INFO:teuthology.orchestra.run.trial043.stdout: "mon_health_log_update_period": "5", 2026-02-20T11:15:22.218 INFO:teuthology.orchestra.run.trial043.stdout: "mon_health_max_detail": "50", 2026-02-20T11:15:22.218 INFO:teuthology.orchestra.run.trial043.stdout: "mon_health_to_clog": "true", 2026-02-20T11:15:22.218 INFO:teuthology.orchestra.run.trial043.stdout: "mon_health_to_clog_interval": "600", 2026-02-20T11:15:22.218 INFO:teuthology.orchestra.run.trial043.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2026-02-20T11:15:22.218 INFO:teuthology.orchestra.run.trial043.stdout: "mon_host": "10.20.193.43", 2026-02-20T11:15:22.218 INFO:teuthology.orchestra.run.trial043.stdout: "mon_host_override": "", 2026-02-20T11:15:22.218 INFO:teuthology.orchestra.run.trial043.stdout: "mon_initial_members": "", 2026-02-20T11:15:22.218 INFO:teuthology.orchestra.run.trial043.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2026-02-20T11:15:22.218 INFO:teuthology.orchestra.run.trial043.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2026-02-20T11:15:22.218 INFO:teuthology.orchestra.run.trial043.stdout: "mon_inject_transaction_delay_max": "10.000000", 2026-02-20T11:15:22.218 INFO:teuthology.orchestra.run.trial043.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2026-02-20T11:15:22.218 INFO:teuthology.orchestra.run.trial043.stdout: "mon_keyvaluedb": "rocksdb", 2026-02-20T11:15:22.218 INFO:teuthology.orchestra.run.trial043.stdout: "mon_lease": "5.000000", 2026-02-20T11:15:22.218 INFO:teuthology.orchestra.run.trial043.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2026-02-20T11:15:22.218 INFO:teuthology.orchestra.run.trial043.stdout: "mon_lease_renew_interval_factor": "0.600000", 2026-02-20T11:15:22.218 INFO:teuthology.orchestra.run.trial043.stdout: "mon_log_full_interval": "50", 2026-02-20T11:15:22.218 INFO:teuthology.orchestra.run.trial043.stdout: "mon_log_max": "10000", 2026-02-20T11:15:22.218 INFO:teuthology.orchestra.run.trial043.stdout: "mon_log_max_summary": "50", 2026-02-20T11:15:22.218 INFO:teuthology.orchestra.run.trial043.stdout: "mon_max_log_entries_per_event": "4096", 2026-02-20T11:15:22.218 INFO:teuthology.orchestra.run.trial043.stdout: "mon_max_log_epochs": "500", 2026-02-20T11:15:22.218 INFO:teuthology.orchestra.run.trial043.stdout: "mon_max_mdsmap_epochs": "500", 2026-02-20T11:15:22.218 INFO:teuthology.orchestra.run.trial043.stdout: "mon_max_mgrmap_epochs": "500", 2026-02-20T11:15:22.218 INFO:teuthology.orchestra.run.trial043.stdout: "mon_max_nvmeof_epochs": "500", 2026-02-20T11:15:22.218 INFO:teuthology.orchestra.run.trial043.stdout: "mon_max_osd": "10000", 2026-02-20T11:15:22.219 INFO:teuthology.orchestra.run.trial043.stdout: "mon_max_pg_per_osd": "10000", 2026-02-20T11:15:22.219 INFO:teuthology.orchestra.run.trial043.stdout: "mon_max_pool_pg_num": "65536", 2026-02-20T11:15:22.219 INFO:teuthology.orchestra.run.trial043.stdout: "mon_max_snap_prune_per_epoch": "100", 2026-02-20T11:15:22.219 INFO:teuthology.orchestra.run.trial043.stdout: "mon_mds_blocklist_interval": "86400.000000", 2026-02-20T11:15:22.219 INFO:teuthology.orchestra.run.trial043.stdout: "mon_mds_force_trim_to": "0", 2026-02-20T11:15:22.219 INFO:teuthology.orchestra.run.trial043.stdout: "mon_mds_skip_sanity": "false", 2026-02-20T11:15:22.219 INFO:teuthology.orchestra.run.trial043.stdout: "mon_memory_autotune": "true", 2026-02-20T11:15:22.219 INFO:teuthology.orchestra.run.trial043.stdout: "mon_memory_target": "2147483648", 2026-02-20T11:15:22.219 INFO:teuthology.orchestra.run.trial043.stdout: "mon_mgr_beacon_grace": "30", 2026-02-20T11:15:22.219 INFO:teuthology.orchestra.run.trial043.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2026-02-20T11:15:22.219 INFO:teuthology.orchestra.run.trial043.stdout: "mon_mgr_digest_period": "5", 2026-02-20T11:15:22.219 INFO:teuthology.orchestra.run.trial043.stdout: "mon_mgr_inactive_grace": "60", 2026-02-20T11:15:22.219 INFO:teuthology.orchestra.run.trial043.stdout: "mon_mgr_mkfs_grace": "240", 2026-02-20T11:15:22.219 INFO:teuthology.orchestra.run.trial043.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2026-02-20T11:15:22.219 INFO:teuthology.orchestra.run.trial043.stdout: "mon_min_osdmap_epochs": "50", 2026-02-20T11:15:22.219 INFO:teuthology.orchestra.run.trial043.stdout: "mon_netsplit_grace_period": "9", 2026-02-20T11:15:22.219 INFO:teuthology.orchestra.run.trial043.stdout: "mon_nvmeofgw_beacon_grace": "7", 2026-02-20T11:15:22.219 INFO:teuthology.orchestra.run.trial043.stdout: "mon_nvmeofgw_beacons_till_ack": "15", 2026-02-20T11:15:22.219 INFO:teuthology.orchestra.run.trial043.stdout: "mon_nvmeofgw_delete_grace": "900", 2026-02-20T11:15:22.219 INFO:teuthology.orchestra.run.trial043.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2026-02-20T11:15:22.219 INFO:teuthology.orchestra.run.trial043.stdout: "mon_nvmeofgw_skip_failovers_interval": "16", 2026-02-20T11:15:22.219 INFO:teuthology.orchestra.run.trial043.stdout: "mon_nvmeofgw_wrong_map_ignore_sec": "15", 2026-02-20T11:15:22.219 INFO:teuthology.orchestra.run.trial043.stdout: "mon_op_complaint_time": "30", 2026-02-20T11:15:22.219 INFO:teuthology.orchestra.run.trial043.stdout: "mon_op_history_duration": "600", 2026-02-20T11:15:22.219 INFO:teuthology.orchestra.run.trial043.stdout: "mon_op_history_size": "20", 2026-02-20T11:15:22.219 INFO:teuthology.orchestra.run.trial043.stdout: "mon_op_history_slow_op_size": "20", 2026-02-20T11:15:22.219 INFO:teuthology.orchestra.run.trial043.stdout: "mon_op_history_slow_op_threshold": "10", 2026-02-20T11:15:22.219 INFO:teuthology.orchestra.run.trial043.stdout: "mon_op_log_threshold": "5", 2026-02-20T11:15:22.219 INFO:teuthology.orchestra.run.trial043.stdout: "mon_osd_adjust_down_out_interval": "true", 2026-02-20T11:15:22.219 INFO:teuthology.orchestra.run.trial043.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2026-02-20T11:15:22.219 INFO:teuthology.orchestra.run.trial043.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2026-02-20T11:15:22.219 INFO:teuthology.orchestra.run.trial043.stdout: "mon_osd_auto_mark_in": "false", 2026-02-20T11:15:22.219 INFO:teuthology.orchestra.run.trial043.stdout: "mon_osd_auto_mark_new_in": "true", 2026-02-20T11:15:22.219 INFO:teuthology.orchestra.run.trial043.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2026-02-20T11:15:22.219 INFO:teuthology.orchestra.run.trial043.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2026-02-20T11:15:22.219 INFO:teuthology.orchestra.run.trial043.stdout: "mon_osd_cache_size": "500", 2026-02-20T11:15:22.219 INFO:teuthology.orchestra.run.trial043.stdout: "mon_osd_cache_size_min": "134217728", 2026-02-20T11:15:22.220 INFO:teuthology.orchestra.run.trial043.stdout: "mon_osd_crush_smoke_test": "true", 2026-02-20T11:15:22.220 INFO:teuthology.orchestra.run.trial043.stdout: "mon_osd_destroyed_out_interval": "600", 2026-02-20T11:15:22.220 INFO:teuthology.orchestra.run.trial043.stdout: "mon_osd_down_out_interval": "600", 2026-02-20T11:15:22.220 INFO:teuthology.orchestra.run.trial043.stdout: "mon_osd_down_out_subtree_limit": "rack", 2026-02-20T11:15:22.220 INFO:teuthology.orchestra.run.trial043.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2026-02-20T11:15:22.220 INFO:teuthology.orchestra.run.trial043.stdout: "mon_osd_force_trim_to": "0", 2026-02-20T11:15:22.220 INFO:teuthology.orchestra.run.trial043.stdout: "mon_osd_full_ratio": "0.950000", 2026-02-20T11:15:22.220 INFO:teuthology.orchestra.run.trial043.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2026-02-20T11:15:22.220 INFO:teuthology.orchestra.run.trial043.stdout: "mon_osd_laggy_halflife": "3600", 2026-02-20T11:15:22.220 INFO:teuthology.orchestra.run.trial043.stdout: "mon_osd_laggy_max_interval": "300", 2026-02-20T11:15:22.220 INFO:teuthology.orchestra.run.trial043.stdout: "mon_osd_laggy_weight": "0.300000", 2026-02-20T11:15:22.220 INFO:teuthology.orchestra.run.trial043.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2026-02-20T11:15:22.220 INFO:teuthology.orchestra.run.trial043.stdout: "mon_osd_max_initial_pgs": "1024", 2026-02-20T11:15:22.220 INFO:teuthology.orchestra.run.trial043.stdout: "mon_osd_min_down_reporters": "2", 2026-02-20T11:15:22.220 INFO:teuthology.orchestra.run.trial043.stdout: "mon_osd_min_in_ratio": "0.750000", 2026-02-20T11:15:22.220 INFO:teuthology.orchestra.run.trial043.stdout: "mon_osd_min_up_ratio": "0.300000", 2026-02-20T11:15:22.220 INFO:teuthology.orchestra.run.trial043.stdout: "mon_osd_nearfull_ratio": "0.850000", 2026-02-20T11:15:22.220 INFO:teuthology.orchestra.run.trial043.stdout: "mon_osd_prime_pg_temp": "true", 2026-02-20T11:15:22.220 INFO:teuthology.orchestra.run.trial043.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2026-02-20T11:15:22.220 INFO:teuthology.orchestra.run.trial043.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2026-02-20T11:15:22.220 INFO:teuthology.orchestra.run.trial043.stdout: "mon_osd_report_timeout": "900", 2026-02-20T11:15:22.220 INFO:teuthology.orchestra.run.trial043.stdout: "mon_osd_reporter_subtree_level": "osd", 2026-02-20T11:15:22.220 INFO:teuthology.orchestra.run.trial043.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2026-02-20T11:15:22.220 INFO:teuthology.orchestra.run.trial043.stdout: "mon_osd_warn_num_repaired": "10", 2026-02-20T11:15:22.220 INFO:teuthology.orchestra.run.trial043.stdout: "mon_osd_warn_op_age": "32.000000", 2026-02-20T11:15:22.220 INFO:teuthology.orchestra.run.trial043.stdout: "mon_osdmap_full_prune_enabled": "true", 2026-02-20T11:15:22.220 INFO:teuthology.orchestra.run.trial043.stdout: "mon_osdmap_full_prune_interval": "2", 2026-02-20T11:15:22.220 INFO:teuthology.orchestra.run.trial043.stdout: "mon_osdmap_full_prune_min": "15", 2026-02-20T11:15:22.220 INFO:teuthology.orchestra.run.trial043.stdout: "mon_osdmap_full_prune_txsize": "2", 2026-02-20T11:15:22.220 INFO:teuthology.orchestra.run.trial043.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2026-02-20T11:15:22.220 INFO:teuthology.orchestra.run.trial043.stdout: "mon_pg_stuck_threshold": "60", 2026-02-20T11:15:22.220 INFO:teuthology.orchestra.run.trial043.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2026-02-20T11:15:22.220 INFO:teuthology.orchestra.run.trial043.stdout: "mon_pg_warn_min_objects": "10000", 2026-02-20T11:15:22.220 INFO:teuthology.orchestra.run.trial043.stdout: "mon_pg_warn_min_per_osd": "0", 2026-02-20T11:15:22.220 INFO:teuthology.orchestra.run.trial043.stdout: "mon_pg_warn_min_pool_objects": "1000", 2026-02-20T11:15:22.221 INFO:teuthology.orchestra.run.trial043.stdout: "mon_pool_quota_crit_threshold": "0", 2026-02-20T11:15:22.221 INFO:teuthology.orchestra.run.trial043.stdout: "mon_pool_quota_warn_threshold": "0", 2026-02-20T11:15:22.221 INFO:teuthology.orchestra.run.trial043.stdout: "mon_probe_timeout": "2.000000", 2026-02-20T11:15:22.221 INFO:teuthology.orchestra.run.trial043.stdout: "mon_reweight_max_change": "0.050000", 2026-02-20T11:15:22.221 INFO:teuthology.orchestra.run.trial043.stdout: "mon_reweight_max_osds": "4", 2026-02-20T11:15:22.221 INFO:teuthology.orchestra.run.trial043.stdout: "mon_reweight_min_bytes_per_osd": "10", 2026-02-20T11:15:22.221 INFO:teuthology.orchestra.run.trial043.stdout: "mon_reweight_min_pgs_per_osd": "4", 2026-02-20T11:15:22.221 INFO:teuthology.orchestra.run.trial043.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2026-02-20T11:15:22.221 INFO:teuthology.orchestra.run.trial043.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2026-02-20T11:15:22.221 INFO:teuthology.orchestra.run.trial043.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2026-02-20T11:15:22.221 INFO:teuthology.orchestra.run.trial043.stdout: "mon_scrub_interval": "86400", 2026-02-20T11:15:22.221 INFO:teuthology.orchestra.run.trial043.stdout: "mon_scrub_max_keys": "100", 2026-02-20T11:15:22.221 INFO:teuthology.orchestra.run.trial043.stdout: "mon_scrub_timeout": "300", 2026-02-20T11:15:22.221 INFO:teuthology.orchestra.run.trial043.stdout: "mon_session_timeout": "300", 2026-02-20T11:15:22.221 INFO:teuthology.orchestra.run.trial043.stdout: "mon_smart_report_timeout": "5", 2026-02-20T11:15:22.221 INFO:teuthology.orchestra.run.trial043.stdout: "mon_stat_smooth_intervals": "6", 2026-02-20T11:15:22.221 INFO:teuthology.orchestra.run.trial043.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2026-02-20T11:15:22.221 INFO:teuthology.orchestra.run.trial043.stdout: "mon_stretch_pool_min_size": "2", 2026-02-20T11:15:22.221 INFO:teuthology.orchestra.run.trial043.stdout: "mon_stretch_pool_size": "4", 2026-02-20T11:15:22.221 INFO:teuthology.orchestra.run.trial043.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2026-02-20T11:15:22.221 INFO:teuthology.orchestra.run.trial043.stdout: "mon_subscribe_interval": "86400.000000", 2026-02-20T11:15:22.221 INFO:teuthology.orchestra.run.trial043.stdout: "mon_sync_debug": "false", 2026-02-20T11:15:22.221 INFO:teuthology.orchestra.run.trial043.stdout: "mon_sync_max_payload_keys": "2000", 2026-02-20T11:15:22.221 INFO:teuthology.orchestra.run.trial043.stdout: "mon_sync_max_payload_size": "1048576", 2026-02-20T11:15:22.222 INFO:teuthology.orchestra.run.trial043.stdout: "mon_sync_provider_kill_at": "0", 2026-02-20T11:15:22.222 INFO:teuthology.orchestra.run.trial043.stdout: "mon_sync_requester_kill_at": "0", 2026-02-20T11:15:22.222 INFO:teuthology.orchestra.run.trial043.stdout: "mon_sync_timeout": "60.000000", 2026-02-20T11:15:22.222 INFO:teuthology.orchestra.run.trial043.stdout: "mon_target_pg_per_osd": "100", 2026-02-20T11:15:22.222 INFO:teuthology.orchestra.run.trial043.stdout: "mon_tick_interval": "5", 2026-02-20T11:15:22.222 INFO:teuthology.orchestra.run.trial043.stdout: "mon_timecheck_interval": "300.000000", 2026-02-20T11:15:22.222 INFO:teuthology.orchestra.run.trial043.stdout: "mon_timecheck_skew_interval": "30.000000", 2026-02-20T11:15:22.222 INFO:teuthology.orchestra.run.trial043.stdout: "mon_use_min_delay_socket": "false", 2026-02-20T11:15:22.222 INFO:teuthology.orchestra.run.trial043.stdout: "mon_warn_older_version_delay": "604800", 2026-02-20T11:15:22.222 INFO:teuthology.orchestra.run.trial043.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2026-02-20T11:15:22.222 INFO:teuthology.orchestra.run.trial043.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2026-02-20T11:15:22.222 INFO:teuthology.orchestra.run.trial043.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2026-02-20T11:15:22.222 INFO:teuthology.orchestra.run.trial043.stdout: "mon_warn_on_filestore_osds": "false", 2026-02-20T11:15:22.222 INFO:teuthology.orchestra.run.trial043.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2026-02-20T11:15:22.222 INFO:teuthology.orchestra.run.trial043.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2026-02-20T11:15:22.222 INFO:teuthology.orchestra.run.trial043.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2026-02-20T11:15:22.222 INFO:teuthology.orchestra.run.trial043.stdout: "mon_warn_on_misplaced": "false", 2026-02-20T11:15:22.222 INFO:teuthology.orchestra.run.trial043.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2026-02-20T11:15:22.222 INFO:teuthology.orchestra.run.trial043.stdout: "mon_warn_on_older_version": "true", 2026-02-20T11:15:22.222 INFO:teuthology.orchestra.run.trial043.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2026-02-20T11:15:22.222 INFO:teuthology.orchestra.run.trial043.stdout: "mon_warn_on_pool_no_app": "true", 2026-02-20T11:15:22.222 INFO:teuthology.orchestra.run.trial043.stdout: "mon_warn_on_pool_no_app_grace": "300", 2026-02-20T11:15:22.222 INFO:teuthology.orchestra.run.trial043.stdout: "mon_warn_on_pool_no_redundancy": "false", 2026-02-20T11:15:22.222 INFO:teuthology.orchestra.run.trial043.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2026-02-20T11:15:22.222 INFO:teuthology.orchestra.run.trial043.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2026-02-20T11:15:22.222 INFO:teuthology.orchestra.run.trial043.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2026-02-20T11:15:22.222 INFO:teuthology.orchestra.run.trial043.stdout: "mon_warn_on_too_few_osds": "false", 2026-02-20T11:15:22.222 INFO:teuthology.orchestra.run.trial043.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2026-02-20T11:15:22.222 INFO:teuthology.orchestra.run.trial043.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2026-02-20T11:15:22.222 INFO:teuthology.orchestra.run.trial043.stdout: "monmap": "", 2026-02-20T11:15:22.222 INFO:teuthology.orchestra.run.trial043.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2026-02-20T11:15:22.222 INFO:teuthology.orchestra.run.trial043.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2026-02-20T11:15:22.222 INFO:teuthology.orchestra.run.trial043.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2026-02-20T11:15:22.222 INFO:teuthology.orchestra.run.trial043.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2026-02-20T11:15:22.222 INFO:teuthology.orchestra.run.trial043.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2026-02-20T11:15:22.222 INFO:teuthology.orchestra.run.trial043.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2026-02-20T11:15:22.222 INFO:teuthology.orchestra.run.trial043.stdout: "motr_tracing_enabled": "false", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_async_op_threads": "3", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_async_rdma_buffer_size": "131072", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_async_rdma_cm": "false", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_async_rdma_device_name": "", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_async_rdma_dscp": "96", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_async_rdma_enable_hugepage": "false", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_async_rdma_gid_idx": "0", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_async_rdma_local_gid": "", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_async_rdma_polling_us": "1000", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_async_rdma_port_num": "1", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_async_rdma_receive_buffers": "32768", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_async_rdma_receive_queue_len": "4096", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_async_rdma_roce_ver": "1", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_async_rdma_send_buffers": "1024", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_async_rdma_sl": "3", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_async_rdma_support_srq": "true", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_async_rdma_type": "ib", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_async_reap_threshold": "5", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_bind_before_connect": "false", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_bind_ipv4": "true", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_bind_ipv6": "false", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_bind_msgr1": "true", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_bind_msgr2": "true", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_bind_port_max": "7568", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_bind_port_min": "6800", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_bind_prefer_ipv4": "false", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_bind_retry_count": "3", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_bind_retry_delay": "5", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_blackhole_client": "false", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_blackhole_mds": "false", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_blackhole_mgr": "false", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_blackhole_mon": "false", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_blackhole_osd": "false", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_client_mode": "crc", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_client_throttle_retry_time_interval": "5000", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_cluster_mode": "crc", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_cluster_type": "", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_compress_secure": "false", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_connection_idle_timeout": "900", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_connection_ready_timeout": "10", 2026-02-20T11:15:22.223 INFO:teuthology.orchestra.run.trial043.stdout: "ms_crc_data": "true", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_crc_header": "true", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_die_on_bad_msg": "false", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_die_on_bug": "true", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_die_on_old_message": "true", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_die_on_skipped_message": "false", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_die_on_unhandled_msg": "false", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_dispatch_throttle_bytes": "104857600", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_dpdk_coremask": "0xF", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_dpdk_debug_allow_loopback": "false", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_dpdk_devs_allowlist": "", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_dpdk_enable_tso": "true", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_dpdk_host_ipv4_addr": "", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_dpdk_hugepages": "", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_dpdk_hw_flow_control": "true", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_dpdk_lro": "true", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_dpdk_memory_channel": "4", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_dpdk_pmd": "", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_dpdk_port_id": "0", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_dump_corrupt_message_level": "1", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_dump_on_send": "false", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_initial_backoff": "0.200000", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_inject_delay_max": "1.000000", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_inject_delay_probability": "0.000000", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_inject_delay_type": "", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_inject_internal_delays": "0.000000", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_inject_network_congestion": "0", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_inject_socket_failures": "0", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_learn_addr_from_peer": "true", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_max_accept_failures": "4", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_max_backoff": "15.000000", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_mon_client_mode": "crc", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_mon_cluster_mode": "crc", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_mon_service_mode": "crc", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_osd_compress_min_size": "1024", 2026-02-20T11:15:22.224 INFO:teuthology.orchestra.run.trial043.stdout: "ms_osd_compress_mode": "none", 2026-02-20T11:15:22.225 INFO:teuthology.orchestra.run.trial043.stdout: "ms_osd_compression_algorithm": "snappy", 2026-02-20T11:15:22.225 INFO:teuthology.orchestra.run.trial043.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2026-02-20T11:15:22.225 INFO:teuthology.orchestra.run.trial043.stdout: "ms_pq_min_cost": "65536", 2026-02-20T11:15:22.225 INFO:teuthology.orchestra.run.trial043.stdout: "ms_public_type": "", 2026-02-20T11:15:22.225 INFO:teuthology.orchestra.run.trial043.stdout: "ms_service_mode": "crc", 2026-02-20T11:15:22.225 INFO:teuthology.orchestra.run.trial043.stdout: "ms_tcp_listen_backlog": "512", 2026-02-20T11:15:22.225 INFO:teuthology.orchestra.run.trial043.stdout: "ms_tcp_nodelay": "true", 2026-02-20T11:15:22.225 INFO:teuthology.orchestra.run.trial043.stdout: "ms_tcp_prefetch_max_size": "65536", 2026-02-20T11:15:22.225 INFO:teuthology.orchestra.run.trial043.stdout: "ms_tcp_rcvbuf": "0", 2026-02-20T11:15:22.225 INFO:teuthology.orchestra.run.trial043.stdout: "ms_time_events_min_wait_interval": "1000", 2026-02-20T11:15:22.225 INFO:teuthology.orchestra.run.trial043.stdout: "ms_type": "async+posix", 2026-02-20T11:15:22.225 INFO:teuthology.orchestra.run.trial043.stdout: "no_config_file": "false", 2026-02-20T11:15:22.225 INFO:teuthology.orchestra.run.trial043.stdout: "nvmeof_mon_client_connect_panic": "30", 2026-02-20T11:15:22.225 INFO:teuthology.orchestra.run.trial043.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2026-02-20T11:15:22.225 INFO:teuthology.orchestra.run.trial043.stdout: "nvmeof_mon_client_tick_period": "1", 2026-02-20T11:15:22.225 INFO:teuthology.orchestra.run.trial043.stdout: "objecter_completion_locks_per_session": "32", 2026-02-20T11:15:22.225 INFO:teuthology.orchestra.run.trial043.stdout: "objecter_debug_inject_relock_delay": "false", 2026-02-20T11:15:22.225 INFO:teuthology.orchestra.run.trial043.stdout: "objecter_inflight_op_bytes": "104857600", 2026-02-20T11:15:22.225 INFO:teuthology.orchestra.run.trial043.stdout: "objecter_inflight_ops": "1024", 2026-02-20T11:15:22.225 INFO:teuthology.orchestra.run.trial043.stdout: "objecter_inject_no_watch_ping": "false", 2026-02-20T11:15:22.225 INFO:teuthology.orchestra.run.trial043.stdout: "objecter_retry_writes_after_first_reply": "false", 2026-02-20T11:15:22.225 INFO:teuthology.orchestra.run.trial043.stdout: "objecter_tick_interval": "5.000000", 2026-02-20T11:15:22.225 INFO:teuthology.orchestra.run.trial043.stdout: "objecter_timeout": "10.000000", 2026-02-20T11:15:22.225 INFO:teuthology.orchestra.run.trial043.stdout: "objectstore_blackhole": "false", 2026-02-20T11:15:22.225 INFO:teuthology.orchestra.run.trial043.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2026-02-20T11:15:22.225 INFO:teuthology.orchestra.run.trial043.stdout: "openssl_conf": "", 2026-02-20T11:15:22.225 INFO:teuthology.orchestra.run.trial043.stdout: "osd_agent_delay_time": "5.000000", 2026-02-20T11:15:22.225 INFO:teuthology.orchestra.run.trial043.stdout: "osd_agent_hist_halflife": "1000", 2026-02-20T11:15:22.225 INFO:teuthology.orchestra.run.trial043.stdout: "osd_agent_max_low_ops": "2", 2026-02-20T11:15:22.225 INFO:teuthology.orchestra.run.trial043.stdout: "osd_agent_max_ops": "4", 2026-02-20T11:15:22.225 INFO:teuthology.orchestra.run.trial043.stdout: "osd_agent_min_evict_effort": "0.100000", 2026-02-20T11:15:22.225 INFO:teuthology.orchestra.run.trial043.stdout: "osd_agent_quantize_effort": "0.100000", 2026-02-20T11:15:22.225 INFO:teuthology.orchestra.run.trial043.stdout: "osd_agent_slop": "0.020000", 2026-02-20T11:15:22.225 INFO:teuthology.orchestra.run.trial043.stdout: "osd_aggregated_slow_ops_logging": "true", 2026-02-20T11:15:22.225 INFO:teuthology.orchestra.run.trial043.stdout: "osd_allow_recovery_below_min_size": "true", 2026-02-20T11:15:22.225 INFO:teuthology.orchestra.run.trial043.stdout: "osd_asio_thread_count": "2", 2026-02-20T11:15:22.225 INFO:teuthology.orchestra.run.trial043.stdout: "osd_async_recovery_min_cost": "100", 2026-02-20T11:15:22.226 INFO:teuthology.orchestra.run.trial043.stdout: "osd_auto_mark_unfound_lost": "false", 2026-02-20T11:15:22.226 INFO:teuthology.orchestra.run.trial043.stdout: "osd_backfill_retry_interval": "30.000000", 2026-02-20T11:15:22.226 INFO:teuthology.orchestra.run.trial043.stdout: "osd_backfill_scan_max": "512", 2026-02-20T11:15:22.226 INFO:teuthology.orchestra.run.trial043.stdout: "osd_backfill_scan_min": "64", 2026-02-20T11:15:22.226 INFO:teuthology.orchestra.run.trial043.stdout: "osd_backoff_on_degraded": "false", 2026-02-20T11:15:22.226 INFO:teuthology.orchestra.run.trial043.stdout: "osd_backoff_on_peering": "false", 2026-02-20T11:15:22.226 INFO:teuthology.orchestra.run.trial043.stdout: "osd_backoff_on_unfound": "true", 2026-02-20T11:15:22.226 INFO:teuthology.orchestra.run.trial043.stdout: "osd_beacon_report_interval": "300", 2026-02-20T11:15:22.226 INFO:teuthology.orchestra.run.trial043.stdout: "osd_bench_duration": "30", 2026-02-20T11:15:22.226 INFO:teuthology.orchestra.run.trial043.stdout: "osd_bench_large_size_max_throughput": "104857600", 2026-02-20T11:15:22.226 INFO:teuthology.orchestra.run.trial043.stdout: "osd_bench_max_block_size": "67108864", 2026-02-20T11:15:22.226 INFO:teuthology.orchestra.run.trial043.stdout: "osd_bench_small_size_max_iops": "100", 2026-02-20T11:15:22.226 INFO:teuthology.orchestra.run.trial043.stdout: "osd_blkin_trace_all": "false", 2026-02-20T11:15:22.226 INFO:teuthology.orchestra.run.trial043.stdout: "osd_blocked_scrub_grace_period": "120", 2026-02-20T11:15:22.226 INFO:teuthology.orchestra.run.trial043.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2026-02-20T11:15:22.226 INFO:teuthology.orchestra.run.trial043.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2026-02-20T11:15:22.226 INFO:teuthology.orchestra.run.trial043.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2026-02-20T11:15:22.226 INFO:teuthology.orchestra.run.trial043.stdout: "osd_check_for_log_corruption": "false", 2026-02-20T11:15:22.226 INFO:teuthology.orchestra.run.trial043.stdout: "osd_check_max_object_name_len_on_startup": "true", 2026-02-20T11:15:22.226 INFO:teuthology.orchestra.run.trial043.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo sem_set", 2026-02-20T11:15:22.226 INFO:teuthology.orchestra.run.trial043.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2026-02-20T11:15:22.226 INFO:teuthology.orchestra.run.trial043.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo sem_set", 2026-02-20T11:15:22.226 INFO:teuthology.orchestra.run.trial043.stdout: "osd_class_update_on_start": "true", 2026-02-20T11:15:22.226 INFO:teuthology.orchestra.run.trial043.stdout: "osd_client_message_cap": "256", 2026-02-20T11:15:22.226 INFO:teuthology.orchestra.run.trial043.stdout: "osd_client_message_size_cap": "524288000", 2026-02-20T11:15:22.226 INFO:teuthology.orchestra.run.trial043.stdout: "osd_client_op_priority": "63", 2026-02-20T11:15:22.226 INFO:teuthology.orchestra.run.trial043.stdout: "osd_client_watch_timeout": "30", 2026-02-20T11:15:22.226 INFO:teuthology.orchestra.run.trial043.stdout: "osd_command_max_records": "256", 2026-02-20T11:15:22.226 INFO:teuthology.orchestra.run.trial043.stdout: "osd_compact_on_start": "false", 2026-02-20T11:15:22.226 INFO:teuthology.orchestra.run.trial043.stdout: "osd_copyfrom_max_chunk": "8388608", 2026-02-20T11:15:22.226 INFO:teuthology.orchestra.run.trial043.stdout: "osd_crush_chooseleaf_type": "0", 2026-02-20T11:15:22.226 INFO:teuthology.orchestra.run.trial043.stdout: "osd_crush_initial_weight": "-1.000000", 2026-02-20T11:15:22.226 INFO:teuthology.orchestra.run.trial043.stdout: "osd_crush_update_on_start": "true", 2026-02-20T11:15:22.226 INFO:teuthology.orchestra.run.trial043.stdout: "osd_crush_update_weight_set": "true", 2026-02-20T11:15:22.226 INFO:teuthology.orchestra.run.trial043.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2026-02-20T11:15:22.226 INFO:teuthology.orchestra.run.trial043.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2026-02-20T11:15:22.226 INFO:teuthology.orchestra.run.trial043.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2026-02-20T11:15:22.226 INFO:teuthology.orchestra.run.trial043.stdout: "osd_debug_drop_ping_duration": "0", 2026-02-20T11:15:22.227 INFO:teuthology.orchestra.run.trial043.stdout: "osd_debug_drop_ping_probability": "0.000000", 2026-02-20T11:15:22.227 INFO:teuthology.orchestra.run.trial043.stdout: "osd_debug_feed_pullee": "-1", 2026-02-20T11:15:22.227 INFO:teuthology.orchestra.run.trial043.stdout: "osd_debug_inject_copyfrom_error": "false", 2026-02-20T11:15:22.227 INFO:teuthology.orchestra.run.trial043.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2026-02-20T11:15:22.227 INFO:teuthology.orchestra.run.trial043.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2026-02-20T11:15:22.227 INFO:teuthology.orchestra.run.trial043.stdout: "osd_debug_misdirected_ops": "false", 2026-02-20T11:15:22.227 INFO:teuthology.orchestra.run.trial043.stdout: "osd_debug_no_acting_change": "false", 2026-02-20T11:15:22.227 INFO:teuthology.orchestra.run.trial043.stdout: "osd_debug_no_purge_strays": "false", 2026-02-20T11:15:22.227 INFO:teuthology.orchestra.run.trial043.stdout: "osd_debug_op_order": "false", 2026-02-20T11:15:22.227 INFO:teuthology.orchestra.run.trial043.stdout: "osd_debug_pg_log_writeout": "false", 2026-02-20T11:15:22.227 INFO:teuthology.orchestra.run.trial043.stdout: "osd_debug_pretend_recovery_active": "false", 2026-02-20T11:15:22.227 INFO:teuthology.orchestra.run.trial043.stdout: "osd_debug_random_push_read_error": "0.000000", 2026-02-20T11:15:22.227 INFO:teuthology.orchestra.run.trial043.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2026-02-20T11:15:22.227 INFO:teuthology.orchestra.run.trial043.stdout: "osd_debug_shutdown": "false", 2026-02-20T11:15:22.227 INFO:teuthology.orchestra.run.trial043.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2026-02-20T11:15:22.227 INFO:teuthology.orchestra.run.trial043.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2026-02-20T11:15:22.227 INFO:teuthology.orchestra.run.trial043.stdout: "osd_debug_trim_objects": "false", 2026-02-20T11:15:22.227 INFO:teuthology.orchestra.run.trial043.stdout: "osd_debug_verify_cached_snaps": "false", 2026-02-20T11:15:22.227 INFO:teuthology.orchestra.run.trial043.stdout: "osd_debug_verify_missing_on_start": "false", 2026-02-20T11:15:22.227 INFO:teuthology.orchestra.run.trial043.stdout: "osd_debug_verify_snaps": "false", 2026-02-20T11:15:22.227 INFO:teuthology.orchestra.run.trial043.stdout: "osd_debug_verify_stray_on_activate": "false", 2026-02-20T11:15:22.227 INFO:teuthology.orchestra.run.trial043.stdout: "osd_deep_scrub_interval": "604800.000000", 2026-02-20T11:15:22.227 INFO:teuthology.orchestra.run.trial043.stdout: "osd_deep_scrub_interval_cv": "0.200000", 2026-02-20T11:15:22.227 INFO:teuthology.orchestra.run.trial043.stdout: "osd_deep_scrub_keys": "1024", 2026-02-20T11:15:22.227 INFO:teuthology.orchestra.run.trial043.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2026-02-20T11:15:22.228 INFO:teuthology.orchestra.run.trial043.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2026-02-20T11:15:22.228 INFO:teuthology.orchestra.run.trial043.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2026-02-20T11:15:22.228 INFO:teuthology.orchestra.run.trial043.stdout: "osd_deep_scrub_stride": "4194304", 2026-02-20T11:15:22.228 INFO:teuthology.orchestra.run.trial043.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2026-02-20T11:15:22.228 INFO:teuthology.orchestra.run.trial043.stdout: "osd_default_data_pool_replay_window": "5", 2026-02-20T11:15:22.228 INFO:teuthology.orchestra.run.trial043.stdout: "osd_default_notify_timeout": "30", 2026-02-20T11:15:22.228 INFO:teuthology.orchestra.run.trial043.stdout: "osd_delete_sleep": "0.000000", 2026-02-20T11:15:22.228 INFO:teuthology.orchestra.run.trial043.stdout: "osd_delete_sleep_hdd": "5.000000", 2026-02-20T11:15:22.228 INFO:teuthology.orchestra.run.trial043.stdout: "osd_delete_sleep_hybrid": "1.000000", 2026-02-20T11:15:22.228 INFO:teuthology.orchestra.run.trial043.stdout: "osd_delete_sleep_ssd": "1.000000", 2026-02-20T11:15:22.228 INFO:teuthology.orchestra.run.trial043.stdout: "osd_discard_disconnected_ops": "true", 2026-02-20T11:15:22.228 INFO:teuthology.orchestra.run.trial043.stdout: "osd_ec_partial_reads": "true", 2026-02-20T11:15:22.228 INFO:teuthology.orchestra.run.trial043.stdout: "osd_enable_op_tracker": "true", 2026-02-20T11:15:22.228 INFO:teuthology.orchestra.run.trial043.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2026-02-20T11:15:22.228 INFO:teuthology.orchestra.run.trial043.stdout: "osd_extblkdev_plugins": "vdo", 2026-02-20T11:15:22.228 INFO:teuthology.orchestra.run.trial043.stdout: "osd_failsafe_full_ratio": "0.970000", 2026-02-20T11:15:22.228 INFO:teuthology.orchestra.run.trial043.stdout: "osd_fast_fail_on_connection_refused": "true", 2026-02-20T11:15:22.228 INFO:teuthology.orchestra.run.trial043.stdout: "osd_fast_info": "true", 2026-02-20T11:15:22.228 INFO:teuthology.orchestra.run.trial043.stdout: "osd_fast_shutdown": "true", 2026-02-20T11:15:22.228 INFO:teuthology.orchestra.run.trial043.stdout: "osd_fast_shutdown_notify_mon": "true", 2026-02-20T11:15:22.228 INFO:teuthology.orchestra.run.trial043.stdout: "osd_fast_shutdown_timeout": "15", 2026-02-20T11:15:22.228 INFO:teuthology.orchestra.run.trial043.stdout: "osd_find_best_info_ignore_history_les": "false", 2026-02-20T11:15:22.228 INFO:teuthology.orchestra.run.trial043.stdout: "osd_force_auth_primary_missing_objects": "100", 2026-02-20T11:15:22.228 INFO:teuthology.orchestra.run.trial043.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2026-02-20T11:15:22.228 INFO:teuthology.orchestra.run.trial043.stdout: "osd_function_tracing": "false", 2026-02-20T11:15:22.228 INFO:teuthology.orchestra.run.trial043.stdout: "osd_heartbeat_grace": "20", 2026-02-20T11:15:22.228 INFO:teuthology.orchestra.run.trial043.stdout: "osd_heartbeat_interval": "6", 2026-02-20T11:15:22.228 INFO:teuthology.orchestra.run.trial043.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2026-02-20T11:15:22.228 INFO:teuthology.orchestra.run.trial043.stdout: "osd_heartbeat_min_peers": "10", 2026-02-20T11:15:22.228 INFO:teuthology.orchestra.run.trial043.stdout: "osd_heartbeat_min_size": "2000", 2026-02-20T11:15:22.228 INFO:teuthology.orchestra.run.trial043.stdout: "osd_heartbeat_stale": "600", 2026-02-20T11:15:22.228 INFO:teuthology.orchestra.run.trial043.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2026-02-20T11:15:22.228 INFO:teuthology.orchestra.run.trial043.stdout: "osd_hit_set_max_size": "100000", 2026-02-20T11:15:22.228 INFO:teuthology.orchestra.run.trial043.stdout: "osd_hit_set_min_size": "1000", 2026-02-20T11:15:22.228 INFO:teuthology.orchestra.run.trial043.stdout: "osd_hit_set_namespace": ".ceph-internal", 2026-02-20T11:15:22.228 INFO:teuthology.orchestra.run.trial043.stdout: "osd_ignore_stale_divergent_priors": "false", 2026-02-20T11:15:22.228 INFO:teuthology.orchestra.run.trial043.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_inject_failure_on_pg_removal": "false", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_journal_flush_on_shutdown": "true", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_journal_size": "5120", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_kill_backfill_at": "0", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_loop_before_reset_tphandle": "64", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_map_cache_size": "50", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_map_dedup": "true", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_map_message_max": "40", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_map_message_max_bytes": "10485760", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_map_share_max_epochs": "40", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_max_attr_name_len": "100", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_max_attr_size": "0", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_max_backfills": "1", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_max_markdown_count": "5", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_max_markdown_period": "600", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_max_object_name_len": "2048", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_max_object_namespace_len": "256", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_max_object_size": "134217728", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_max_omap_entries_per_request": "1024", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_max_pg_blocked_by": "16", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_max_pg_log_entries": "2", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_max_pgls": "1024", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_max_push_cost": "8388608", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_max_push_objects": "10", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_max_scrubs": "3", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_max_trimming_pgs": "2", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_max_write_op_reply_len": "64", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_max_write_size": "90", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_mclock_iops_capacity_low_threshold_hdd": "50.000000", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_mclock_iops_capacity_low_threshold_ssd": "1000.000000", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_mclock_override_recovery_settings": "false", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_mclock_profile": "balanced", 2026-02-20T11:15:22.229 INFO:teuthology.orchestra.run.trial043.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_mclock_scheduler_client_wgt": "1", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_mclock_skip_benchmark": "false", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_memory_base": "805306368", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_memory_cache_min": "134217728", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_memory_cache_resize_interval": "1.000000", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_memory_expected_fragmentation": "0.150000", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_memory_target": "4294967296", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_memory_target_autotune": "false", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_min_pg_log_entries": "1", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_min_recovery_priority": "0", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_mon_heartbeat_interval": "30", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_mon_report_interval": "5", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_mon_report_max_in_flight": "2", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_mon_shutdown_timeout": "5.000000", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_num_cache_shards": "32", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_num_op_tracker_shard": "32", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_numa_auto_affinity": "true", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_numa_node": "-1", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_numa_prefer_iface": "true", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_object_clean_region_max_num_intervals": "10", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_objecter_finishers": "1", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_objectstore": "bluestore", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_objectstore_fuse": "false", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_objectstore_ideal_list_max": "64", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_objectstore_tracing": "false", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_op_complaint_time": "30.000000", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_op_history_duration": "600", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_op_history_size": "20", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_op_history_slow_op_size": "20", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_op_log_threshold": "5", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_op_num_shards": "0", 2026-02-20T11:15:22.230 INFO:teuthology.orchestra.run.trial043.stdout: "osd_op_num_shards_hdd": "1", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_op_num_shards_ssd": "8", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_op_num_threads_per_shard": "0", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_op_num_threads_per_shard_hdd": "5", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_op_pq_min_cost": "65536", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_op_queue": "mclock_scheduler", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_op_queue_cut_off": "high", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_op_thread_suicide_timeout": "150", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_op_thread_timeout": "15", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_open_classes_on_start": "true", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_os_flags": "0", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_peering_op_priority": "255", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pg_delete_cost": "1048576", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pg_delete_priority": "5", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pg_log_dups_tracked": "3000", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pg_log_trim_max": "10000", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pg_log_trim_min": "0", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pg_object_context_cache_count": "64", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pool_default_cache_min_evict_age": "0", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pool_default_cache_min_flush_age": "0", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pool_default_crimson": "true", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pool_default_crush_rule": "-1", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pool_default_ec_fast_read": "false", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pool_default_erasure_code_profile": "plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pool_default_flag_bulk": "false", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pool_default_flag_ec_optimizations": "false", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pool_default_flag_hashpspool": "true", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pool_default_flag_nodelete": "false", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pool_default_flag_nopgchange": "false", 2026-02-20T11:15:22.231 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pool_default_flag_nosizechange": "false", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pool_default_flags": "0", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pool_default_min_size": "2", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pool_default_pg_num": "32", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pool_default_pgp_num": "0", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pool_default_read_ratio": "70", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pool_default_size": "3", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pool_default_type": "replicated", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pool_erasure_code_stripe_unit": "0", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_pool_use_gmt_hitset": "true", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_push_per_object_cost": "1000", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_read_ec_check_for_errors": "false", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_recover_clone_overlap": "true", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_recover_clone_overlap_limit": "10", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_recovery_cost": "20971520", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_recovery_delay_start": "0.000000", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_recovery_max_active": "0", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_recovery_max_active_hdd": "3", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_recovery_max_active_ssd": "10", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_recovery_max_chunk": "8388608", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_recovery_max_single_start": "1", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_recovery_op_priority": "3", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_recovery_op_warn_multiple": "16", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_recovery_priority": "5", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_recovery_retry_interval": "30.000000", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_recovery_sleep": "0.000000", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_recovery_sleep_degraded": "0.000000", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_recovery_sleep_degraded_hdd": "0.100000", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_recovery_sleep_degraded_hybrid": "0.025000", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_recovery_sleep_degraded_ssd": "0.000000", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_recovery_sleep_hdd": "0.100000", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_recovery_sleep_ssd": "0.000000", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_requested_scrub_priority": "5", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_rollback_to_cluster_snap": "", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_scrub_auto_repair": "false", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_scrub_auto_repair_num_errors": "5", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_scrub_backoff_ratio": "0.660000", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_scrub_begin_hour": "0", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_scrub_begin_week_day": "0", 2026-02-20T11:15:22.232 INFO:teuthology.orchestra.run.trial043.stdout: "osd_scrub_chunk_max": "15", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_scrub_chunk_min": "5", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_scrub_cost": "52428800", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_scrub_disable_reservation_queuing": "false", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_scrub_during_recovery": "false", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_scrub_end_hour": "0", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_scrub_end_week_day": "0", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_scrub_event_cost": "4096", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_scrub_extended_sleep": "0.000000", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_scrub_invalid_stats": "true", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_scrub_load_threshold": "10.000000", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_scrub_max_interval": "604800.000000", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_scrub_max_preemptions": "5", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_scrub_min_interval": "86400.000000", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_scrub_priority": "5", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_scrub_retry_after_noscrub": "60", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_scrub_retry_delay": "30", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_scrub_retry_new_interval": "10", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_scrub_retry_pg_state": "60", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_scrub_retry_trimming": "10", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_scrub_sleep": "0.000000", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_shallow_scrub_chunk_max": "100", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_shallow_scrub_chunk_min": "50", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_shutdown_pgref_assert": "false", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_skip_check_past_interval_bounds": "false", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_skip_data_digest": "false", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_smart_report_timeout": "5", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_snap_trim_cost": "1048576", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_snap_trim_priority": "5", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_snap_trim_sleep": "0.000000", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_stats_update_period_not_scrubbing": "120", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_stats_update_period_scrubbing": "15", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_target_transaction_size": "30", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_tier_default_cache_hit_set_count": "4", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2026-02-20T11:15:22.233 INFO:teuthology.orchestra.run.trial043.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "osd_tier_default_cache_mode": "writeback", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "osd_tier_promote_max_objects_sec": "25", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "osd_tracing": "false", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "osd_use_stale_snap": "false", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "osdc_blkin_trace_all": "false", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "paxos_kill_at": "0", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "paxos_max_join_drift": "10", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "paxos_min": "500", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "paxos_min_wait": "0.050000", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "paxos_propose_interval": "1.000000", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "paxos_service_trim_max": "500", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "paxos_service_trim_max_multiplier": "20", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "paxos_service_trim_min": "10", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "paxos_stash_full_interval": "25", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "paxos_trim_max": "500", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "paxos_trim_min": "250", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "perf": "true", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "plugin_crypto_accelerator": "crypto_isal", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "plugin_dir": "/usr/lib64/ceph", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "pool_availability_update_interval": "1.000000", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "public_addr": "-", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "public_addrv": "", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "public_bind_addr": "-", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "public_network": "", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "public_network_interface": "", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "qat_compressor_busy_polling": "false", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "qat_compressor_enabled": "false", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "qat_compressor_session_max_number": "256", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "rados_mon_op_timeout": "0", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "rados_osd_op_timeout": "0", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "rados_replica_read_policy": "default", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "rados_replica_read_policy_on_objclass": "false", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "rados_tracing": "false", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_atime_update_interval": "60", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_balance_parent_reads": "false", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_balance_snap_reads": "false", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_blkin_trace_all": "false", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_blocklist_expire_seconds": "0", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_blocklist_on_break_lock": "true", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_cache": "true", 2026-02-20T11:15:22.234 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_cache_block_writes_upfront": "false", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_cache_max_dirty": "25165824", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_cache_max_dirty_age": "1.000000", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_cache_max_dirty_object": "0", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_cache_policy": "writearound", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_cache_size": "33554432", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_cache_target_dirty": "16777216", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_cache_writethrough_until_flush": "true", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_clone_copy_on_read": "false", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_compression_hint": "none", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_concurrent_management_ops": "10", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_config_pool_override_update_timestamp": "0", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_default_clone_format": "auto", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_default_data_pool": "", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_default_features": "61", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_default_format": "2", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_default_map_options": "", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_default_order": "22", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_default_pool": "rbd", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_default_stripe_count": "0", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_default_stripe_unit": "0", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_disable_zero_copy_writes": "true", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_discard_granularity_bytes": "65536", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_discard_on_zeroed_write_same": "true", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_enable_alloc_hint": "true", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_io_scheduler": "simple", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_journal_commit_age": "5.000000", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_journal_max_payload_bytes": "16384", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_journal_object_flush_age": "0.000000", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_journal_object_flush_bytes": "1048576", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_journal_object_flush_interval": "0", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_journal_order": "24", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_journal_pool": "", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_journal_splay_width": "4", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_localize_parent_reads": "false", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_localize_snap_reads": "false", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2026-02-20T11:15:22.235 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_mirror_image_policy_type": "simple", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_mirror_image_state_check_interval": "30", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_mirror_memory_autotune": "true", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_mirror_memory_base": "805306368", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_mirror_memory_cache_min": "134217728", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_mirror_memory_target": "4294967296", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_mirror_perf_stats_prio": "5", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_mirroring_delete_delay": "0", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_mirroring_replay_delay": "0", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_move_to_trash_on_remove": "false", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_mtime_update_interval": "60", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_non_blocking_aio": "true", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_op_thread_timeout": "60", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_op_threads": "1", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_parent_cache_enabled": "false", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_persistent_cache_mode": "disabled", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_persistent_cache_path": "/tmp", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_persistent_cache_size": "1073741824", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_plugins": "", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_qos_bps_burst": "0", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_qos_bps_burst_seconds": "1", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_qos_bps_limit": "0", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_qos_exclude_ops": "0", 2026-02-20T11:15:22.236 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_qos_iops_burst": "0", 2026-02-20T11:15:22.237 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_qos_iops_burst_seconds": "1", 2026-02-20T11:15:22.237 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_qos_iops_limit": "0", 2026-02-20T11:15:22.237 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_qos_read_bps_burst": "0", 2026-02-20T11:15:22.237 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2026-02-20T11:15:22.237 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_qos_read_bps_limit": "0", 2026-02-20T11:15:22.237 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_qos_read_iops_burst": "0", 2026-02-20T11:15:22.237 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2026-02-20T11:15:22.237 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_qos_read_iops_limit": "0", 2026-02-20T11:15:22.237 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_qos_schedule_tick_min": "50", 2026-02-20T11:15:22.237 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_qos_write_bps_burst": "0", 2026-02-20T11:15:22.237 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2026-02-20T11:15:22.237 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_qos_write_bps_limit": "0", 2026-02-20T11:15:22.237 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_qos_write_iops_burst": "0", 2026-02-20T11:15:22.237 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2026-02-20T11:15:22.237 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_qos_write_iops_limit": "0", 2026-02-20T11:15:22.237 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_quiesce_notification_attempts": "10", 2026-02-20T11:15:22.237 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_read_from_replica_policy": "default", 2026-02-20T11:15:22.237 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2026-02-20T11:15:22.237 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_readahead_max_bytes": "524288", 2026-02-20T11:15:22.237 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_readahead_trigger_requests": "10", 2026-02-20T11:15:22.237 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_request_timed_out_seconds": "30", 2026-02-20T11:15:22.237 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_skip_partial_discard": "true", 2026-02-20T11:15:22.237 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2026-02-20T11:15:22.237 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_tracing": "false", 2026-02-20T11:15:22.237 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_validate_names": "true", 2026-02-20T11:15:22.237 INFO:teuthology.orchestra.run.trial043.stdout: "rbd_validate_pool": "true", 2026-02-20T11:15:22.237 INFO:teuthology.orchestra.run.trial043.stdout: "restapi_base_url": "", 2026-02-20T11:15:22.237 INFO:teuthology.orchestra.run.trial043.stdout: "restapi_log_level": "", 2026-02-20T11:15:22.237 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_account_default_quota_max_objects": "-1", 2026-02-20T11:15:22.237 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_account_default_quota_max_size": "-1", 2026-02-20T11:15:22.237 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_acl_grants_max_num": "100", 2026-02-20T11:15:22.237 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_admin_entry": "admin", 2026-02-20T11:15:22.237 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2026-02-20T11:15:22.237 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_asio_assert_yielding": "false", 2026-02-20T11:15:22.237 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_backend_store": "rados", 2026-02-20T11:15:22.237 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_barbican_url": "", 2026-02-20T11:15:22.237 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_beast_enable_async": "true", 2026-02-20T11:15:22.238 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_bucket_counters_cache": "false", 2026-02-20T11:15:22.238 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_bucket_counters_cache_size": "10000", 2026-02-20T11:15:22.238 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2026-02-20T11:15:22.238 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_bucket_default_quota_max_size": "-1", 2026-02-20T11:15:22.238 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_bucket_eexist_override": "false", 2026-02-20T11:15:22.238 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_bucket_index_max_aio": "128", 2026-02-20T11:15:22.238 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2026-02-20T11:15:22.238 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_bucket_logging_obj_roll_time": "300", 2026-02-20T11:15:22.238 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_bucket_persistent_notif_num_shards": "11", 2026-02-20T11:15:22.238 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_bucket_quota_cache_size": "10000", 2026-02-20T11:15:22.238 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_bucket_quota_ttl": "600", 2026-02-20T11:15:22.238 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_bucket_sync_spawn_window": "20", 2026-02-20T11:15:22.238 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_cache_enabled": "true", 2026-02-20T11:15:22.238 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_cache_expiry_interval": "900", 2026-02-20T11:15:22.238 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_cache_lru_size": "25000", 2026-02-20T11:15:22.238 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_config_store": "rados", 2026-02-20T11:15:22.238 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_content_length_compat": "false", 2026-02-20T11:15:22.238 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_copy_obj_progress": "true", 2026-02-20T11:15:22.238 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2026-02-20T11:15:22.238 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_cors_rules_max_num": "100", 2026-02-20T11:15:22.238 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_cross_domain_policy": "", 2026-02-20T11:15:22.239 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_crypt_default_encryption_key": "", 2026-02-20T11:15:22.239 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_crypt_kmip_addr": "", 2026-02-20T11:15:22.239 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_crypt_kmip_ca_path": "", 2026-02-20T11:15:22.239 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_crypt_kmip_client_cert": "", 2026-02-20T11:15:22.239 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_crypt_kmip_client_key": "", 2026-02-20T11:15:22.239 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_crypt_kmip_kms_key_template": "", 2026-02-20T11:15:22.239 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_crypt_kmip_password": "", 2026-02-20T11:15:22.239 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2026-02-20T11:15:22.239 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_crypt_kmip_username": "", 2026-02-20T11:15:22.239 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_crypt_require_ssl": "true", 2026-02-20T11:15:22.239 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2026-02-20T11:15:22.239 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2026-02-20T11:15:22.239 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_crypt_sse_s3_backend": "vault", 2026-02-20T11:15:22.239 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2026-02-20T11:15:22.239 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2026-02-20T11:15:22.239 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2026-02-20T11:15:22.239 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2026-02-20T11:15:22.239 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2026-02-20T11:15:22.239 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2026-02-20T11:15:22.239 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2026-02-20T11:15:22.239 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2026-02-20T11:15:22.239 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2026-02-20T11:15:22.239 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2026-02-20T11:15:22.239 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2026-02-20T11:15:22.239 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_crypt_suppress_logs": "true", 2026-02-20T11:15:22.239 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_crypt_vault_addr": "", 2026-02-20T11:15:22.239 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_crypt_vault_auth": "token", 2026-02-20T11:15:22.239 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_crypt_vault_namespace": "", 2026-02-20T11:15:22.239 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_crypt_vault_prefix": "", 2026-02-20T11:15:22.239 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_crypt_vault_secret_engine": "transit", 2026-02-20T11:15:22.239 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_crypt_vault_ssl_cacert": "", 2026-02-20T11:15:22.239 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2026-02-20T11:15:22.240 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2026-02-20T11:15:22.240 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_crypt_vault_token_file": "", 2026-02-20T11:15:22.240 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_crypt_vault_verify_ssl": "true", 2026-02-20T11:15:22.240 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_curl_buffersize": "524288", 2026-02-20T11:15:22.240 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_curl_low_speed_limit": "1024", 2026-02-20T11:15:22.240 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_curl_low_speed_time": "300", 2026-02-20T11:15:22.240 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_curl_tcp_keepalive": "0", 2026-02-20T11:15:22.240 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_curl_wait_timeout_ms": "1000", 2026-02-20T11:15:22.240 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2026-02-20T11:15:22.240 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2026-02-20T11:15:22.240 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2026-02-20T11:15:22.240 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2026-02-20T11:15:22.240 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_d3n_l1_fadvise": "4", 2026-02-20T11:15:22.240 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2026-02-20T11:15:22.240 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_d3n_libaio_aio_num": "64", 2026-02-20T11:15:22.240 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_d3n_libaio_aio_threads": "20", 2026-02-20T11:15:22.240 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2026-02-20T11:15:22.240 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_d4n_backend_address": "127.0.0.1:6379", 2026-02-20T11:15:22.240 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_d4n_cache_cleaning_interval": "1000", 2026-02-20T11:15:22.240 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2026-02-20T11:15:22.240 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_d4n_l1_datacache_disk_reserve": "1073741824", 2026-02-20T11:15:22.240 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2026-02-20T11:15:22.240 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2026-02-20T11:15:22.240 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_d4n_l1_fadvise": "4", 2026-02-20T11:15:22.240 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_d4n_l1_write_open_flags": "4096", 2026-02-20T11:15:22.240 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_d4n_libaio_aio_num": "64", 2026-02-20T11:15:22.240 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_d4n_libaio_aio_threads": "20", 2026-02-20T11:15:22.240 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_d4n_local_rgw_address": "127.0.0.1:8000", 2026-02-20T11:15:22.240 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2026-02-20T11:15:22.240 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_data_log_changes_size": "1000", 2026-02-20T11:15:22.240 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_data_log_num_shards": "128", 2026-02-20T11:15:22.240 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_data_log_window": "30", 2026-02-20T11:15:22.240 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_data_notify_interval_msec": "0", 2026-02-20T11:15:22.240 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_data_sync_poll_interval": "20", 2026-02-20T11:15:22.241 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_data_sync_spawn_window": "20", 2026-02-20T11:15:22.241 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2026-02-20T11:15:22.241 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2026-02-20T11:15:22.241 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_debug_inject_set_olh_err": "0", 2026-02-20T11:15:22.241 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_default_data_log_backing": "fifo", 2026-02-20T11:15:22.241 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_default_realm_info_oid": "default.realm", 2026-02-20T11:15:22.241 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_default_region_info_oid": "default.region", 2026-02-20T11:15:22.241 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_default_zone_info_oid": "default.zone", 2026-02-20T11:15:22.241 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2026-02-20T11:15:22.241 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_defer_to_bucket_acls": "", 2026-02-20T11:15:22.241 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_delete_multi_obj_max_num": "1000", 2026-02-20T11:15:22.241 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_disable_s3select": "false", 2026-02-20T11:15:22.241 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_dmclock_admin_lim": "0.000000", 2026-02-20T11:15:22.241 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_dmclock_admin_res": "100.000000", 2026-02-20T11:15:22.241 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2026-02-20T11:15:22.241 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_dmclock_auth_lim": "0.000000", 2026-02-20T11:15:22.241 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_dmclock_auth_res": "200.000000", 2026-02-20T11:15:22.241 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2026-02-20T11:15:22.241 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_dmclock_data_lim": "0.000000", 2026-02-20T11:15:22.241 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_dmclock_data_res": "500.000000", 2026-02-20T11:15:22.241 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_dmclock_data_wgt": "500.000000", 2026-02-20T11:15:22.241 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2026-02-20T11:15:22.241 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_dmclock_metadata_res": "500.000000", 2026-02-20T11:15:22.241 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2026-02-20T11:15:22.241 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_dns_name": "", 2026-02-20T11:15:22.241 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_dns_s3website_name": "", 2026-02-20T11:15:22.241 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_dynamic_resharding": "true", 2026-02-20T11:15:22.241 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2026-02-20T11:15:22.241 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2026-02-20T11:15:22.241 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2026-02-20T11:15:22.241 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_enable_gc_threads": "true", 2026-02-20T11:15:22.241 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_enable_jwks_url_verification": "false", 2026-02-20T11:15:22.241 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_enable_lc_threads": "true", 2026-02-20T11:15:22.241 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_enable_mdsearch": "true", 2026-02-20T11:15:22.241 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_enable_ops_log": "false", 2026-02-20T11:15:22.241 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_enable_quota_threads": "true", 2026-02-20T11:15:22.241 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_enable_restore_threads": "true", 2026-02-20T11:15:22.241 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_enable_static_website": "false", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_enable_usage_log": "false", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_enforce_swift_acls": "true", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_exit_timeout_secs": "120", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_expose_bucket": "false", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_extended_http_attrs": "", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_filter": "none", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_frontends": "beast port=7480", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_gc_max_concurrent_io": "10", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_gc_max_deferred": "50", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_gc_max_objs": "32", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_gc_max_queue_size": "134213632", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_gc_max_trim_chunk": "16", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_gc_obj_min_wait": "7200", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_gc_processor_max_time": "3600", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_gc_processor_period": "3600", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_get_obj_max_req_size": "4194304", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_get_obj_window_size": "16777216", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_graceful_stop": "false", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_healthcheck_disabling_path": "", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_http_notif_connection_timeout": "5", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_http_notif_max_inflight": "8192", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_http_notif_message_timeout": "10", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_ignore_get_invalid_range": "false", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_init_timeout": "300", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_kafka_connection_idle": "300", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_kafka_message_timeout": "5000", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_kafka_sleep_timeout": "10", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_keystone_accepted_admin_roles": "", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_keystone_accepted_reader_roles": "", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_keystone_admin_domain": "", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_keystone_admin_password": "", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_keystone_admin_password_path": "", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_keystone_admin_project": "", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_keystone_admin_tenant": "", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_keystone_admin_user": "", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_keystone_barbican_domain": "", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_keystone_barbican_password": "", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_keystone_barbican_project": "", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_keystone_barbican_tenant": "", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_keystone_barbican_user": "", 2026-02-20T11:15:22.242 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_keystone_implicit_tenants": "false", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_keystone_service_token_enabled": "false", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_keystone_token_cache_size": "10000", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_keystone_token_cache_ttl": "300", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_keystone_url": "", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_keystone_verify_ssl": "true", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_lc_debug_interval": "-1", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_lc_list_cnt": "1000", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_lc_lock_max_time": "90", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_lc_max_objs": "32", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_lc_max_rules": "1000", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_lc_max_worker": "3", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_lc_max_wp_worker": "128", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_lc_ordered_list_threshold": "500", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_lc_thread_delay": "0", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_ldap_dnattr": "uid", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_ldap_searchfilter": "", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_ldap_uri": "", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_lfuda_sync_frequency": "60", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_list_bucket_min_readahead": "1000", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_list_buckets_max_chunk": "1000", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_log_http_headers": "", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_log_nonexistent_bucket": "false", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_log_object_name_utc": "false", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_lua_enable": "true", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_lua_max_memory_per_state": "128000", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_lua_max_runtime_per_state": "1000", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_max_attr_name_len": "0", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_max_attr_size": "0", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_max_attrs_num_in_req": "0", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_max_chunk_size": "4194304", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_max_concurrent_requests": "1024", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_max_control_aio": "8", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_max_dynamic_shards": "1999", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_max_listing_results": "5000", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_max_notify_retries": "10", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_max_objs_per_shard": "100000", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_max_put_param_size": "1048576", 2026-02-20T11:15:22.243 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_max_put_size": "5368709120", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_max_slo_entries": "1000", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_md_log_max_shards": "64", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_md_notify_interval_msec": "200", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_meta_sync_poll_interval": "20", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_meta_sync_spawn_window": "20", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_mime_types_file": "/etc/mime.types", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_mp_lock_max_time": "600", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_multi_obj_del_max_aio": "16", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_multipart_min_part_size": "5242880", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_multipart_part_upload_limit": "10000", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_nfs_fhcache_partitions": "3", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_nfs_fhcache_size": "2017", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_nfs_frontends": "rgw-nfs", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_nfs_lru_lanes": "5", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_nfs_max_gc": "300", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_nfs_namespace_expire_secs": "300", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_nfs_run_gc_threads": "false", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_nfs_run_lc_threads": "false", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_nfs_run_quota_threads": "false", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_nfs_run_restore_threads": "false", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_nfs_run_sync_thread": "false", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_nfs_s3_fast_attrs": "false", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_nfs_write_completion_interval_s": "10", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_num_async_rados_threads": "32", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_num_control_oids": "8", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_numa_node": "-1", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_obj_stripe_size": "4194304", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_obj_tombstone_cache_size": "1000", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_objexp_chunk_size": "100", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_objexp_gc_interval": "600", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_objexp_hints_num_shards": "127", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_olh_pending_timeout_sec": "3600", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_op_thread_suicide_timeout": "0", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_op_thread_timeout": "600", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_op_tracing": "false", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_opa_token": "", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_opa_url": "", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_opa_verify_ssl": "true", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_ops_log_data_backlog": "5242880", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_ops_log_rados": "false", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_ops_log_socket_path": "", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_override_bucket_index_max_shards": "0", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_parquet_buffer_size": "16777216", 2026-02-20T11:15:22.244 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2026-02-20T11:15:22.245 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2026-02-20T11:15:22.245 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_period_push_interval": "2.000000", 2026-02-20T11:15:22.245 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_period_push_interval_max": "30.000000", 2026-02-20T11:15:22.245 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_period_root_pool": ".rgw.root", 2026-02-20T11:15:22.245 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_policy_reject_invalid_principals": "true", 2026-02-20T11:15:22.245 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2026-02-20T11:15:22.245 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_posix_cache_lanes": "3", 2026-02-20T11:15:22.245 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_posix_cache_lmdb_count": "3", 2026-02-20T11:15:22.245 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_posix_cache_max_buckets": "100", 2026-02-20T11:15:22.245 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_posix_cache_partitions": "3", 2026-02-20T11:15:22.245 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2026-02-20T11:15:22.245 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_posix_userdb_dir": "/var/lib/ceph/radosgw", 2026-02-20T11:15:22.245 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_print_continue": "true", 2026-02-20T11:15:22.245 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_print_prohibited_content_length": "false", 2026-02-20T11:15:22.245 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_put_obj_max_window_size": "67108864", 2026-02-20T11:15:22.245 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_put_obj_min_window_size": "16777216", 2026-02-20T11:15:22.245 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2026-02-20T11:15:22.245 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_rados_pool_recovery_priority": "5", 2026-02-20T11:15:22.245 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_rados_tracing": "false", 2026-02-20T11:15:22.245 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_ratelimit_interval": "60", 2026-02-20T11:15:22.245 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_realm": "", 2026-02-20T11:15:22.245 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_realm_id": "", 2026-02-20T11:15:22.245 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_realm_root_pool": ".rgw.root", 2026-02-20T11:15:22.245 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_redis_connection_pool_size": "512", 2026-02-20T11:15:22.245 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_region": "", 2026-02-20T11:15:22.245 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_region_root_pool": ".rgw.root", 2026-02-20T11:15:22.245 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_relaxed_region_enforcement": "false", 2026-02-20T11:15:22.245 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_relaxed_s3_bucket_names": "false", 2026-02-20T11:15:22.245 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_request_uri": "", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_reshard_batch_size": "64", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_reshard_bucket_lock_duration": "360", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_reshard_debug_interval": "-1", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_reshard_max_aio": "128", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_reshard_num_logs": "16", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_reshard_progress_judge_interval": "120", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_reshard_progress_judge_ratio": "0.500000", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_reshard_thread_interval": "600", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_reshardlog_threshold": "30000", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_resolve_cname": "false", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_rest_getusage_op_compat": "false", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_restore_debug_interval": "-1", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_restore_lock_max_time": "90", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_restore_max_objs": "32", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_restore_processor_period": "900", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_run_sync_thread": "true", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_s3_auth_disable_signature_url": "false", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_s3_auth_order": "sts, external, local", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_s3_auth_use_keystone": "false", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_s3_auth_use_ldap": "false", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_s3_auth_use_rados": "true", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_s3_auth_use_sts": "false", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_s3_client_max_sig_ver": "-1", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_s3_success_create_obj_status": "0", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_safe_max_objects_per_shard": "102400", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_scheduler_type": "throttler", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_script_uri": "", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_service_provider_name": "", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_shard_warning_threshold": "90.000000", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_sts_client_id": "", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_sts_client_secret": "", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_sts_entry": "sts", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_sts_key": "", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_sts_max_session_duration": "43200", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_sts_min_session_duration": "900", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_sts_token_introspection_url": "", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_swift_account_in_url": "false", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_swift_auth_entry": "auth", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_swift_auth_url": "", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_swift_custom_header": "", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_swift_enforce_content_length": "false", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_swift_need_stats": "true", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_swift_tenant_name": "", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_swift_token_expiration": "86400", 2026-02-20T11:15:22.246 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_swift_url": "", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_swift_url_prefix": "swift", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_swift_versioning_enabled": "false", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_sync_lease_period": "120", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_sync_log_trim_interval": "1200", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_sync_log_trim_max_buckets": "16", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_sync_obj_etag_verify": "false", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_sync_trace_history_size": "4096", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_sync_trace_per_node_log_size": "32", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_thread_pool_size": "512", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_topic_persistency_max_retries": "0", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_topic_persistency_sleep_duration": "0", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_topic_persistency_time_to_live": "0", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_topic_require_publish_policy": "false", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_torrent_comment": "", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_torrent_createby": "", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_torrent_encoding": "", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_torrent_flag": "false", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_torrent_max_size": "5368709120", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_torrent_origin": "", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_torrent_sha_unit": "524288", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_torrent_tracker": "", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_trust_forwarded_https": "false", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_usage_log_flush_threshold": "1024", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_usage_log_key_transition": "true", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_usage_log_tick_interval": "30", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_usage_max_shards": "32", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_usage_max_user_shards": "1", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_use_opa_authz": "false", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_user_counters_cache": "false", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_user_counters_cache_size": "10000", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_user_default_quota_max_objects": "-1", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_user_default_quota_max_size": "-1", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_user_max_buckets": "1000", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_user_policies_max_num": "100", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_user_quota_sync_idle_users": "false", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_user_quota_sync_interval": "86400", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_user_quota_sync_wait_time": "86400", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_user_unique_email": "true", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_verify_ssl": "true", 2026-02-20T11:15:22.247 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_website_routing_rules_max_num": "50", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_zone": "", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_zone_id": "", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_zone_root_pool": ".rgw.root", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_zonegroup": "", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_zonegroup_id": "", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "rgwlc_auto_session_clear": "true", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "rgwlc_skip_bucket_step": "false", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "rocksdb_block_size": "4096", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "rocksdb_bloom_bits_per_key": "20", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "rocksdb_cache_row_ratio": "0.000000", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "rocksdb_cache_shard_bits": "4", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "rocksdb_cache_size": "536870912", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "rocksdb_cache_type": "binned_lru", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "rocksdb_cf_compact_on_deletion": "true", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "rocksdb_collect_compaction_stats": "false", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "rocksdb_collect_extended_stats": "false", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "rocksdb_collect_memory_stats": "false", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "rocksdb_delete_range_threshold": "1048576", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "rocksdb_index_type": "binary_search", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "rocksdb_log_to_ceph_log": "true", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "rocksdb_metadata_block_size": "4096", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "rocksdb_partition_filters": "false", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "rocksdb_perf": "false", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "rotating_keys_bootstrap_timeout": "30", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "rotating_keys_renewal_timeout": "10", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "run_dir": "/var/run/ceph", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "seastore_block_create": "true", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "seastore_cachepin_2q_in_ratio": "0.500000", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "seastore_cachepin_2q_out_ratio": "0.500000", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "seastore_cachepin_size_pershard": "2147483648", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "seastore_cachepin_type": "LRU", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "seastore_cbjournal_size": "5368709120", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "seastore_cold_tier_generations": "3", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "seastore_data_delta_based_overwrite": "0", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "seastore_default_max_object_size": "16777216", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "seastore_default_object_metadata_reservation": "16777216", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "seastore_device_size": "53687091200", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "seastore_disable_end_to_end_data_protection": "true", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "seastore_full_integrity_check": "false", 2026-02-20T11:15:22.248 INFO:teuthology.orchestra.run.trial043.stdout: "seastore_hot_tier_generations": "5", 2026-02-20T11:15:22.249 INFO:teuthology.orchestra.run.trial043.stdout: "seastore_journal_batch_capacity": "16", 2026-02-20T11:15:22.249 INFO:teuthology.orchestra.run.trial043.stdout: "seastore_journal_batch_flush_size": "16777216", 2026-02-20T11:15:22.249 INFO:teuthology.orchestra.run.trial043.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2026-02-20T11:15:22.249 INFO:teuthology.orchestra.run.trial043.stdout: "seastore_journal_iodepth_limit": "5", 2026-02-20T11:15:22.249 INFO:teuthology.orchestra.run.trial043.stdout: "seastore_main_device_type": "SSD", 2026-02-20T11:15:22.249 INFO:teuthology.orchestra.run.trial043.stdout: "seastore_max_concurrent_transactions": "128", 2026-02-20T11:15:22.249 INFO:teuthology.orchestra.run.trial043.stdout: "seastore_max_data_allocation_size": "0", 2026-02-20T11:15:22.249 INFO:teuthology.orchestra.run.trial043.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2026-02-20T11:15:22.249 INFO:teuthology.orchestra.run.trial043.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2026-02-20T11:15:22.249 INFO:teuthology.orchestra.run.trial043.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2026-02-20T11:15:22.249 INFO:teuthology.orchestra.run.trial043.stdout: "seastore_segment_cleaner_gc_formula": "cost_benefit", 2026-02-20T11:15:22.249 INFO:teuthology.orchestra.run.trial043.stdout: "seastore_segment_size": "67108864", 2026-02-20T11:15:22.249 INFO:teuthology.orchestra.run.trial043.stdout: "service_unique_id": "", 2026-02-20T11:15:22.249 INFO:teuthology.orchestra.run.trial043.stdout: "set_keepcaps": "false", 2026-02-20T11:15:22.249 INFO:teuthology.orchestra.run.trial043.stdout: "setgroup": "", 2026-02-20T11:15:22.249 INFO:teuthology.orchestra.run.trial043.stdout: "setuser": "", 2026-02-20T11:15:22.249 INFO:teuthology.orchestra.run.trial043.stdout: "setuser_match_path": "", 2026-02-20T11:15:22.249 INFO:teuthology.orchestra.run.trial043.stdout: "subv_metrics_window_interval": "30", 2026-02-20T11:15:22.249 INFO:teuthology.orchestra.run.trial043.stdout: "target_max_misplaced_ratio": "0.050000", 2026-02-20T11:15:22.249 INFO:teuthology.orchestra.run.trial043.stdout: "thp": "false", 2026-02-20T11:15:22.249 INFO:teuthology.orchestra.run.trial043.stdout: "threadpool_default_timeout": "60", 2026-02-20T11:15:22.249 INFO:teuthology.orchestra.run.trial043.stdout: "threadpool_empty_queue_max_wait": "2", 2026-02-20T11:15:22.249 INFO:teuthology.orchestra.run.trial043.stdout: "throttler_perf_counter": "true", 2026-02-20T11:15:22.249 INFO:teuthology.orchestra.run.trial043.stdout: "tmp_dir": "/tmp", 2026-02-20T11:15:22.249 INFO:teuthology.orchestra.run.trial043.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX", 2026-02-20T11:15:22.249 INFO:teuthology.orchestra.run.trial043.stdout: "uadk_compressor_enabled": "false", 2026-02-20T11:15:22.249 INFO:teuthology.orchestra.run.trial043.stdout: "uadk_wd_sync_ctx_num": "2" 2026-02-20T11:15:22.249 INFO:teuthology.orchestra.run.trial043.stdout:} 2026-02-20T11:15:22.250 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2026-02-20T11:15:22.326 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:22.324+0000 7feb43e51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:22.329 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:22.328+0000 7feb43e51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:22.337 INFO:teuthology.orchestra.run.trial043.stdout:{} 2026-02-20T11:15:22.337 INFO:teuthology.orchestra.run.trial043.stderr:mon_osd_down_out_interval = '' (not observed, change may require restart) 2026-02-20T11:15:22.343 DEBUG:teuthology.orchestra.run.trial109:> type ceph-objectstore-tool 2026-02-20T11:15:22.360 INFO:teuthology.orchestra.run.trial109.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2026-02-20T11:15:22.360 DEBUG:teuthology.orchestra.run.trial043:> type ceph-objectstore-tool 2026-02-20T11:15:22.376 INFO:teuthology.orchestra.run.trial043.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2026-02-20T11:15:22.376 INFO:teuthology.run_tasks:Running task rados... 2026-02-20T11:15:22.396 INFO:tasks.rados:Beginning rados... 2026-02-20T11:15:22.396 INFO:tasks.rados:config is {'clients': ['client.0'], 'objects': 50, 'op_weights': {'copy_from': 0, 'delete': 50, 'read': 100, 'rollback': 0, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000, 'pool_snaps': True} 2026-02-20T11:15:22.396 INFO:tasks.rados:overrides is {'admin_socket': {'branch': 'wip-perezjos-crimson-only-19-02-2026-PR64439'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd target transaction size': 5, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Degraded data redundancy', 'is down', 'osds down', 'pg .*? is .*?degraded.*?, acting', 'pg .*? is stuck', 'pg degraded', 'PG_BACKFILL_FULL', 'Low space hindering backfill .*? backfill_toofull', 'OSD_ROOT_DOWN', 'pgs degraded', 'pgs undersized', 'is active.*backfill_toofull.*'], 'sha1': '0c3148ca19c827ca2a7a9b92c2554d99507fd1f4'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'conf': {'osd': {'osd deep scrub update digest min age': 0}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': '0c3148ca19c827ca2a7a9b92c2554d99507fd1f4'}}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '445d5c9ba18a4040e70e25303230c36c77773acb'}} 2026-02-20T11:15:22.396 INFO:tasks.rados:config is {'clients': ['client.0'], 'objects': 50, 'op_weights': {'copy_from': 0, 'delete': 50, 'read': 100, 'rollback': 0, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000, 'pool_snaps': True} 2026-02-20T11:15:22.396 DEBUG:teuthology.run_tasks:Unwinding manager rados 2026-02-20T11:15:22.414 INFO:tasks.rados:joining rados 2026-02-20T11:15:22.414 INFO:tasks.thrashosds.thrasher:starting do_thrash 2026-02-20T11:15:22.415 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2026-02-20T11:15:22.415 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2026-02-20T11:15:22.415 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2026-02-20T11:15:22.416 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:15:22.416 INFO:tasks.rados:clients are ['client.0', 'client.1'] 2026-02-20T11:15:22.417 INFO:tasks.rados:starting run 0 out of 1 2026-02-20T11:15:22.417 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2026-02-20T11:15:22.417 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2026-02-20T11:15:22.496 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:22.494+0000 7f84a96cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:22.498 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:22.494+0000 7f301645a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:22.498 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:22.498+0000 7f84a96cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:22.500 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:22.495+0000 7fe236c1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:22.500 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:22.500+0000 7f301645a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:22.502 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:22.501+0000 7fe236c1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:23.489 INFO:teuthology.orchestra.run.trial043.stderr:pool 'unique_pool_0' created 2026-02-20T11:15:23.489 INFO:teuthology.orchestra.run.trial043.stderr:reweighted osd.0 to 1 (10000) 2026-02-20T11:15:23.505 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2026-02-20T11:15:23.507 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2026-02-20T11:15:23.628 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:23.623+0000 7f88eb50b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:23.629 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:23.623+0000 7f70e79a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:23.631 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:23.630+0000 7f88eb50b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:23.631 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:23.630+0000 7f70e79a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:24.492 INFO:teuthology.orchestra.run.trial043.stderr:enabled application 'rados' on pool 'unique_pool_0' 2026-02-20T11:15:24.493 INFO:teuthology.orchestra.run.trial043.stderr:reweighted osd.1 to 1 (10000) 2026-02-20T11:15:24.508 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2026-02-20T11:15:24.560 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:15:24.633 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:24.628+0000 7fae1061a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:24.635 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:24.635+0000 7fae1061a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:24.670 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:24.668+0000 7f5673dc1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:24.672 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:24.671+0000 7f5673dc1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:25.493 INFO:teuthology.orchestra.run.trial043.stderr:reweighted osd.2 to 1 (10000) 2026-02-20T11:15:25.508 DEBUG:teuthology.orchestra.run.trial043:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph_test_rados --pool-snaps --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0 2026-02-20T11:15:25.510 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2026-02-20T11:15:25.598 INFO:tasks.rados.rados.0.trial043.stdout:adding op weight read -> 100 2026-02-20T11:15:25.598 INFO:tasks.rados.rados.0.trial043.stdout:adding op weight write -> 50 2026-02-20T11:15:25.598 INFO:tasks.rados.rados.0.trial043.stdout:adding op weight delete -> 50 2026-02-20T11:15:25.598 INFO:tasks.rados.rados.0.trial043.stdout:adding op weight snap_create -> 50 2026-02-20T11:15:25.598 INFO:tasks.rados.rados.0.trial043.stdout:adding op weight snap_remove -> 50 2026-02-20T11:15:25.598 INFO:tasks.rados.rados.0.trial043.stdout:adding op weight write_excl -> 50 2026-02-20T11:15:25.598 INFO:tasks.rados.rados.0.trial043.stdout:ceph version 20.3.0-5357-g0c3148ca (0c3148ca19c827ca2a7a9b92c2554d99507fd1f4) tentacle (dev - Debug) 2026-02-20T11:15:25.598 INFO:tasks.rados.rados.0.trial043.stdout:Configuration: 2026-02-20T11:15:25.598 INFO:tasks.rados.rados.0.trial043.stdout: Number of operations: 4000 2026-02-20T11:15:25.599 INFO:tasks.rados.rados.0.trial043.stdout: Number of objects: 50 2026-02-20T11:15:25.599 INFO:tasks.rados.rados.0.trial043.stdout: Max in flight operations: 16 2026-02-20T11:15:25.599 INFO:tasks.rados.rados.0.trial043.stdout: Object size (in bytes): 4000000 2026-02-20T11:15:25.599 INFO:tasks.rados.rados.0.trial043.stdout: Write stride min: 400000 2026-02-20T11:15:25.599 INFO:tasks.rados.rados.0.trial043.stdout: Write stride max: 800000 2026-02-20T11:15:25.628 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:25.624+0000 7f6c85e06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:25.630 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:25.630+0000 7f6c85e06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:26.501 INFO:tasks.rados.rados.0.trial043.stdout:1: write initial oid 1 2026-02-20T11:15:26.501 INFO:tasks.rados.rados.0.trial043.stdout:1: seq_num 0 ranges {416807=475249,1542129=474678} 2026-02-20T11:15:26.502 INFO:teuthology.orchestra.run.trial043.stderr:reweighted osd.3 to 1 (10000) 2026-02-20T11:15:26.505 INFO:tasks.rados.rados.0.trial043.stdout:1: writing trial04342147-1 from 416807 to 892056 tid 1 2026-02-20T11:15:26.510 INFO:tasks.rados.rados.0.trial043.stdout:1: writing trial04342147-1 from 1542129 to 2016807 tid 2 2026-02-20T11:15:26.510 INFO:tasks.rados.rados.0.trial043.stdout:2: write initial oid 2 2026-02-20T11:15:26.510 INFO:tasks.rados.rados.0.trial043.stdout:2: seq_num 1 ranges {416807=475249,1542129=474678} 2026-02-20T11:15:26.513 INFO:tasks.rados.rados.0.trial043.stdout:2: writing trial04342147-2 from 416807 to 892056 tid 1 2026-02-20T11:15:26.517 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2026-02-20T11:15:26.517 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-20T11:15:26.518 INFO:tasks.rados.rados.0.trial043.stdout:2: writing trial04342147-2 from 1542129 to 2016807 tid 2 2026-02-20T11:15:26.518 INFO:tasks.rados.rados.0.trial043.stdout:3: write initial oid 3 2026-02-20T11:15:26.518 INFO:tasks.rados.rados.0.trial043.stdout:3: seq_num 2 ranges {433614=550498,1600611=433003} 2026-02-20T11:15:26.521 INFO:tasks.rados.rados.0.trial043.stdout:3: writing trial04342147-3 from 433614 to 984112 tid 1 2026-02-20T11:15:26.526 INFO:tasks.rados.rados.0.trial043.stdout:3: writing trial04342147-3 from 1600611 to 2033614 tid 2 2026-02-20T11:15:26.526 INFO:tasks.rados.rados.0.trial043.stdout:4: write initial oid 4 2026-02-20T11:15:26.527 INFO:tasks.rados.rados.0.trial043.stdout:4: seq_num 3 ranges {450421=625747,1659093=391328} 2026-02-20T11:15:26.531 INFO:tasks.rados.rados.0.trial043.stdout:4: writing trial04342147-4 from 450421 to 1076168 tid 1 2026-02-20T11:15:26.535 INFO:tasks.rados.rados.0.trial043.stdout:4: writing trial04342147-4 from 1659093 to 2050421 tid 2 2026-02-20T11:15:26.536 INFO:tasks.rados.rados.0.trial043.stdout:5: write initial oid 5 2026-02-20T11:15:26.536 INFO:tasks.rados.rados.0.trial043.stdout:5: seq_num 4 ranges {467228=700996,1717575=349653} 2026-02-20T11:15:26.539 INFO:tasks.rados.rados.0.trial043.stdout:5: writing trial04342147-5 from 467228 to 1168224 tid 1 2026-02-20T11:15:26.543 INFO:tasks.rados.rados.0.trial043.stdout:5: writing trial04342147-5 from 1717575 to 2067228 tid 2 2026-02-20T11:15:26.544 INFO:tasks.rados.rados.0.trial043.stdout:6: write initial oid 6 2026-02-20T11:15:26.544 INFO:tasks.rados.rados.0.trial043.stdout:1: finishing write tid 1 to trial04342147-1 2026-02-20T11:15:26.544 INFO:tasks.rados.rados.0.trial043.stdout:6: seq_num 5 ranges {484035=776245,2059704=24331} 2026-02-20T11:15:26.548 INFO:tasks.rados.rados.0.trial043.stdout:6: writing trial04342147-6 from 484035 to 1260280 tid 1 2026-02-20T11:15:26.551 INFO:tasks.rados.rados.0.trial043.stdout:6: writing trial04342147-6 from 2059704 to 2084035 tid 2 2026-02-20T11:15:26.551 INFO:tasks.rados.rados.0.trial043.stdout:7: write initial oid 7 2026-02-20T11:15:26.551 INFO:tasks.rados.rados.0.trial043.stdout:7: seq_num 6 ranges {500842=451494,1718186=382656} 2026-02-20T11:15:26.554 INFO:tasks.rados.rados.0.trial043.stdout:7: writing trial04342147-7 from 500842 to 952336 tid 1 2026-02-20T11:15:26.559 INFO:tasks.rados.rados.0.trial043.stdout:7: writing trial04342147-7 from 1718186 to 2100842 tid 2 2026-02-20T11:15:26.559 INFO:tasks.rados.rados.0.trial043.stdout:8: write initial oid 8 2026-02-20T11:15:26.559 INFO:tasks.rados.rados.0.trial043.stdout:8: seq_num 7 ranges {517649=526743,1776668=340981} 2026-02-20T11:15:26.563 INFO:tasks.rados.rados.0.trial043.stdout:8: writing trial04342147-8 from 517649 to 1044392 tid 1 2026-02-20T11:15:26.568 INFO:tasks.rados.rados.0.trial043.stdout:8: writing trial04342147-8 from 1776668 to 2117649 tid 2 2026-02-20T11:15:26.568 INFO:tasks.rados.rados.0.trial043.stdout:9: write initial oid 9 2026-02-20T11:15:26.568 INFO:tasks.rados.rados.0.trial043.stdout:9: seq_num 8 ranges {534456=718345,1951503=182953} 2026-02-20T11:15:26.572 INFO:tasks.rados.rados.0.trial043.stdout:9: writing trial04342147-9 from 534456 to 1252801 tid 1 2026-02-20T11:15:26.576 INFO:tasks.rados.rados.0.trial043.stdout:9: writing trial04342147-9 from 1951503 to 2134456 tid 2 2026-02-20T11:15:26.576 INFO:tasks.rados.rados.0.trial043.stdout:10: write initial oid 10 2026-02-20T11:15:26.576 INFO:tasks.rados.rados.0.trial043.stdout:10: seq_num 9 ranges {551263=793594,1893632=257631} 2026-02-20T11:15:26.581 INFO:tasks.rados.rados.0.trial043.stdout:10: writing trial04342147-10 from 551263 to 1344857 tid 1 2026-02-20T11:15:26.584 INFO:tasks.rados.rados.0.trial043.stdout:10: writing trial04342147-10 from 1893632 to 2151263 tid 2 2026-02-20T11:15:26.584 INFO:tasks.rados.rados.0.trial043.stdout:11: write initial oid 11 2026-02-20T11:15:26.584 INFO:tasks.rados.rados.0.trial043.stdout:11: seq_num 10 ranges {568070=468843,1552114=615956} 2026-02-20T11:15:26.588 INFO:tasks.rados.rados.0.trial043.stdout:11: writing trial04342147-11 from 568070 to 1036913 tid 1 2026-02-20T11:15:26.592 INFO:tasks.rados.rados.0.trial043.stdout:11: writing trial04342147-11 from 1552114 to 2168070 tid 2 2026-02-20T11:15:26.592 INFO:tasks.rados.rados.0.trial043.stdout:12: write initial oid 12 2026-02-20T11:15:26.593 INFO:tasks.rados.rados.0.trial043.stdout:12: seq_num 11 ranges {584877=544092,1610596=562003} 2026-02-20T11:15:26.596 INFO:tasks.rados.rados.0.trial043.stdout:12: writing trial04342147-12 from 584877 to 1128969 tid 1 2026-02-20T11:15:26.599 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:26.596+0000 7fbe1a85a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:26.601 INFO:tasks.rados.rados.0.trial043.stdout:12: writing trial04342147-12 from 1610596 to 2172599 tid 2 2026-02-20T11:15:26.601 INFO:tasks.rados.rados.0.trial043.stdout:13: write initial oid 13 2026-02-20T11:15:26.601 INFO:tasks.rados.rados.0.trial043.stdout:13: seq_num 12 ranges {601684=619341,1669078=532606} 2026-02-20T11:15:26.603 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:26.602+0000 7fbe1a85a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:26.605 INFO:tasks.rados.rados.0.trial043.stdout:13: writing trial04342147-13 from 601684 to 1221025 tid 1 2026-02-20T11:15:26.609 INFO:tasks.rados.rados.0.trial043.stdout:13: writing trial04342147-13 from 1669078 to 2201684 tid 2 2026-02-20T11:15:26.610 INFO:tasks.rados.rados.0.trial043.stdout:14: write initial oid 14 2026-02-20T11:15:26.610 INFO:tasks.rados.rados.0.trial043.stdout:14: seq_num 13 ranges {618491=694590,2011207=207284} 2026-02-20T11:15:26.614 INFO:tasks.rados.rados.0.trial043.stdout:14: writing trial04342147-14 from 618491 to 1313081 tid 1 2026-02-20T11:15:26.618 INFO:tasks.rados.rados.0.trial043.stdout:14: writing trial04342147-14 from 2011207 to 2218491 tid 2 2026-02-20T11:15:26.618 INFO:tasks.rados.rados.0.trial043.stdout:15: write initial oid 15 2026-02-20T11:15:26.618 INFO:tasks.rados.rados.0.trial043.stdout:15: seq_num 14 ranges {635298=769839,2069689=165609} 2026-02-20T11:15:26.623 INFO:tasks.rados.rados.0.trial043.stdout:15: writing trial04342147-15 from 635298 to 1405137 tid 1 2026-02-20T11:15:26.626 INFO:tasks.rados.rados.0.trial043.stdout:15: writing trial04342147-15 from 2069689 to 2235298 tid 2 2026-02-20T11:15:26.627 INFO:tasks.rados.rados.0.trial043.stdout:16: write initial oid 16 2026-02-20T11:15:26.627 INFO:tasks.rados.rados.0.trial043.stdout:16: seq_num 15 ranges {652105=445088,1728171=452988} 2026-02-20T11:15:26.630 INFO:tasks.rados.rados.0.trial043.stdout:16: writing trial04342147-16 from 652105 to 1097193 tid 1 2026-02-20T11:15:26.635 INFO:tasks.rados.rados.0.trial043.stdout:16: writing trial04342147-16 from 1728171 to 2181159 tid 2 2026-02-20T11:15:26.635 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:15:26.635 INFO:tasks.rados.rados.0.trial043.stdout:2: finishing write tid 1 to trial04342147-2 2026-02-20T11:15:26.635 INFO:tasks.rados.rados.0.trial043.stdout:4: finishing write tid 1 to trial04342147-4 2026-02-20T11:15:26.636 INFO:tasks.rados.rados.0.trial043.stdout:3: finishing write tid 1 to trial04342147-3 2026-02-20T11:15:26.636 INFO:tasks.rados.rados.0.trial043.stdout:5: finishing write tid 1 to trial04342147-5 2026-02-20T11:15:26.636 INFO:tasks.rados.rados.0.trial043.stdout:2: finishing write tid 2 to trial04342147-2 2026-02-20T11:15:26.636 INFO:tasks.rados.rados.0.trial043.stdout:2: finishing write tid 3 to trial04342147-2 2026-02-20T11:15:26.636 INFO:tasks.rados.rados.0.trial043.stdout:9: finishing write tid 1 to trial04342147-9 2026-02-20T11:15:26.636 INFO:tasks.rados.rados.0.trial043.stdout:2: finishing write tid 4 to trial04342147-2 2026-02-20T11:15:26.636 INFO:tasks.rados.rados.0.trial043.stdout:2: oid 2 updating version 0 to 1 2026-02-20T11:15:26.636 INFO:tasks.rados.rados.0.trial043.stdout:2: oid 2 updating version 1 to 3 2026-02-20T11:15:26.636 INFO:tasks.rados.rados.0.trial043.stdout:2: oid 2 version 3 is already newer than 3 2026-02-20T11:15:26.636 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2026-02-20T11:15:26.637 INFO:tasks.rados.rados.0.trial043.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2026-02-20T11:15:26.637 INFO:tasks.rados.rados.0.trial043.stdout:1: finishing write tid 2 to trial04342147-1 2026-02-20T11:15:26.637 INFO:tasks.rados.rados.0.trial043.stdout:2: done (15 left) 2026-02-20T11:15:26.637 INFO:tasks.rados.rados.0.trial043.stdout:17: write initial oid 17 2026-02-20T11:15:26.637 INFO:tasks.rados.rados.0.trial043.stdout:17: seq_num 16 ranges {668912=636690,1903006=365906} 2026-02-20T11:15:26.639 INFO:tasks.rados.rados.0.trial043.stdout:17: writing trial04342147-17 from 668912 to 1305602 tid 1 2026-02-20T11:15:26.644 INFO:tasks.rados.rados.0.trial043.stdout:17: writing trial04342147-17 from 1903006 to 2268912 tid 2 2026-02-20T11:15:26.644 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:15:26.644 INFO:tasks.rados.rados.0.trial043.stdout:11: finishing write tid 1 to trial04342147-11 2026-02-20T11:15:26.644 INFO:tasks.rados.rados.0.trial043.stdout:9: finishing write tid 2 to trial04342147-9 2026-02-20T11:15:26.644 INFO:tasks.rados.rados.0.trial043.stdout:9: finishing write tid 3 to trial04342147-9 2026-02-20T11:15:26.644 INFO:tasks.rados.rados.0.trial043.stdout:9: finishing write tid 4 to trial04342147-9 2026-02-20T11:15:26.644 INFO:tasks.rados.rados.0.trial043.stdout:9: oid 9 updating version 0 to 3 2026-02-20T11:15:26.644 INFO:tasks.rados.rados.0.trial043.stdout:9: oid 9 version 3 is already newer than 3 2026-02-20T11:15:26.644 INFO:tasks.rados.rados.0.trial043.stdout:9: oid 9 version 3 is already newer than 1 2026-02-20T11:15:26.645 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 9 v 3 (ObjNum 8 snap 0 seq_num 8) dirty exists 2026-02-20T11:15:26.645 INFO:tasks.rados.rados.0.trial043.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2026-02-20T11:15:26.645 INFO:tasks.rados.rados.0.trial043.stdout:9: done (15 left) 2026-02-20T11:15:26.645 INFO:tasks.rados.rados.0.trial043.stdout:18: write initial oid 18 2026-02-20T11:15:26.645 INFO:tasks.rados.rados.0.trial043.stdout:18: seq_num 17 ranges {685719=711939,1845135=440584} 2026-02-20T11:15:26.650 INFO:tasks.rados.rados.0.trial043.stdout:18: writing trial04342147-18 from 685719 to 1397658 tid 1 2026-02-20T11:15:26.654 INFO:tasks.rados.rados.0.trial043.stdout:18: writing trial04342147-18 from 1845135 to 2285719 tid 2 2026-02-20T11:15:26.654 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:15:26.654 INFO:tasks.rados.rados.0.trial043.stdout:4: finishing write tid 2 to trial04342147-4 2026-02-20T11:15:26.654 INFO:tasks.rados.rados.0.trial043.stdout:1: finishing write tid 3 to trial04342147-1 2026-02-20T11:15:26.655 INFO:tasks.rados.rados.0.trial043.stdout:11: finishing write tid 2 to trial04342147-11 2026-02-20T11:15:26.655 INFO:tasks.rados.rados.0.trial043.stdout:11: finishing write tid 3 to trial04342147-11 2026-02-20T11:15:26.655 INFO:tasks.rados.rados.0.trial043.stdout:1: finishing write tid 4 to trial04342147-1 2026-02-20T11:15:26.655 INFO:tasks.rados.rados.0.trial043.stdout:1: oid 1 updating version 0 to 1 2026-02-20T11:15:26.655 INFO:tasks.rados.rados.0.trial043.stdout:1: oid 1 updating version 1 to 3 2026-02-20T11:15:26.655 INFO:tasks.rados.rados.0.trial043.stdout:1: oid 1 version 3 is already newer than 3 2026-02-20T11:15:26.655 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2026-02-20T11:15:26.655 INFO:tasks.rados.rados.0.trial043.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2026-02-20T11:15:26.655 INFO:tasks.rados.rados.0.trial043.stdout:1: done (15 left) 2026-02-20T11:15:26.655 INFO:tasks.rados.rados.0.trial043.stdout:19: write initial oid 19 2026-02-20T11:15:26.655 INFO:tasks.rados.rados.0.trial043.stdout:19: seq_num 18 ranges {702526=787188,1903617=398909} 2026-02-20T11:15:26.659 INFO:tasks.rados.rados.0.trial043.stdout:19: writing trial04342147-19 from 702526 to 1489714 tid 1 2026-02-20T11:15:26.664 INFO:tasks.rados.rados.0.trial043.stdout:19: writing trial04342147-19 from 1903617 to 2302526 tid 2 2026-02-20T11:15:26.664 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:15:26.665 INFO:tasks.rados.rados.0.trial043.stdout:11: finishing write tid 4 to trial04342147-11 2026-02-20T11:15:26.665 INFO:tasks.rados.rados.0.trial043.stdout:11: oid 11 updating version 0 to 3 2026-02-20T11:15:26.665 INFO:tasks.rados.rados.0.trial043.stdout:11: oid 11 version 3 is already newer than 3 2026-02-20T11:15:26.665 INFO:tasks.rados.rados.0.trial043.stdout:11: oid 11 version 3 is already newer than 1 2026-02-20T11:15:26.665 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 11 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2026-02-20T11:15:26.666 INFO:tasks.rados.rados.0.trial043.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2026-02-20T11:15:26.666 INFO:tasks.rados.rados.0.trial043.stdout:4: finishing write tid 3 to trial04342147-4 2026-02-20T11:15:26.666 INFO:tasks.rados.rados.0.trial043.stdout:11: done (15 left) 2026-02-20T11:15:26.666 INFO:tasks.rados.rados.0.trial043.stdout:20: write initial oid 20 2026-02-20T11:15:26.666 INFO:tasks.rados.rados.0.trial043.stdout:20: seq_num 19 ranges {719333=462437,1962099=357234} 2026-02-20T11:15:26.667 INFO:tasks.rados.rados.0.trial043.stdout:20: writing trial04342147-20 from 719333 to 1181770 tid 1 2026-02-20T11:15:26.672 INFO:tasks.rados.rados.0.trial043.stdout:20: writing trial04342147-20 from 1962099 to 2319333 tid 2 2026-02-20T11:15:26.672 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:15:26.672 INFO:tasks.rados.rados.0.trial043.stdout:5: finishing write tid 2 to trial04342147-5 2026-02-20T11:15:26.673 INFO:tasks.rados.rados.0.trial043.stdout:14: finishing write tid 1 to trial04342147-14 2026-02-20T11:15:26.673 INFO:tasks.rados.rados.0.trial043.stdout:4: finishing write tid 4 to trial04342147-4 2026-02-20T11:15:26.673 INFO:tasks.rados.rados.0.trial043.stdout:4: oid 4 updating version 0 to 1 2026-02-20T11:15:26.673 INFO:tasks.rados.rados.0.trial043.stdout:4: oid 4 updating version 1 to 3 2026-02-20T11:15:26.673 INFO:tasks.rados.rados.0.trial043.stdout:4: oid 4 version 3 is already newer than 3 2026-02-20T11:15:26.673 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2026-02-20T11:15:26.673 INFO:tasks.rados.rados.0.trial043.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2026-02-20T11:15:26.673 INFO:tasks.rados.rados.0.trial043.stdout:4: done (15 left) 2026-02-20T11:15:26.673 INFO:tasks.rados.rados.0.trial043.stdout:21: write initial oid 21 2026-02-20T11:15:26.673 INFO:tasks.rados.rados.0.trial043.stdout:21: seq_num 20 ranges {736140=537686,2020581=315559} 2026-02-20T11:15:26.676 INFO:tasks.rados.rados.0.trial043.stdout:21: writing trial04342147-21 from 736140 to 1273826 tid 1 2026-02-20T11:15:26.682 INFO:tasks.rados.rados.0.trial043.stdout:21: writing trial04342147-21 from 2020581 to 2336140 tid 2 2026-02-20T11:15:26.682 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:15:26.682 INFO:tasks.rados.rados.0.trial043.stdout:5: finishing write tid 3 to trial04342147-5 2026-02-20T11:15:26.682 INFO:tasks.rados.rados.0.trial043.stdout:8: finishing write tid 1 to trial04342147-8 2026-02-20T11:15:26.683 INFO:tasks.rados.rados.0.trial043.stdout:5: finishing write tid 4 to trial04342147-5 2026-02-20T11:15:26.683 INFO:tasks.rados.rados.0.trial043.stdout:5: oid 5 updating version 0 to 3 2026-02-20T11:15:26.683 INFO:tasks.rados.rados.0.trial043.stdout:5: oid 5 version 3 is already newer than 3 2026-02-20T11:15:26.683 INFO:tasks.rados.rados.0.trial043.stdout:5: oid 5 version 3 is already newer than 1 2026-02-20T11:15:26.683 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2026-02-20T11:15:26.683 INFO:tasks.rados.rados.0.trial043.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2026-02-20T11:15:26.683 INFO:tasks.rados.rados.0.trial043.stdout:3: finishing write tid 2 to trial04342147-3 2026-02-20T11:15:26.683 INFO:tasks.rados.rados.0.trial043.stdout:12: finishing write tid 1 to trial04342147-12 2026-02-20T11:15:26.683 INFO:tasks.rados.rados.0.trial043.stdout:14: finishing write tid 2 to trial04342147-14 2026-02-20T11:15:26.684 INFO:tasks.rados.rados.0.trial043.stdout:3: finishing write tid 3 to trial04342147-3 2026-02-20T11:15:26.684 INFO:tasks.rados.rados.0.trial043.stdout:5: done (15 left) 2026-02-20T11:15:26.684 INFO:tasks.rados.rados.0.trial043.stdout:22: write initial oid 22 2026-02-20T11:15:26.684 INFO:tasks.rados.rados.0.trial043.stdout:22: seq_num 21 ranges {752947=612935,1962710=390237} 2026-02-20T11:15:26.686 INFO:tasks.rados.rados.0.trial043.stdout:22: writing trial04342147-22 from 752947 to 1365882 tid 1 2026-02-20T11:15:26.690 INFO:tasks.rados.rados.0.trial043.stdout:22: writing trial04342147-22 from 1962710 to 2352947 tid 2 2026-02-20T11:15:26.690 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:15:26.690 INFO:tasks.rados.rados.0.trial043.stdout:3: finishing write tid 4 to trial04342147-3 2026-02-20T11:15:26.691 INFO:tasks.rados.rados.0.trial043.stdout:3: oid 3 updating version 0 to 1 2026-02-20T11:15:26.691 INFO:tasks.rados.rados.0.trial043.stdout:3: oid 3 updating version 1 to 3 2026-02-20T11:15:26.691 INFO:tasks.rados.rados.0.trial043.stdout:3: oid 3 version 3 is already newer than 3 2026-02-20T11:15:26.691 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2026-02-20T11:15:26.691 INFO:tasks.rados.rados.0.trial043.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2026-02-20T11:15:26.691 INFO:tasks.rados.rados.0.trial043.stdout:3: done (15 left) 2026-02-20T11:15:26.691 INFO:tasks.rados.rados.0.trial043.stdout:23: write initial oid 23 2026-02-20T11:15:26.691 INFO:tasks.rados.rados.0.trial043.stdout:23: seq_num 22 ranges {769754=688184,2021192=348562} 2026-02-20T11:15:26.695 INFO:tasks.rados.rados.0.trial043.stdout:23: writing trial04342147-23 from 769754 to 1457938 tid 1 2026-02-20T11:15:26.700 INFO:tasks.rados.rados.0.trial043.stdout:23: writing trial04342147-23 from 2021192 to 2369754 tid 2 2026-02-20T11:15:26.700 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:15:26.700 INFO:tasks.rados.rados.0.trial043.stdout:18: finishing write tid 1 to trial04342147-18 2026-02-20T11:15:26.700 INFO:tasks.rados.rados.0.trial043.stdout:14: finishing write tid 3 to trial04342147-14 2026-02-20T11:15:26.700 INFO:tasks.rados.rados.0.trial043.stdout:13: finishing write tid 1 to trial04342147-13 2026-02-20T11:15:26.700 INFO:tasks.rados.rados.0.trial043.stdout:7: finishing write tid 1 to trial04342147-7 2026-02-20T11:15:26.700 INFO:tasks.rados.rados.0.trial043.stdout:14: finishing write tid 4 to trial04342147-14 2026-02-20T11:15:26.700 INFO:tasks.rados.rados.0.trial043.stdout:14: oid 14 updating version 0 to 3 2026-02-20T11:15:26.700 INFO:tasks.rados.rados.0.trial043.stdout:14: oid 14 version 3 is already newer than 1 2026-02-20T11:15:26.701 INFO:tasks.rados.rados.0.trial043.stdout:14: oid 14 version 3 is already newer than 3 2026-02-20T11:15:26.701 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 14 v 3 (ObjNum 13 snap 0 seq_num 13) dirty exists 2026-02-20T11:15:26.701 INFO:tasks.rados.rados.0.trial043.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2026-02-20T11:15:26.701 INFO:tasks.rados.rados.0.trial043.stdout:16: finishing write tid 1 to trial04342147-16 2026-02-20T11:15:26.701 INFO:tasks.rados.rados.0.trial043.stdout:14: done (15 left) 2026-02-20T11:15:26.701 INFO:tasks.rados.rados.0.trial043.stdout:24: write initial oid 24 2026-02-20T11:15:26.701 INFO:tasks.rados.rados.0.trial043.stdout:24: seq_num 23 ranges {786561=479786,1796027=467664} 2026-02-20T11:15:26.703 INFO:tasks.rados.rados.0.trial043.stdout:24: writing trial04342147-24 from 786561 to 1266347 tid 1 2026-02-20T11:15:26.708 INFO:tasks.rados.rados.0.trial043.stdout:24: writing trial04342147-24 from 1796027 to 2263691 tid 2 2026-02-20T11:15:26.708 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:15:26.708 INFO:tasks.rados.rados.0.trial043.stdout:19: finishing write tid 1 to trial04342147-19 2026-02-20T11:15:26.708 INFO:tasks.rados.rados.0.trial043.stdout:18: finishing write tid 2 to trial04342147-18 2026-02-20T11:15:26.708 INFO:tasks.rados.rados.0.trial043.stdout:6: finishing write tid 1 to trial04342147-6 2026-02-20T11:15:26.708 INFO:tasks.rados.rados.0.trial043.stdout:17: finishing write tid 1 to trial04342147-17 2026-02-20T11:15:26.717 INFO:tasks.rados.rados.0.trial043.stdout:18: finishing write tid 3 to trial04342147-18 2026-02-20T11:15:26.719 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:15:26.720 INFO:teuthology.orchestra.run.trial043.stderr:dumped all 2026-02-20T11:15:26.723 INFO:tasks.rados.rados.0.trial043.stdout:18: finishing write tid 4 to trial04342147-18 2026-02-20T11:15:26.723 INFO:tasks.rados.rados.0.trial043.stdout:18: oid 18 updating version 0 to 1 2026-02-20T11:15:26.724 INFO:tasks.rados.rados.0.trial043.stdout:18: oid 18 updating version 1 to 3 2026-02-20T11:15:26.724 INFO:tasks.rados.rados.0.trial043.stdout:18: oid 18 version 3 is already newer than 3 2026-02-20T11:15:26.724 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 18 v 3 (ObjNum 17 snap 0 seq_num 17) dirty exists 2026-02-20T11:15:26.724 INFO:tasks.rados.rados.0.trial043.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2026-02-20T11:15:26.724 INFO:tasks.rados.rados.0.trial043.stdout:18: done (15 left) 2026-02-20T11:15:26.724 INFO:tasks.rados.rados.0.trial043.stdout:25: write initial oid 25 2026-02-20T11:15:26.725 INFO:tasks.rados.rados.0.trial043.stdout:25: seq_num 24 ranges {403368=555035,1454509=727675} 2026-02-20T11:15:26.727 INFO:tasks.rados.rados.0.trial043.stdout:25: writing trial04342147-25 from 403368 to 958403 tid 1 2026-02-20T11:15:26.731 INFO:teuthology.orchestra.run.trial043.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2026-02-20T11:15:26.500322+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209715136,"kb_used":83890120,"kb_used_data":83890120,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":125825016,"statfs":{"total":214748299264,"available":128844816384,"internally_reserved":0,"allocated":85903482880,"data_stored":85903482880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.561720"},"pg_stats":[{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.981411+0000","last_change":"2026-02-20T11:15:16.984535+0000","last_active":"2026-02-20T11:15:16.981411+0000","last_peered":"2026-02-20T11:15:16.981411+0000","last_clean":"2026-02-20T11:15:16.981411+0000","last_became_active":"2026-02-20T11:15:15.006596+0000","last_became_peered":"2026-02-20T11:15:15.006596+0000","last_unstale":"2026-02-20T11:15:16.981411+0000","last_undegraded":"2026-02-20T11:15:16.981411+0000","last_fullsized":"2026-02-20T11:15:16.981411+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.983008+0000","last_change":"2026-02-20T11:15:16.985758+0000","last_active":"2026-02-20T11:15:16.983008+0000","last_peered":"2026-02-20T11:15:16.983008+0000","last_clean":"2026-02-20T11:15:16.983008+0000","last_became_active":"2026-02-20T11:15:15.019055+0000","last_became_peered":"2026-02-20T11:15:15.019055+0000","last_unstale":"2026-02-20T11:15:16.983008+0000","last_undegraded":"2026-02-20T11:15:16.983008+0000","last_fullsized":"2026-02-20T11:15:16.983008+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.2","version":"14'1","reported_seq":18,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.983646+0000","last_change":"2026-02-20T11:15:16.983646+0000","last_active":"2026-02-20T11:15:16.983646+0000","last_peered":"2026-02-20T11:15:16.983646+0000","last_clean":"2026-02-20T11:15:16.983646+0000","last_became_active":"2026-02-20T11:15:15.012729+0000","last_became_peered":"2026-02-20T11:15:15.012729+0000","last_unstale":"2026-02-20T11:15:16.983646+0000","last_undegraded":"2026-02-20T11:15:16.983646+0000","last_fullsized":"2026-02-20T11:15:16.983646+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.980370+0000","last_change":"2026-02-20T11:15:16.983291+0000","last_active":"2026-02-20T11:15:16.980370+0000","last_peered":"2026-02-20T11:15:16.980370+0000","last_clean":"2026-02-20T11:15:16.980370+0000","last_became_active":"2026-02-20T11:15:15.009983+0000","last_became_peered":"2026-02-20T11:15:15.009983+0000","last_unstale":"2026-02-20T11:15:16.980370+0000","last_undegraded":"2026-02-20T11:15:16.980370+0000","last_fullsized":"2026-02-20T11:15:16.980370+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'52","reported_seq":82,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.980870+0000","last_change":"2026-02-20T11:15:15.035494+0000","last_active":"2026-02-20T11:15:16.980870+0000","last_peered":"2026-02-20T11:15:16.980870+0000","last_clean":"2026-02-20T11:15:16.980870+0000","last_became_active":"2026-02-20T11:15:15.012645+0000","last_became_peered":"2026-02-20T11:15:15.012645+0000","last_unstale":"2026-02-20T11:15:16.980870+0000","last_undegraded":"2026-02-20T11:15:16.980870+0000","last_fullsized":"2026-02-20T11:15:16.980870+0000","mapping_epoch":12,"log_start":"13'48","ondisk_log_start":"13'48","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":48,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.982312+0000","last_change":"2026-02-20T11:15:16.984208+0000","last_active":"2026-02-20T11:15:16.982312+0000","last_peered":"2026-02-20T11:15:16.982312+0000","last_clean":"2026-02-20T11:15:16.982312+0000","last_became_active":"2026-02-20T11:15:15.027750+0000","last_became_peered":"2026-02-20T11:15:15.027750+0000","last_unstale":"2026-02-20T11:15:16.982312+0000","last_undegraded":"2026-02-20T11:15:16.982312+0000","last_fullsized":"2026-02-20T11:15:16.982312+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.983812+0000","last_change":"2026-02-20T11:15:16.985671+0000","last_active":"2026-02-20T11:15:16.983812+0000","last_peered":"2026-02-20T11:15:16.983812+0000","last_clean":"2026-02-20T11:15:16.983812+0000","last_became_active":"2026-02-20T11:15:15.003549+0000","last_became_peered":"2026-02-20T11:15:15.003549+0000","last_unstale":"2026-02-20T11:15:16.983812+0000","last_undegraded":"2026-02-20T11:15:16.983812+0000","last_fullsized":"2026-02-20T11:15:16.983812+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.982620+0000","last_change":"2026-02-20T11:15:16.986873+0000","last_active":"2026-02-20T11:15:16.982620+0000","last_peered":"2026-02-20T11:15:16.982620+0000","last_clean":"2026-02-20T11:15:16.982620+0000","last_became_active":"2026-02-20T11:15:15.018131+0000","last_became_peered":"2026-02-20T11:15:15.018131+0000","last_unstale":"2026-02-20T11:15:16.982620+0000","last_undegraded":"2026-02-20T11:15:16.982620+0000","last_fullsized":"2026-02-20T11:15:16.982620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.983722+0000","last_change":"2026-02-20T11:15:16.986570+0000","last_active":"2026-02-20T11:15:16.983722+0000","last_peered":"2026-02-20T11:15:16.983722+0000","last_clean":"2026-02-20T11:15:16.983722+0000","last_became_active":"2026-02-20T11:15:15.057391+0000","last_became_peered":"2026-02-20T11:15:15.057391+0000","last_unstale":"2026-02-20T11:15:16.983722+0000","last_undegraded":"2026-02-20T11:15:16.983722+0000","last_fullsized":"2026-02-20T11:15:16.983722+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32210939904,"internally_reserved":0,"allocated":21476134912,"data_stored":21476134912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96633778176,"internally_reserved":0,"allocated":64427446272,"data_stored":64427446272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":24,"acting":24,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640265,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972788,"kb_used_data":20972788,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31455996,"statfs":{"total":53687074816,"available":32210939904,"internally_reserved":0,"allocated":21476134912,"data_stored":21476134912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640266,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20971984,"kb_used_data":20971984,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456800,"statfs":{"total":53687074816,"available":32211763200,"internally_reserved":0,"allocated":21475311616,"data_stored":21475311616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972656,"kb_used_data":20972656,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456128,"statfs":{"total":53687074816,"available":32211075072,"internally_reserved":0,"allocated":21475999744,"data_stored":21475999744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640266,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972692,"kb_used_data":20972692,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456092,"statfs":{"total":53687074816,"available":32211038208,"internally_reserved":0,"allocated":21476036608,"data_stored":21476036608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32211075072,"internally_reserved":0,"allocated":21475999744,"data_stored":21475999744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53687074816,"available":32211763200,"internally_reserved":0,"allocated":21475311616,"data_stored":21475311616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":53687074816,"available":32210939904,"internally_reserved":0,"allocated":21476134912,"data_stored":21476134912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53687074816,"available":32210939904,"internally_reserved":0,"allocated":21476134912,"data_stored":21476134912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-20T11:15:26.734 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-20T11:15:26.735 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:15:26.737 INFO:tasks.rados.rados.0.trial043.stdout:25: writing trial04342147-25 from 1454509 to 2182184 tid 2 2026-02-20T11:15:26.737 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:15:26.737 INFO:tasks.rados.rados.0.trial043.stdout:19: finishing write tid 2 to trial04342147-19 2026-02-20T11:15:26.737 INFO:tasks.rados.rados.0.trial043.stdout:13: finishing write tid 2 to trial04342147-13 2026-02-20T11:15:26.737 INFO:tasks.rados.rados.0.trial043.stdout:19: finishing write tid 3 to trial04342147-19 2026-02-20T11:15:26.737 INFO:tasks.rados.rados.0.trial043.stdout:15: finishing write tid 1 to trial04342147-15 2026-02-20T11:15:26.738 INFO:tasks.rados.rados.0.trial043.stdout:19: finishing write tid 4 to trial04342147-19 2026-02-20T11:15:26.738 INFO:tasks.rados.rados.0.trial043.stdout:19: oid 19 updating version 0 to 3 2026-02-20T11:15:26.738 INFO:tasks.rados.rados.0.trial043.stdout:19: oid 19 version 3 is already newer than 3 2026-02-20T11:15:26.738 INFO:tasks.rados.rados.0.trial043.stdout:19: oid 19 version 3 is already newer than 1 2026-02-20T11:15:26.738 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 19 v 3 (ObjNum 18 snap 0 seq_num 18) dirty exists 2026-02-20T11:15:26.738 INFO:tasks.rados.rados.0.trial043.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2026-02-20T11:15:26.738 INFO:tasks.rados.rados.0.trial043.stdout:19: done (15 left) 2026-02-20T11:15:26.738 INFO:tasks.rados.rados.0.trial043.stdout:26: write initial oid 26 2026-02-20T11:15:26.738 INFO:tasks.rados.rados.0.trial043.stdout:26: seq_num 25 ranges {420175=630284,1796638=471333} 2026-02-20T11:15:26.739 INFO:tasks.rados.rados.0.trial043.stdout:26: writing trial04342147-26 from 420175 to 1050459 tid 1 2026-02-20T11:15:26.742 INFO:tasks.rados.rados.0.trial043.stdout:26: writing trial04342147-26 from 1796638 to 2267971 tid 2 2026-02-20T11:15:26.742 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:15:26.742 INFO:tasks.rados.rados.0.trial043.stdout:6: finishing write tid 2 to trial04342147-6 2026-02-20T11:15:26.743 INFO:tasks.rados.rados.0.trial043.stdout:16: finishing write tid 2 to trial04342147-16 2026-02-20T11:15:26.743 INFO:tasks.rados.rados.0.trial043.stdout:6: finishing write tid 3 to trial04342147-6 2026-02-20T11:15:26.745 INFO:tasks.rados.rados.0.trial043.stdout:21: finishing write tid 1 to trial04342147-21 2026-02-20T11:15:26.748 INFO:tasks.rados.rados.0.trial043.stdout:6: finishing write tid 4 to trial04342147-6 2026-02-20T11:15:26.749 INFO:tasks.rados.rados.0.trial043.stdout:6: oid 6 updating version 0 to 3 2026-02-20T11:15:26.749 INFO:tasks.rados.rados.0.trial043.stdout:6: oid 6 version 3 is already newer than 1 2026-02-20T11:15:26.749 INFO:tasks.rados.rados.0.trial043.stdout:6: oid 6 version 3 is already newer than 3 2026-02-20T11:15:26.749 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 6 v 3 (ObjNum 5 snap 0 seq_num 5) dirty exists 2026-02-20T11:15:26.749 INFO:tasks.rados.rados.0.trial043.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2026-02-20T11:15:26.749 INFO:tasks.rados.rados.0.trial043.stdout:6: done (15 left) 2026-02-20T11:15:26.749 INFO:tasks.rados.rados.0.trial043.stdout:27: write initial oid 27 2026-02-20T11:15:26.749 INFO:tasks.rados.rados.0.trial043.stdout:27: seq_num 26 ranges {436982=705533,1855120=581862} 2026-02-20T11:15:26.752 INFO:tasks.rados.rados.0.trial043.stdout:27: writing trial04342147-27 from 436982 to 1142515 tid 1 2026-02-20T11:15:26.758 INFO:tasks.rados.rados.0.trial043.stdout:27: writing trial04342147-27 from 1855120 to 2436982 tid 2 2026-02-20T11:15:26.758 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:15:26.759 INFO:tasks.rados.rados.0.trial043.stdout:7: finishing write tid 2 to trial04342147-7 2026-02-20T11:15:26.759 INFO:tasks.rados.rados.0.trial043.stdout:13: finishing write tid 3 to trial04342147-13 2026-02-20T11:15:26.760 INFO:tasks.rados.rados.0.trial043.stdout:13: finishing write tid 4 to trial04342147-13 2026-02-20T11:15:26.761 INFO:tasks.rados.rados.0.trial043.stdout:13: oid 13 updating version 0 to 4 2026-02-20T11:15:26.761 INFO:tasks.rados.rados.0.trial043.stdout:13: oid 13 updating version 4 to 6 2026-02-20T11:15:26.761 INFO:tasks.rados.rados.0.trial043.stdout:13: oid 13 version 6 is already newer than 6 2026-02-20T11:15:26.761 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 13 v 6 (ObjNum 12 snap 0 seq_num 12) dirty exists 2026-02-20T11:15:26.761 INFO:tasks.rados.rados.0.trial043.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2026-02-20T11:15:26.761 INFO:tasks.rados.rados.0.trial043.stdout:13: done (15 left) 2026-02-20T11:15:26.762 INFO:tasks.rados.rados.0.trial043.stdout:28: write initial oid 28 2026-02-20T11:15:26.762 INFO:tasks.rados.rados.0.trial043.stdout:28: seq_num 27 ranges {453789=780782,1913602=475002} 2026-02-20T11:15:26.765 INFO:tasks.rados.rados.0.trial043.stdout:28: writing trial04342147-28 from 453789 to 1234571 tid 1 2026-02-20T11:15:26.770 INFO:tasks.rados.rados.0.trial043.stdout:28: writing trial04342147-28 from 1913602 to 2388604 tid 2 2026-02-20T11:15:26.770 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:15:26.770 INFO:tasks.rados.rados.0.trial043.stdout:7: finishing write tid 3 to trial04342147-7 2026-02-20T11:15:26.770 INFO:tasks.rados.rados.0.trial043.stdout:12: finishing write tid 2 to trial04342147-12 2026-02-20T11:15:26.770 INFO:tasks.rados.rados.0.trial043.stdout:15: finishing write tid 2 to trial04342147-15 2026-02-20T11:15:26.770 INFO:tasks.rados.rados.0.trial043.stdout:21: finishing write tid 2 to trial04342147-21 2026-02-20T11:15:26.771 INFO:tasks.rados.rados.0.trial043.stdout:25: finishing write tid 1 to trial04342147-25 2026-02-20T11:15:26.776 INFO:tasks.rados.rados.0.trial043.stdout:12: finishing write tid 3 to trial04342147-12 2026-02-20T11:15:26.780 INFO:tasks.rados.rados.0.trial043.stdout:12: finishing write tid 4 to trial04342147-12 2026-02-20T11:15:26.780 INFO:tasks.rados.rados.0.trial043.stdout:12: oid 12 updating version 0 to 4 2026-02-20T11:15:26.780 INFO:tasks.rados.rados.0.trial043.stdout:12: oid 12 updating version 4 to 6 2026-02-20T11:15:26.780 INFO:tasks.rados.rados.0.trial043.stdout:12: oid 12 version 6 is already newer than 6 2026-02-20T11:15:26.780 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 12 v 6 (ObjNum 11 snap 0 seq_num 11) dirty exists 2026-02-20T11:15:26.780 INFO:tasks.rados.rados.0.trial043.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2026-02-20T11:15:26.780 INFO:tasks.rados.rados.0.trial043.stdout:12: done (15 left) 2026-02-20T11:15:26.780 INFO:tasks.rados.rados.0.trial043.stdout:29: write initial oid 29 2026-02-20T11:15:26.781 INFO:tasks.rados.rados.0.trial043.stdout:29: seq_num 28 ranges {470596=456031,1572084=618660} 2026-02-20T11:15:26.783 INFO:tasks.rados.rados.0.trial043.stdout:29: writing trial04342147-29 from 470596 to 926627 tid 1 2026-02-20T11:15:26.788 INFO:tasks.rados.rados.0.trial043.stdout:29: writing trial04342147-29 from 1572084 to 2190744 tid 2 2026-02-20T11:15:26.789 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:15:26.789 INFO:tasks.rados.rados.0.trial043.stdout:7: finishing write tid 4 to trial04342147-7 2026-02-20T11:15:26.789 INFO:tasks.rados.rados.0.trial043.stdout:7: oid 7 updating version 0 to 4 2026-02-20T11:15:26.789 INFO:tasks.rados.rados.0.trial043.stdout:7: oid 7 updating version 4 to 6 2026-02-20T11:15:26.789 INFO:tasks.rados.rados.0.trial043.stdout:7: oid 7 version 6 is already newer than 6 2026-02-20T11:15:26.789 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 7 v 6 (ObjNum 6 snap 0 seq_num 6) dirty exists 2026-02-20T11:15:26.789 INFO:tasks.rados.rados.0.trial043.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2026-02-20T11:15:26.789 INFO:tasks.rados.rados.0.trial043.stdout:7: done (15 left) 2026-02-20T11:15:26.789 INFO:tasks.rados.rados.0.trial043.stdout:30: write initial oid 30 2026-02-20T11:15:26.790 INFO:tasks.rados.rados.0.trial043.stdout:30: seq_num 29 ranges {487403=531280,1514213=478671} 2026-02-20T11:15:26.792 INFO:tasks.rados.rados.0.trial043.stdout:30: writing trial04342147-30 from 487403 to 1018683 tid 1 2026-02-20T11:15:26.797 INFO:tasks.rados.rados.0.trial043.stdout:30: writing trial04342147-30 from 1514213 to 1992884 tid 2 2026-02-20T11:15:26.797 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:15:26.798 INFO:tasks.rados.rados.0.trial043.stdout:15: finishing write tid 3 to trial04342147-15 2026-02-20T11:15:26.798 INFO:tasks.rados.rados.0.trial043.stdout:21: finishing write tid 3 to trial04342147-21 2026-02-20T11:15:26.798 INFO:tasks.rados.rados.0.trial043.stdout:15: finishing write tid 4 to trial04342147-15 2026-02-20T11:15:26.798 INFO:tasks.rados.rados.0.trial043.stdout:15: oid 15 updating version 0 to 3 2026-02-20T11:15:26.798 INFO:tasks.rados.rados.0.trial043.stdout:15: oid 15 version 3 is already newer than 3 2026-02-20T11:15:26.798 INFO:tasks.rados.rados.0.trial043.stdout:15: oid 15 version 3 is already newer than 1 2026-02-20T11:15:26.798 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 15 v 3 (ObjNum 14 snap 0 seq_num 14) dirty exists 2026-02-20T11:15:26.798 INFO:tasks.rados.rados.0.trial043.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2026-02-20T11:15:26.798 INFO:tasks.rados.rados.0.trial043.stdout:21: finishing write tid 4 to trial04342147-21 2026-02-20T11:15:26.799 INFO:tasks.rados.rados.0.trial043.stdout:21: oid 21 updating version 0 to 4 2026-02-20T11:15:26.799 INFO:tasks.rados.rados.0.trial043.stdout:21: oid 21 updating version 4 to 6 2026-02-20T11:15:26.799 INFO:tasks.rados.rados.0.trial043.stdout:21: oid 21 version 6 is already newer than 6 2026-02-20T11:15:26.799 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 21 v 6 (ObjNum 20 snap 0 seq_num 20) dirty exists 2026-02-20T11:15:26.799 INFO:tasks.rados.rados.0.trial043.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2026-02-20T11:15:26.799 INFO:tasks.rados.rados.0.trial043.stdout:15: done (15 left) 2026-02-20T11:15:26.799 INFO:tasks.rados.rados.0.trial043.stdout:21: done (14 left) 2026-02-20T11:15:26.799 INFO:tasks.rados.rados.0.trial043.stdout:31: write initial oid 31 2026-02-20T11:15:26.799 INFO:tasks.rados.rados.0.trial043.stdout:31: seq_num 30 ranges {504210=606529,1572695=622329} 2026-02-20T11:15:26.800 INFO:tasks.rados.rados.0.trial043.stdout:31: writing trial04342147-31 from 504210 to 1110739 tid 1 2026-02-20T11:15:26.804 INFO:tasks.rados.rados.0.trial043.stdout:31: writing trial04342147-31 from 1572695 to 2195024 tid 2 2026-02-20T11:15:26.805 INFO:tasks.rados.rados.0.trial043.stdout:32: write initial oid 32 2026-02-20T11:15:26.805 INFO:tasks.rados.rados.0.trial043.stdout:32: seq_num 31 ranges {521017=798131,1747530=482340} 2026-02-20T11:15:26.809 INFO:tasks.rados.rados.0.trial043.stdout:32: writing trial04342147-32 from 521017 to 1319148 tid 1 2026-02-20T11:15:26.814 INFO:tasks.rados.rados.0.trial043.stdout:32: writing trial04342147-32 from 1747530 to 2229870 tid 2 2026-02-20T11:15:26.814 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:15:26.814 INFO:tasks.rados.rados.0.trial043.stdout:24: finishing write tid 1 to trial04342147-24 2026-02-20T11:15:26.814 INFO:tasks.rados.rados.0.trial043.stdout:10: finishing write tid 1 to trial04342147-10 2026-02-20T11:15:26.814 INFO:tasks.rados.rados.0.trial043.stdout:30: finishing write tid 1 to trial04342147-30 2026-02-20T11:15:26.815 INFO:tasks.rados.rados.0.trial043.stdout:8: finishing write tid 2 to trial04342147-8 2026-02-20T11:15:26.820 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:26.818+0000 7fded3c1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:26.821 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:26.817+0000 7f780721a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:26.823 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:26.823+0000 7fded3c1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:26.824 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:26.824+0000 7f780721a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:26.831 INFO:tasks.rados.rados.0.trial043.stdout:20: finishing write tid 1 to trial04342147-20 2026-02-20T11:15:26.831 INFO:tasks.rados.rados.0.trial043.stdout:8: finishing write tid 3 to trial04342147-8 2026-02-20T11:15:26.833 INFO:tasks.rados.rados.0.trial043.stdout:8: finishing write tid 4 to trial04342147-8 2026-02-20T11:15:26.834 INFO:tasks.rados.rados.0.trial043.stdout:8: oid 8 updating version 0 to 6 2026-02-20T11:15:26.834 INFO:tasks.rados.rados.0.trial043.stdout:8: oid 8 version 6 is already newer than 4 2026-02-20T11:15:26.834 INFO:tasks.rados.rados.0.trial043.stdout:8: oid 8 version 6 is already newer than 6 2026-02-20T11:15:26.834 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 8 v 6 (ObjNum 7 snap 0 seq_num 7) dirty exists 2026-02-20T11:15:26.834 INFO:tasks.rados.rados.0.trial043.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2026-02-20T11:15:26.834 INFO:tasks.rados.rados.0.trial043.stdout:8: done (15 left) 2026-02-20T11:15:26.834 INFO:tasks.rados.rados.0.trial043.stdout:33: write initial oid 33 2026-02-20T11:15:26.834 INFO:tasks.rados.rados.0.trial043.stdout:33: seq_num 32 ranges {537824=473380,1806012=625998} 2026-02-20T11:15:26.837 INFO:tasks.rados.rados.0.trial043.stdout:33: writing trial04342147-33 from 537824 to 1011204 tid 1 2026-02-20T11:15:26.843 INFO:tasks.rados.rados.0.trial043.stdout:33: writing trial04342147-33 from 1806012 to 2432010 tid 2 2026-02-20T11:15:26.843 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:15:26.843 INFO:tasks.rados.rados.0.trial043.stdout:32: finishing write tid 1 to trial04342147-32 2026-02-20T11:15:26.843 INFO:tasks.rados.rados.0.trial043.stdout:30: finishing write tid 2 to trial04342147-30 2026-02-20T11:15:26.845 INFO:tasks.rados.rados.0.trial043.stdout:10: finishing write tid 2 to trial04342147-10 2026-02-20T11:15:26.846 INFO:tasks.rados.rados.0.trial043.stdout:31: finishing write tid 1 to trial04342147-31 2026-02-20T11:15:26.847 INFO:tasks.rados.rados.0.trial043.stdout:17: finishing write tid 2 to trial04342147-17 2026-02-20T11:15:26.856 INFO:tasks.rados.rados.0.trial043.stdout:30: finishing write tid 3 to trial04342147-30 2026-02-20T11:15:26.858 INFO:tasks.rados.rados.0.trial043.stdout:30: finishing write tid 4 to trial04342147-30 2026-02-20T11:15:26.858 INFO:tasks.rados.rados.0.trial043.stdout:30: oid 30 updating version 0 to 4 2026-02-20T11:15:26.859 INFO:tasks.rados.rados.0.trial043.stdout:30: oid 30 updating version 4 to 6 2026-02-20T11:15:26.859 INFO:tasks.rados.rados.0.trial043.stdout:30: oid 30 version 6 is already newer than 6 2026-02-20T11:15:26.859 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 30 v 6 (ObjNum 29 snap 0 seq_num 29) dirty exists 2026-02-20T11:15:26.859 INFO:tasks.rados.rados.0.trial043.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2026-02-20T11:15:26.859 INFO:tasks.rados.rados.0.trial043.stdout:30: done (15 left) 2026-02-20T11:15:26.859 INFO:tasks.rados.rados.0.trial043.stdout:34: write initial oid 34 2026-02-20T11:15:26.859 INFO:tasks.rados.rados.0.trial043.stdout:34: seq_num 33 ranges {554631=548629,1748141=486009} 2026-02-20T11:15:26.862 INFO:tasks.rados.rados.0.trial043.stdout:34: writing trial04342147-34 from 554631 to 1103260 tid 1 2026-02-20T11:15:26.866 INFO:tasks.rados.rados.0.trial043.stdout:34: writing trial04342147-34 from 1748141 to 2234150 tid 2 2026-02-20T11:15:26.867 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:15:26.867 INFO:tasks.rados.rados.0.trial043.stdout:32: finishing write tid 2 to trial04342147-32 2026-02-20T11:15:26.867 INFO:tasks.rados.rados.0.trial043.stdout:10: finishing write tid 3 to trial04342147-10 2026-02-20T11:15:26.867 INFO:tasks.rados.rados.0.trial043.stdout:17: finishing write tid 3 to trial04342147-17 2026-02-20T11:15:26.870 INFO:tasks.rados.rados.0.trial043.stdout:10: finishing write tid 4 to trial04342147-10 2026-02-20T11:15:26.870 INFO:tasks.rados.rados.0.trial043.stdout:10: oid 10 updating version 0 to 7 2026-02-20T11:15:26.870 INFO:tasks.rados.rados.0.trial043.stdout:10: oid 10 updating version 7 to 9 2026-02-20T11:15:26.870 INFO:tasks.rados.rados.0.trial043.stdout:10: oid 10 version 9 is already newer than 9 2026-02-20T11:15:26.871 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 10 v 9 (ObjNum 9 snap 0 seq_num 9) dirty exists 2026-02-20T11:15:26.871 INFO:tasks.rados.rados.0.trial043.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2026-02-20T11:15:26.871 INFO:tasks.rados.rados.0.trial043.stdout:10: done (15 left) 2026-02-20T11:15:26.871 INFO:tasks.rados.rados.0.trial043.stdout:35: write initial oid 35 2026-02-20T11:15:26.871 INFO:tasks.rados.rados.0.trial043.stdout:35: seq_num 34 ranges {571438=623878,1806623=629667} 2026-02-20T11:15:26.874 INFO:tasks.rados.rados.0.trial043.stdout:35: writing trial04342147-35 from 571438 to 1195316 tid 1 2026-02-20T11:15:26.880 INFO:tasks.rados.rados.0.trial043.stdout:35: writing trial04342147-35 from 1806623 to 2436290 tid 2 2026-02-20T11:15:26.880 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:15:26.880 INFO:tasks.rados.rados.0.trial043.stdout:32: finishing write tid 3 to trial04342147-32 2026-02-20T11:15:26.880 INFO:tasks.rados.rados.0.trial043.stdout:17: finishing write tid 4 to trial04342147-17 2026-02-20T11:15:26.880 INFO:tasks.rados.rados.0.trial043.stdout:17: oid 17 updating version 0 to 6 2026-02-20T11:15:26.880 INFO:tasks.rados.rados.0.trial043.stdout:17: oid 17 version 6 is already newer than 6 2026-02-20T11:15:26.880 INFO:tasks.rados.rados.0.trial043.stdout:17: oid 17 version 6 is already newer than 4 2026-02-20T11:15:26.881 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 17 v 6 (ObjNum 16 snap 0 seq_num 16) dirty exists 2026-02-20T11:15:26.881 INFO:tasks.rados.rados.0.trial043.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2026-02-20T11:15:26.881 INFO:tasks.rados.rados.0.trial043.stdout:32: finishing write tid 4 to trial04342147-32 2026-02-20T11:15:26.881 INFO:tasks.rados.rados.0.trial043.stdout:32: oid 32 updating version 0 to 1 2026-02-20T11:15:26.881 INFO:tasks.rados.rados.0.trial043.stdout:32: oid 32 updating version 1 to 3 2026-02-20T11:15:26.881 INFO:tasks.rados.rados.0.trial043.stdout:32: oid 32 version 3 is already newer than 3 2026-02-20T11:15:26.881 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 32 v 3 (ObjNum 31 snap 0 seq_num 31) dirty exists 2026-02-20T11:15:26.881 INFO:tasks.rados.rados.0.trial043.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2026-02-20T11:15:26.881 INFO:tasks.rados.rados.0.trial043.stdout:31: finishing write tid 2 to trial04342147-31 2026-02-20T11:15:26.881 INFO:tasks.rados.rados.0.trial043.stdout:17: done (15 left) 2026-02-20T11:15:26.881 INFO:tasks.rados.rados.0.trial043.stdout:32: done (14 left) 2026-02-20T11:15:26.881 INFO:tasks.rados.rados.0.trial043.stdout:36: write initial oid 36 2026-02-20T11:15:26.882 INFO:tasks.rados.rados.0.trial043.stdout:36: seq_num 35 ranges {588245=699127,1865105=489678} 2026-02-20T11:15:26.884 INFO:tasks.rados.rados.0.trial043.stdout:36: writing trial04342147-36 from 588245 to 1287372 tid 1 2026-02-20T11:15:26.888 INFO:tasks.rados.rados.0.trial043.stdout:36: writing trial04342147-36 from 1865105 to 2354783 tid 2 2026-02-20T11:15:26.889 INFO:tasks.rados.rados.0.trial043.stdout:37: write initial oid 37 2026-02-20T11:15:26.889 INFO:tasks.rados.rados.0.trial043.stdout:37: seq_num 36 ranges {605052=774376,1923587=633336} 2026-02-20T11:15:26.893 INFO:tasks.rados.rados.0.trial043.stdout:37: writing trial04342147-37 from 605052 to 1379428 tid 1 2026-02-20T11:15:26.898 INFO:tasks.rados.rados.0.trial043.stdout:37: writing trial04342147-37 from 1923587 to 2556923 tid 2 2026-02-20T11:15:26.898 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:15:26.898 INFO:tasks.rados.rados.0.trial043.stdout:31: finishing write tid 3 to trial04342147-31 2026-02-20T11:15:26.899 INFO:tasks.rados.rados.0.trial043.stdout:31: finishing write tid 4 to trial04342147-31 2026-02-20T11:15:26.899 INFO:tasks.rados.rados.0.trial043.stdout:31: oid 31 updating version 0 to 3 2026-02-20T11:15:26.899 INFO:tasks.rados.rados.0.trial043.stdout:31: oid 31 version 3 is already newer than 1 2026-02-20T11:15:26.899 INFO:tasks.rados.rados.0.trial043.stdout:31: oid 31 version 3 is already newer than 3 2026-02-20T11:15:26.899 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 31 v 3 (ObjNum 30 snap 0 seq_num 30) dirty exists 2026-02-20T11:15:26.899 INFO:tasks.rados.rados.0.trial043.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2026-02-20T11:15:26.899 INFO:tasks.rados.rados.0.trial043.stdout:31: done (15 left) 2026-02-20T11:15:26.899 INFO:tasks.rados.rados.0.trial043.stdout:38: write initial oid 38 2026-02-20T11:15:26.899 INFO:tasks.rados.rados.0.trial043.stdout:38: seq_num 37 ranges {621859=449625,1865716=756143} 2026-02-20T11:15:26.902 INFO:tasks.rados.rados.0.trial043.stdout:38: writing trial04342147-38 from 621859 to 1071484 tid 1 2026-02-20T11:15:26.908 INFO:tasks.rados.rados.0.trial043.stdout:38: writing trial04342147-38 from 1865716 to 2621859 tid 2 2026-02-20T11:15:26.908 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:15:26.909 INFO:tasks.rados.rados.0.trial043.stdout:20: finishing write tid 2 to trial04342147-20 2026-02-20T11:15:26.909 INFO:tasks.rados.rados.0.trial043.stdout:34: finishing write tid 1 to trial04342147-34 2026-02-20T11:15:26.909 INFO:tasks.rados.rados.0.trial043.stdout:22: finishing write tid 1 to trial04342147-22 2026-02-20T11:15:26.909 INFO:tasks.rados.rados.0.trial043.stdout:24: finishing write tid 2 to trial04342147-24 2026-02-20T11:15:26.911 INFO:tasks.rados.rados.0.trial043.stdout:34: finishing write tid 2 to trial04342147-34 2026-02-20T11:15:26.917 INFO:tasks.rados.rados.0.trial043.stdout:16: finishing write tid 3 to trial04342147-16 2026-02-20T11:15:26.921 INFO:tasks.rados.rados.0.trial043.stdout:36: finishing write tid 1 to trial04342147-36 2026-02-20T11:15:26.924 INFO:tasks.rados.rados.0.trial043.stdout:34: finishing write tid 3 to trial04342147-34 2026-02-20T11:15:26.924 INFO:tasks.rados.rados.0.trial043.stdout:16: finishing write tid 4 to trial04342147-16 2026-02-20T11:15:26.924 INFO:tasks.rados.rados.0.trial043.stdout:16: oid 16 updating version 0 to 4 2026-02-20T11:15:26.925 INFO:tasks.rados.rados.0.trial043.stdout:16: oid 16 updating version 4 to 6 2026-02-20T11:15:26.925 INFO:tasks.rados.rados.0.trial043.stdout:16: oid 16 version 6 is already newer than 6 2026-02-20T11:15:26.925 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 16 v 6 (ObjNum 15 snap 0 seq_num 15) dirty exists 2026-02-20T11:15:26.925 INFO:tasks.rados.rados.0.trial043.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2026-02-20T11:15:26.925 INFO:tasks.rados.rados.0.trial043.stdout:16: done (15 left) 2026-02-20T11:15:26.925 INFO:tasks.rados.rados.0.trial043.stdout:39: write initial oid 39 2026-02-20T11:15:26.925 INFO:tasks.rados.rados.0.trial043.stdout:39: seq_num 38 ranges {638666=524874,1924198=637005} 2026-02-20T11:15:26.928 INFO:tasks.rados.rados.0.trial043.stdout:39: writing trial04342147-39 from 638666 to 1163540 tid 1 2026-02-20T11:15:26.934 INFO:tasks.rados.rados.0.trial043.stdout:39: writing trial04342147-39 from 1924198 to 2561203 tid 2 2026-02-20T11:15:26.934 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:15:26.934 INFO:tasks.rados.rados.0.trial043.stdout:20: finishing write tid 3 to trial04342147-20 2026-02-20T11:15:26.934 INFO:tasks.rados.rados.0.trial043.stdout:20: finishing write tid 4 to trial04342147-20 2026-02-20T11:15:26.934 INFO:tasks.rados.rados.0.trial043.stdout:20: oid 20 updating version 0 to 7 2026-02-20T11:15:26.934 INFO:tasks.rados.rados.0.trial043.stdout:20: oid 20 updating version 7 to 9 2026-02-20T11:15:26.934 INFO:tasks.rados.rados.0.trial043.stdout:20: oid 20 version 9 is already newer than 9 2026-02-20T11:15:26.935 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 20 v 9 (ObjNum 19 snap 0 seq_num 19) dirty exists 2026-02-20T11:15:26.935 INFO:tasks.rados.rados.0.trial043.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2026-02-20T11:15:26.935 INFO:tasks.rados.rados.0.trial043.stdout:34: finishing write tid 4 to trial04342147-34 2026-02-20T11:15:26.935 INFO:tasks.rados.rados.0.trial043.stdout:34: oid 34 updating version 0 to 9 2026-02-20T11:15:26.935 INFO:tasks.rados.rados.0.trial043.stdout:34: oid 34 version 9 is already newer than 7 2026-02-20T11:15:26.935 INFO:tasks.rados.rados.0.trial043.stdout:34: oid 34 version 9 is already newer than 9 2026-02-20T11:15:26.935 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 34 v 9 (ObjNum 33 snap 0 seq_num 33) dirty exists 2026-02-20T11:15:26.935 INFO:tasks.rados.rados.0.trial043.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2026-02-20T11:15:26.936 INFO:tasks.rados.rados.0.trial043.stdout:20: done (15 left) 2026-02-20T11:15:26.936 INFO:tasks.rados.rados.0.trial043.stdout:34: done (14 left) 2026-02-20T11:15:26.936 INFO:tasks.rados.rados.0.trial043.stdout:40: write initial oid 40 2026-02-20T11:15:26.936 INFO:tasks.rados.rados.0.trial043.stdout:40: seq_num 39 ranges {655473=716476,2099033=556440} 2026-02-20T11:15:26.938 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:15:26.940 INFO:teuthology.orchestra.run.trial043.stderr:dumped all 2026-02-20T11:15:26.940 INFO:tasks.rados.rados.0.trial043.stdout:40: writing trial04342147-40 from 655473 to 1371949 tid 1 2026-02-20T11:15:26.945 INFO:tasks.rados.rados.0.trial043.stdout:40: writing trial04342147-40 from 2099033 to 2655473 tid 2 2026-02-20T11:15:26.945 INFO:tasks.rados.rados.0.trial043.stdout:41: write initial oid 41 2026-02-20T11:15:26.945 INFO:tasks.rados.rados.0.trial043.stdout:41: seq_num 40 ranges {672280=791725,2157515=514765} 2026-02-20T11:15:26.950 INFO:teuthology.orchestra.run.trial043.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2026-02-20T11:15:26.500322+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209715136,"kb_used":83890120,"kb_used_data":83890120,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":125825016,"statfs":{"total":214748299264,"available":128844816384,"internally_reserved":0,"allocated":85903482880,"data_stored":85903482880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.561720"},"pg_stats":[{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.981411+0000","last_change":"2026-02-20T11:15:16.984535+0000","last_active":"2026-02-20T11:15:16.981411+0000","last_peered":"2026-02-20T11:15:16.981411+0000","last_clean":"2026-02-20T11:15:16.981411+0000","last_became_active":"2026-02-20T11:15:15.006596+0000","last_became_peered":"2026-02-20T11:15:15.006596+0000","last_unstale":"2026-02-20T11:15:16.981411+0000","last_undegraded":"2026-02-20T11:15:16.981411+0000","last_fullsized":"2026-02-20T11:15:16.981411+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.983008+0000","last_change":"2026-02-20T11:15:16.985758+0000","last_active":"2026-02-20T11:15:16.983008+0000","last_peered":"2026-02-20T11:15:16.983008+0000","last_clean":"2026-02-20T11:15:16.983008+0000","last_became_active":"2026-02-20T11:15:15.019055+0000","last_became_peered":"2026-02-20T11:15:15.019055+0000","last_unstale":"2026-02-20T11:15:16.983008+0000","last_undegraded":"2026-02-20T11:15:16.983008+0000","last_fullsized":"2026-02-20T11:15:16.983008+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.2","version":"14'1","reported_seq":18,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.983646+0000","last_change":"2026-02-20T11:15:16.983646+0000","last_active":"2026-02-20T11:15:16.983646+0000","last_peered":"2026-02-20T11:15:16.983646+0000","last_clean":"2026-02-20T11:15:16.983646+0000","last_became_active":"2026-02-20T11:15:15.012729+0000","last_became_peered":"2026-02-20T11:15:15.012729+0000","last_unstale":"2026-02-20T11:15:16.983646+0000","last_undegraded":"2026-02-20T11:15:16.983646+0000","last_fullsized":"2026-02-20T11:15:16.983646+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.980370+0000","last_change":"2026-02-20T11:15:16.983291+0000","last_active":"2026-02-20T11:15:16.980370+0000","last_peered":"2026-02-20T11:15:16.980370+0000","last_clean":"2026-02-20T11:15:16.980370+0000","last_became_active":"2026-02-20T11:15:15.009983+0000","last_became_peered":"2026-02-20T11:15:15.009983+0000","last_unstale":"2026-02-20T11:15:16.980370+0000","last_undegraded":"2026-02-20T11:15:16.980370+0000","last_fullsized":"2026-02-20T11:15:16.980370+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'52","reported_seq":82,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.980870+0000","last_change":"2026-02-20T11:15:15.035494+0000","last_active":"2026-02-20T11:15:16.980870+0000","last_peered":"2026-02-20T11:15:16.980870+0000","last_clean":"2026-02-20T11:15:16.980870+0000","last_became_active":"2026-02-20T11:15:15.012645+0000","last_became_peered":"2026-02-20T11:15:15.012645+0000","last_unstale":"2026-02-20T11:15:16.980870+0000","last_undegraded":"2026-02-20T11:15:16.980870+0000","last_fullsized":"2026-02-20T11:15:16.980870+0000","mapping_epoch":12,"log_start":"13'48","ondisk_log_start":"13'48","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":48,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.982312+0000","last_change":"2026-02-20T11:15:16.984208+0000","last_active":"2026-02-20T11:15:16.982312+0000","last_peered":"2026-02-20T11:15:16.982312+0000","last_clean":"2026-02-20T11:15:16.982312+0000","last_became_active":"2026-02-20T11:15:15.027750+0000","last_became_peered":"2026-02-20T11:15:15.027750+0000","last_unstale":"2026-02-20T11:15:16.982312+0000","last_undegraded":"2026-02-20T11:15:16.982312+0000","last_fullsized":"2026-02-20T11:15:16.982312+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.983812+0000","last_change":"2026-02-20T11:15:16.985671+0000","last_active":"2026-02-20T11:15:16.983812+0000","last_peered":"2026-02-20T11:15:16.983812+0000","last_clean":"2026-02-20T11:15:16.983812+0000","last_became_active":"2026-02-20T11:15:15.003549+0000","last_became_peered":"2026-02-20T11:15:15.003549+0000","last_unstale":"2026-02-20T11:15:16.983812+0000","last_undegraded":"2026-02-20T11:15:16.983812+0000","last_fullsized":"2026-02-20T11:15:16.983812+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.982620+0000","last_change":"2026-02-20T11:15:16.986873+0000","last_active":"2026-02-20T11:15:16.982620+0000","last_peered":"2026-02-20T11:15:16.982620+0000","last_clean":"2026-02-20T11:15:16.982620+0000","last_became_active":"2026-02-20T11:15:15.018131+0000","last_became_peered":"2026-02-20T11:15:15.018131+0000","last_unstale":"2026-02-20T11:15:16.982620+0000","last_undegraded":"2026-02-20T11:15:16.982620+0000","last_fullsized":"2026-02-20T11:15:16.982620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.983722+0000","last_change":"2026-02-20T11:15:16.986570+0000","last_active":"2026-02-20T11:15:16.983722+0000","last_peered":"2026-02-20T11:15:16.983722+0000","last_clean":"2026-02-20T11:15:16.983722+0000","last_became_active":"2026-02-20T11:15:15.057391+0000","last_became_peered":"2026-02-20T11:15:15.057391+0000","last_unstale":"2026-02-20T11:15:16.983722+0000","last_undegraded":"2026-02-20T11:15:16.983722+0000","last_fullsized":"2026-02-20T11:15:16.983722+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32210939904,"internally_reserved":0,"allocated":21476134912,"data_stored":21476134912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96633778176,"internally_reserved":0,"allocated":64427446272,"data_stored":64427446272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":24,"acting":24,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640265,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972788,"kb_used_data":20972788,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31455996,"statfs":{"total":53687074816,"available":32210939904,"internally_reserved":0,"allocated":21476134912,"data_stored":21476134912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640266,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20971984,"kb_used_data":20971984,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456800,"statfs":{"total":53687074816,"available":32211763200,"internally_reserved":0,"allocated":21475311616,"data_stored":21475311616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972656,"kb_used_data":20972656,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456128,"statfs":{"total":53687074816,"available":32211075072,"internally_reserved":0,"allocated":21475999744,"data_stored":21475999744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640266,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972692,"kb_used_data":20972692,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456092,"statfs":{"total":53687074816,"available":32211038208,"internally_reserved":0,"allocated":21476036608,"data_stored":21476036608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32211075072,"internally_reserved":0,"allocated":21475999744,"data_stored":21475999744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53687074816,"available":32211763200,"internally_reserved":0,"allocated":21475311616,"data_stored":21475311616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":53687074816,"available":32210939904,"internally_reserved":0,"allocated":21476134912,"data_stored":21476134912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53687074816,"available":32210939904,"internally_reserved":0,"allocated":21476134912,"data_stored":21476134912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-20T11:15:26.954 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-20T11:15:26.955 INFO:tasks.rados.rados.0.trial043.stdout:41: writing trial04342147-41 from 672280 to 1464005 tid 1 2026-02-20T11:15:26.957 INFO:tasks.rados.rados.0.trial043.stdout:41: writing trial04342147-41 from 2157515 to 2672280 tid 2 2026-02-20T11:15:26.957 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:15:26.957 INFO:tasks.rados.rados.0.trial043.stdout:38: finishing write tid 1 to trial04342147-38 2026-02-20T11:15:26.957 INFO:tasks.rados.rados.0.trial043.stdout:36: finishing write tid 2 to trial04342147-36 2026-02-20T11:15:26.957 INFO:tasks.rados.rados.0.trial043.stdout:23: finishing write tid 1 to trial04342147-23 2026-02-20T11:15:26.958 INFO:tasks.rados.rados.0.trial043.stdout:22: finishing write tid 2 to trial04342147-22 2026-02-20T11:15:26.958 INFO:tasks.rados.rados.0.trial043.stdout:36: finishing write tid 3 to trial04342147-36 2026-02-20T11:15:26.958 INFO:tasks.rados.rados.0.trial043.stdout:36: finishing write tid 4 to trial04342147-36 2026-02-20T11:15:26.958 INFO:tasks.rados.rados.0.trial043.stdout:36: oid 36 updating version 0 to 3 2026-02-20T11:15:26.958 INFO:tasks.rados.rados.0.trial043.stdout:36: oid 36 version 3 is already newer than 1 2026-02-20T11:15:26.958 INFO:tasks.rados.rados.0.trial043.stdout:36: oid 36 version 3 is already newer than 3 2026-02-20T11:15:26.958 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 36 v 3 (ObjNum 35 snap 0 seq_num 35) dirty exists 2026-02-20T11:15:26.958 INFO:tasks.rados.rados.0.trial043.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2026-02-20T11:15:26.958 INFO:tasks.rados.rados.0.trial043.stdout:39: finishing write tid 1 to trial04342147-39 2026-02-20T11:15:26.958 INFO:tasks.rados.rados.0.trial043.stdout:36: done (15 left) 2026-02-20T11:15:26.958 INFO:tasks.rados.rados.0.trial043.stdout:42: write initial oid 42 2026-02-20T11:15:26.959 INFO:tasks.rados.rados.0.trial043.stdout:42: seq_num 41 ranges {689087=466974,1699644=784332} 2026-02-20T11:15:26.959 INFO:tasks.rados.rados.0.trial043.stdout:42: writing trial04342147-42 from 689087 to 1156061 tid 1 2026-02-20T11:15:26.965 INFO:tasks.rados.rados.0.trial043.stdout:42: writing trial04342147-42 from 1699644 to 2483976 tid 2 2026-02-20T11:15:26.965 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:15:26.966 INFO:tasks.rados.rados.0.trial043.stdout:22: finishing write tid 3 to trial04342147-22 2026-02-20T11:15:26.966 INFO:tasks.rados.rados.0.trial043.stdout:22: finishing write tid 4 to trial04342147-22 2026-02-20T11:15:26.966 INFO:tasks.rados.rados.0.trial043.stdout:22: oid 22 updating version 0 to 7 2026-02-20T11:15:26.966 INFO:tasks.rados.rados.0.trial043.stdout:22: oid 22 updating version 7 to 9 2026-02-20T11:15:26.966 INFO:tasks.rados.rados.0.trial043.stdout:22: oid 22 version 9 is already newer than 9 2026-02-20T11:15:26.966 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 22 v 9 (ObjNum 21 snap 0 seq_num 21) dirty exists 2026-02-20T11:15:26.966 INFO:tasks.rados.rados.0.trial043.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2026-02-20T11:15:26.966 INFO:tasks.rados.rados.0.trial043.stdout:38: finishing write tid 2 to trial04342147-38 2026-02-20T11:15:26.966 INFO:tasks.rados.rados.0.trial043.stdout:22: done (15 left) 2026-02-20T11:15:26.966 INFO:tasks.rados.rados.0.trial043.stdout:43: write initial oid 43 2026-02-20T11:15:26.967 INFO:tasks.rados.rados.0.trial043.stdout:43: seq_num 42 ranges {705894=542223,1758126=644343} 2026-02-20T11:15:26.969 INFO:tasks.rados.rados.0.trial043.stdout:43: writing trial04342147-43 from 705894 to 1248117 tid 1 2026-02-20T11:15:26.974 INFO:tasks.rados.rados.0.trial043.stdout:43: writing trial04342147-43 from 1758126 to 2402469 tid 2 2026-02-20T11:15:26.974 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:15:26.974 INFO:tasks.rados.rados.0.trial043.stdout:40: finishing write tid 1 to trial04342147-40 2026-02-20T11:15:26.974 INFO:tasks.rados.rados.0.trial043.stdout:26: finishing write tid 1 to trial04342147-26 2026-02-20T11:15:26.974 INFO:tasks.rados.rados.0.trial043.stdout:39: finishing write tid 2 to trial04342147-39 2026-02-20T11:15:26.975 INFO:tasks.rados.rados.0.trial043.stdout:25: finishing write tid 2 to trial04342147-25 2026-02-20T11:15:26.981 INFO:tasks.rados.rados.0.trial043.stdout:39: finishing write tid 3 to trial04342147-39 2026-02-20T11:15:26.983 INFO:tasks.rados.rados.0.trial043.stdout:39: finishing write tid 4 to trial04342147-39 2026-02-20T11:15:26.983 INFO:tasks.rados.rados.0.trial043.stdout:39: oid 39 updating version 0 to 6 2026-02-20T11:15:26.983 INFO:tasks.rados.rados.0.trial043.stdout:39: oid 39 version 6 is already newer than 6 2026-02-20T11:15:26.983 INFO:tasks.rados.rados.0.trial043.stdout:39: oid 39 version 6 is already newer than 4 2026-02-20T11:15:26.983 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 39 v 6 (ObjNum 38 snap 0 seq_num 38) dirty exists 2026-02-20T11:15:26.983 INFO:tasks.rados.rados.0.trial043.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2026-02-20T11:15:26.983 INFO:tasks.rados.rados.0.trial043.stdout:39: done (15 left) 2026-02-20T11:15:26.983 INFO:tasks.rados.rados.0.trial043.stdout:44: write initial oid 44 2026-02-20T11:15:26.984 INFO:tasks.rados.rados.0.trial043.stdout:44: seq_num 43 ranges {722701=617472,1816608=788001} 2026-02-20T11:15:26.987 INFO:tasks.rados.rados.0.trial043.stdout:44: writing trial04342147-44 from 722701 to 1340173 tid 1 2026-02-20T11:15:26.993 INFO:tasks.rados.rados.0.trial043.stdout:44: writing trial04342147-44 from 1816608 to 2604609 tid 2 2026-02-20T11:15:26.993 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:15:26.993 INFO:tasks.rados.rados.0.trial043.stdout:29: finishing write tid 1 to trial04342147-29 2026-02-20T11:15:26.993 INFO:tasks.rados.rados.0.trial043.stdout:23: finishing write tid 2 to trial04342147-23 2026-02-20T11:15:26.995 INFO:tasks.rados.rados.0.trial043.stdout:38: finishing write tid 3 to trial04342147-38 2026-02-20T11:15:26.997 INFO:tasks.rados.rados.0.trial043.stdout:38: finishing write tid 4 to trial04342147-38 2026-02-20T11:15:26.998 INFO:tasks.rados.rados.0.trial043.stdout:38: oid 38 updating version 0 to 6 2026-02-20T11:15:26.998 INFO:tasks.rados.rados.0.trial043.stdout:38: oid 38 version 6 is already newer than 4 2026-02-20T11:15:26.998 INFO:tasks.rados.rados.0.trial043.stdout:38: oid 38 version 6 is already newer than 6 2026-02-20T11:15:26.998 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 38 v 6 (ObjNum 37 snap 0 seq_num 37) dirty exists 2026-02-20T11:15:26.998 INFO:tasks.rados.rados.0.trial043.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2026-02-20T11:15:26.998 INFO:tasks.rados.rados.0.trial043.stdout:38: done (15 left) 2026-02-20T11:15:26.998 INFO:tasks.rados.rados.0.trial043.stdout:45: write initial oid 45 2026-02-20T11:15:26.998 INFO:tasks.rados.rados.0.trial043.stdout:45: seq_num 44 ranges {739508=692721,1875090=648012} 2026-02-20T11:15:27.002 INFO:tasks.rados.rados.0.trial043.stdout:45: writing trial04342147-45 from 739508 to 1432229 tid 1 2026-02-20T11:15:27.007 INFO:tasks.rados.rados.0.trial043.stdout:45: writing trial04342147-45 from 1875090 to 2523102 tid 2 2026-02-20T11:15:27.007 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:15:27.007 INFO:tasks.rados.rados.0.trial043.stdout:40: finishing write tid 2 to trial04342147-40 2026-02-20T11:15:27.011 INFO:tasks.rados.rados.0.trial043.stdout:26: finishing write tid 2 to trial04342147-26 2026-02-20T11:15:27.011 INFO:tasks.rados.rados.0.trial043.stdout:25: finishing write tid 3 to trial04342147-25 2026-02-20T11:15:27.014 INFO:tasks.rados.rados.0.trial043.stdout:40: finishing write tid 3 to trial04342147-40 2026-02-20T11:15:27.017 INFO:tasks.rados.rados.0.trial043.stdout:40: finishing write tid 4 to trial04342147-40 2026-02-20T11:15:27.018 INFO:tasks.rados.rados.0.trial043.stdout:40: oid 40 updating version 0 to 6 2026-02-20T11:15:27.018 INFO:tasks.rados.rados.0.trial043.stdout:40: oid 40 version 6 is already newer than 4 2026-02-20T11:15:27.018 INFO:tasks.rados.rados.0.trial043.stdout:40: oid 40 version 6 is already newer than 6 2026-02-20T11:15:27.018 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 40 v 6 (ObjNum 39 snap 0 seq_num 39) dirty exists 2026-02-20T11:15:27.018 INFO:tasks.rados.rados.0.trial043.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2026-02-20T11:15:27.018 INFO:tasks.rados.rados.0.trial043.stdout:40: done (15 left) 2026-02-20T11:15:27.018 INFO:tasks.rados.rados.0.trial043.stdout:46: write initial oid 46 2026-02-20T11:15:27.018 INFO:tasks.rados.rados.0.trial043.stdout:46: seq_num 45 ranges {756315=767970,1933572=791670} 2026-02-20T11:15:27.023 INFO:tasks.rados.rados.0.trial043.stdout:46: writing trial04342147-46 from 756315 to 1524285 tid 1 2026-02-20T11:15:27.029 INFO:tasks.rados.rados.0.trial043.stdout:46: writing trial04342147-46 from 1933572 to 2725242 tid 2 2026-02-20T11:15:27.029 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:15:27.029 INFO:tasks.rados.rados.0.trial043.stdout:25: finishing write tid 4 to trial04342147-25 2026-02-20T11:15:27.029 INFO:tasks.rados.rados.0.trial043.stdout:25: oid 25 updating version 0 to 6 2026-02-20T11:15:27.029 INFO:tasks.rados.rados.0.trial043.stdout:25: oid 25 version 6 is already newer than 6 2026-02-20T11:15:27.029 INFO:tasks.rados.rados.0.trial043.stdout:25: oid 25 version 6 is already newer than 4 2026-02-20T11:15:27.029 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 25 v 6 (ObjNum 24 snap 0 seq_num 24) dirty exists 2026-02-20T11:15:27.030 INFO:tasks.rados.rados.0.trial043.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2026-02-20T11:15:27.030 INFO:tasks.rados.rados.0.trial043.stdout:25: done (15 left) 2026-02-20T11:15:27.030 INFO:tasks.rados.rados.0.trial043.stdout:47: write initial oid 47 2026-02-20T11:15:27.030 INFO:tasks.rados.rados.0.trial043.stdout:47: seq_num 46 ranges {773122=559572,1992054=651681} 2026-02-20T11:15:27.034 INFO:tasks.rados.rados.0.trial043.stdout:47: writing trial04342147-47 from 773122 to 1332694 tid 1 2026-02-20T11:15:27.039 INFO:tasks.rados.rados.0.trial043.stdout:47: writing trial04342147-47 from 1992054 to 2643735 tid 2 2026-02-20T11:15:27.040 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:15:27.040 INFO:tasks.rados.rados.0.trial043.stdout:29: finishing write tid 2 to trial04342147-29 2026-02-20T11:15:27.040 INFO:tasks.rados.rados.0.trial043.stdout:42: finishing write tid 1 to trial04342147-42 2026-02-20T11:15:27.040 INFO:tasks.rados.rados.0.trial043.stdout:41: finishing write tid 1 to trial04342147-41 2026-02-20T11:15:27.040 INFO:tasks.rados.rados.0.trial043.stdout:35: finishing write tid 1 to trial04342147-35 2026-02-20T11:15:27.041 INFO:tasks.rados.rados.0.trial043.stdout:46: finishing write tid 1 to trial04342147-46 2026-02-20T11:15:27.042 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:27.039+0000 7fd99ee06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:27.045 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:27.044+0000 7fd99ee06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:27.046 INFO:tasks.rados.rados.0.trial043.stdout:29: finishing write tid 3 to trial04342147-29 2026-02-20T11:15:27.050 INFO:tasks.rados.rados.0.trial043.stdout:23: finishing write tid 3 to trial04342147-23 2026-02-20T11:15:27.050 INFO:tasks.rados.rados.0.trial043.stdout:41: finishing write tid 2 to trial04342147-41 2026-02-20T11:15:27.052 INFO:tasks.rados.rados.0.trial043.stdout:29: finishing write tid 4 to trial04342147-29 2026-02-20T11:15:27.052 INFO:tasks.rados.rados.0.trial043.stdout:29: oid 29 updating version 0 to 9 2026-02-20T11:15:27.052 INFO:tasks.rados.rados.0.trial043.stdout:29: oid 29 version 9 is already newer than 7 2026-02-20T11:15:27.052 INFO:tasks.rados.rados.0.trial043.stdout:29: oid 29 version 9 is already newer than 9 2026-02-20T11:15:27.052 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 29 v 9 (ObjNum 28 snap 0 seq_num 28) dirty exists 2026-02-20T11:15:27.053 INFO:tasks.rados.rados.0.trial043.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2026-02-20T11:15:27.053 INFO:tasks.rados.rados.0.trial043.stdout:29: done (15 left) 2026-02-20T11:15:27.053 INFO:tasks.rados.rados.0.trial043.stdout:48: write initial oid 48 2026-02-20T11:15:27.053 INFO:tasks.rados.rados.0.trial043.stdout:48: seq_num 47 ranges {789929=634821,2050536=739393} 2026-02-20T11:15:27.057 INFO:tasks.rados.rados.0.trial043.stdout:48: writing trial04342147-48 from 789929 to 1424750 tid 1 2026-02-20T11:15:27.063 INFO:tasks.rados.rados.0.trial043.stdout:48: writing trial04342147-48 from 2050536 to 2789929 tid 2 2026-02-20T11:15:27.063 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:15:27.064 INFO:tasks.rados.rados.0.trial043.stdout:46: finishing write tid 2 to trial04342147-46 2026-02-20T11:15:27.064 INFO:tasks.rados.rados.0.trial043.stdout:23: finishing write tid 4 to trial04342147-23 2026-02-20T11:15:27.064 INFO:tasks.rados.rados.0.trial043.stdout:23: oid 23 updating version 0 to 12 2026-02-20T11:15:27.064 INFO:tasks.rados.rados.0.trial043.stdout:23: oid 23 version 12 is already newer than 10 2026-02-20T11:15:27.064 INFO:tasks.rados.rados.0.trial043.stdout:23: oid 23 version 12 is already newer than 12 2026-02-20T11:15:27.064 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 23 v 12 (ObjNum 22 snap 0 seq_num 22) dirty exists 2026-02-20T11:15:27.064 INFO:tasks.rados.rados.0.trial043.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2026-02-20T11:15:27.065 INFO:tasks.rados.rados.0.trial043.stdout:46: finishing write tid 3 to trial04342147-46 2026-02-20T11:15:27.065 INFO:tasks.rados.rados.0.trial043.stdout:23: done (15 left) 2026-02-20T11:15:27.065 INFO:tasks.rados.rados.0.trial043.stdout:49: write initial oid 49 2026-02-20T11:15:27.065 INFO:tasks.rados.rados.0.trial043.stdout:49: seq_num 48 ranges {406736=710070,1709018=655350} 2026-02-20T11:15:27.066 INFO:tasks.rados.rados.0.trial043.stdout:49: writing trial04342147-49 from 406736 to 1116806 tid 1 2026-02-20T11:15:27.072 INFO:tasks.rados.rados.0.trial043.stdout:49: writing trial04342147-49 from 1709018 to 2364368 tid 2 2026-02-20T11:15:27.072 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:15:27.072 INFO:tasks.rados.rados.0.trial043.stdout:42: finishing write tid 2 to trial04342147-42 2026-02-20T11:15:27.072 INFO:tasks.rados.rados.0.trial043.stdout:46: finishing write tid 4 to trial04342147-46 2026-02-20T11:15:27.072 INFO:tasks.rados.rados.0.trial043.stdout:46: oid 46 updating version 0 to 1 2026-02-20T11:15:27.072 INFO:tasks.rados.rados.0.trial043.stdout:46: oid 46 updating version 1 to 3 2026-02-20T11:15:27.072 INFO:tasks.rados.rados.0.trial043.stdout:46: oid 46 version 3 is already newer than 3 2026-02-20T11:15:27.072 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 46 v 3 (ObjNum 45 snap 0 seq_num 45) dirty exists 2026-02-20T11:15:27.072 INFO:tasks.rados.rados.0.trial043.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2026-02-20T11:15:27.072 INFO:tasks.rados.rados.0.trial043.stdout:46: done (15 left) 2026-02-20T11:15:27.073 INFO:tasks.rados.rados.0.trial043.stdout:50: write initial oid 50 2026-02-20T11:15:27.073 INFO:tasks.rados.rados.0.trial043.stdout:50: seq_num 49 ranges {423543=785319,1767500=799008} 2026-02-20T11:15:27.076 INFO:tasks.rados.rados.0.trial043.stdout:50: writing trial04342147-50 from 423543 to 1208862 tid 1 2026-02-20T11:15:27.082 INFO:tasks.rados.rados.0.trial043.stdout:50: writing trial04342147-50 from 1767500 to 2566508 tid 2 2026-02-20T11:15:27.082 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:15:27.082 INFO:tasks.rados.rados.0.trial043.stdout:33: finishing write tid 1 to trial04342147-33 2026-02-20T11:15:27.082 INFO:tasks.rados.rados.0.trial043.stdout:35: finishing write tid 2 to trial04342147-35 2026-02-20T11:15:27.082 INFO:tasks.rados.rados.0.trial043.stdout:41: finishing write tid 3 to trial04342147-41 2026-02-20T11:15:27.083 INFO:tasks.rados.rados.0.trial043.stdout:41: finishing write tid 4 to trial04342147-41 2026-02-20T11:15:27.083 INFO:tasks.rados.rados.0.trial043.stdout:41: oid 41 updating version 0 to 7 2026-02-20T11:15:27.083 INFO:tasks.rados.rados.0.trial043.stdout:41: oid 41 updating version 7 to 9 2026-02-20T11:15:27.083 INFO:tasks.rados.rados.0.trial043.stdout:41: oid 41 version 9 is already newer than 9 2026-02-20T11:15:27.083 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 41 v 9 (ObjNum 40 snap 0 seq_num 40) dirty exists 2026-02-20T11:15:27.083 INFO:tasks.rados.rados.0.trial043.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2026-02-20T11:15:27.083 INFO:tasks.rados.rados.0.trial043.stdout:48: finishing write tid 1 to trial04342147-48 2026-02-20T11:15:27.083 INFO:tasks.rados.rados.0.trial043.stdout:41: done (15 left) 2026-02-20T11:15:27.083 INFO:tasks.rados.rados.0.trial043.stdout:51: write (excl) oid 2 current snap is 0 2026-02-20T11:15:27.083 INFO:tasks.rados.rados.0.trial043.stdout:28: finishing write tid 1 to trial04342147-28 2026-02-20T11:15:27.083 INFO:tasks.rados.rados.0.trial043.stdout:42: finishing write tid 3 to trial04342147-42 2026-02-20T11:15:27.084 INFO:tasks.rados.rados.0.trial043.stdout:51: seq_num 50 ranges {440350=460568,1309629=542666,2323973=516377} 2026-02-20T11:15:27.085 INFO:tasks.rados.rados.0.trial043.stdout:51: writing trial04342147-2 from 440350 to 900918 tid 1 2026-02-20T11:15:27.088 INFO:tasks.rados.rados.0.trial043.stdout:51: writing trial04342147-2 from 1309629 to 1852295 tid 2 2026-02-20T11:15:27.093 INFO:tasks.rados.rados.0.trial043.stdout:51: writing trial04342147-2 from 2323973 to 2840350 tid 3 2026-02-20T11:15:27.093 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:15:27.093 INFO:tasks.rados.rados.0.trial043.stdout:33: finishing write tid 2 to trial04342147-33 2026-02-20T11:15:27.093 INFO:tasks.rados.rados.0.trial043.stdout:42: finishing write tid 4 to trial04342147-42 2026-02-20T11:15:27.093 INFO:tasks.rados.rados.0.trial043.stdout:42: oid 42 updating version 0 to 9 2026-02-20T11:15:27.094 INFO:tasks.rados.rados.0.trial043.stdout:42: oid 42 version 9 is already newer than 9 2026-02-20T11:15:27.094 INFO:tasks.rados.rados.0.trial043.stdout:42: oid 42 version 9 is already newer than 7 2026-02-20T11:15:27.094 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 42 v 9 (ObjNum 41 snap 0 seq_num 41) dirty exists 2026-02-20T11:15:27.094 INFO:tasks.rados.rados.0.trial043.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2026-02-20T11:15:27.094 INFO:tasks.rados.rados.0.trial043.stdout:44: finishing write tid 1 to trial04342147-44 2026-02-20T11:15:27.094 INFO:tasks.rados.rados.0.trial043.stdout:42: done (15 left) 2026-02-20T11:15:27.094 INFO:tasks.rados.rados.0.trial043.stdout:52: write (excl) oid 14 current snap is 0 2026-02-20T11:15:27.094 INFO:tasks.rados.rados.0.trial043.stdout:52: seq_num 51 ranges {457157=535817,1768111=402677} 2026-02-20T11:15:27.096 INFO:tasks.rados.rados.0.trial043.stdout:52: writing trial04342147-14 from 457157 to 992974 tid 1 2026-02-20T11:15:27.101 INFO:tasks.rados.rados.0.trial043.stdout:52: writing trial04342147-14 from 1768111 to 2170788 tid 2 2026-02-20T11:15:27.101 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:15:27.105 INFO:tasks.rados.rados.0.trial043.stdout:48: finishing write tid 2 to trial04342147-48 2026-02-20T11:15:27.105 INFO:tasks.rados.rados.0.trial043.stdout:35: finishing write tid 3 to trial04342147-35 2026-02-20T11:15:27.107 INFO:tasks.rados.rados.0.trial043.stdout:35: finishing write tid 4 to trial04342147-35 2026-02-20T11:15:27.107 INFO:tasks.rados.rados.0.trial043.stdout:35: oid 35 updating version 0 to 12 2026-02-20T11:15:27.108 INFO:tasks.rados.rados.0.trial043.stdout:35: oid 35 version 12 is already newer than 10 2026-02-20T11:15:27.108 INFO:tasks.rados.rados.0.trial043.stdout:35: oid 35 version 12 is already newer than 12 2026-02-20T11:15:27.108 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 35 v 12 (ObjNum 34 snap 0 seq_num 34) dirty exists 2026-02-20T11:15:27.108 INFO:tasks.rados.rados.0.trial043.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2026-02-20T11:15:27.108 INFO:tasks.rados.rados.0.trial043.stdout:35: done (15 left) 2026-02-20T11:15:27.108 INFO:tasks.rados.rados.0.trial043.stdout:53: read oid 20 snap -1 2026-02-20T11:15:27.108 INFO:tasks.rados.rados.0.trial043.stdout:53: expect (ObjNum 19 snap 0 seq_num 19) 2026-02-20T11:15:27.108 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:15:27.109 INFO:tasks.rados.rados.0.trial043.stdout:26: finishing write tid 3 to trial04342147-26 2026-02-20T11:15:27.112 INFO:tasks.rados.rados.0.trial043.stdout:48: finishing write tid 3 to trial04342147-48 2026-02-20T11:15:27.115 INFO:tasks.rados.rados.0.trial043.stdout:48: finishing write tid 4 to trial04342147-48 2026-02-20T11:15:27.116 INFO:tasks.rados.rados.0.trial043.stdout:48: oid 48 updating version 0 to 6 2026-02-20T11:15:27.116 INFO:tasks.rados.rados.0.trial043.stdout:48: oid 48 version 6 is already newer than 4 2026-02-20T11:15:27.116 INFO:tasks.rados.rados.0.trial043.stdout:48: oid 48 version 6 is already newer than 6 2026-02-20T11:15:27.116 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 48 v 6 (ObjNum 47 snap 0 seq_num 47) dirty exists 2026-02-20T11:15:27.116 INFO:tasks.rados.rados.0.trial043.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2026-02-20T11:15:27.116 INFO:tasks.rados.rados.0.trial043.stdout:48: done (15 left) 2026-02-20T11:15:27.116 INFO:tasks.rados.rados.0.trial043.stdout:54: read oid 36 snap -1 2026-02-20T11:15:27.116 INFO:tasks.rados.rados.0.trial043.stdout:54: expect (ObjNum 35 snap 0 seq_num 35) 2026-02-20T11:15:27.116 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:15:27.117 INFO:tasks.rados.rados.0.trial043.stdout:26: finishing write tid 4 to trial04342147-26 2026-02-20T11:15:27.117 INFO:tasks.rados.rados.0.trial043.stdout:26: oid 26 updating version 0 to 9 2026-02-20T11:15:27.117 INFO:tasks.rados.rados.0.trial043.stdout:26: oid 26 version 9 is already newer than 7 2026-02-20T11:15:27.117 INFO:tasks.rados.rados.0.trial043.stdout:26: oid 26 version 9 is already newer than 9 2026-02-20T11:15:27.118 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 26 v 9 (ObjNum 25 snap 0 seq_num 25) dirty exists 2026-02-20T11:15:27.118 INFO:tasks.rados.rados.0.trial043.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2026-02-20T11:15:27.118 INFO:tasks.rados.rados.0.trial043.stdout:26: done (15 left) 2026-02-20T11:15:27.118 INFO:tasks.rados.rados.0.trial043.stdout:55: read oid 25 snap -1 2026-02-20T11:15:27.118 INFO:tasks.rados.rados.0.trial043.stdout:55: expect (ObjNum 24 snap 0 seq_num 24) 2026-02-20T11:15:27.118 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:15:27.118 INFO:tasks.rados.rados.0.trial043.stdout:49: finishing write tid 1 to trial04342147-49 2026-02-20T11:15:27.119 INFO:tasks.rados.rados.0.trial043.stdout:28: finishing write tid 2 to trial04342147-28 2026-02-20T11:15:27.126 INFO:tasks.rados.rados.0.trial043.stdout:44: finishing write tid 2 to trial04342147-44 2026-02-20T11:15:27.129 INFO:tasks.rados.rados.0.trial043.stdout:54: expect (ObjNum 35 snap 0 seq_num 35) 2026-02-20T11:15:27.148 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1358123 2026-02-20T11:15:27.173 INFO:tasks.rados.rados.0.trial043.stdout:54: done (15 left) 2026-02-20T11:15:27.173 INFO:tasks.rados.rados.0.trial043.stdout:56: read oid 30 snap -1 2026-02-20T11:15:27.173 INFO:tasks.rados.rados.0.trial043.stdout:56: expect (ObjNum 29 snap 0 seq_num 29) 2026-02-20T11:15:27.174 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:15:27.174 INFO:tasks.rados.rados.0.trial043.stdout:50: finishing write tid 1 to trial04342147-50 2026-02-20T11:15:27.174 INFO:tasks.rados.rados.0.trial043.stdout:28: finishing write tid 3 to trial04342147-28 2026-02-20T11:15:27.174 INFO:tasks.rados.rados.0.trial043.stdout:49: finishing write tid 2 to trial04342147-49 2026-02-20T11:15:27.174 INFO:tasks.rados.rados.0.trial043.stdout:44: finishing write tid 3 to trial04342147-44 2026-02-20T11:15:27.174 INFO:tasks.rados.rados.0.trial043.stdout:49: finishing write tid 3 to trial04342147-49 2026-02-20T11:15:27.174 INFO:tasks.rados.rados.0.trial043.stdout:49: finishing write tid 4 to trial04342147-49 2026-02-20T11:15:27.174 INFO:tasks.rados.rados.0.trial043.stdout:49: oid 49 updating version 0 to 9 2026-02-20T11:15:27.174 INFO:tasks.rados.rados.0.trial043.stdout:49: oid 49 version 9 is already newer than 9 2026-02-20T11:15:27.174 INFO:tasks.rados.rados.0.trial043.stdout:49: oid 49 version 9 is already newer than 7 2026-02-20T11:15:27.175 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 49 v 9 (ObjNum 48 snap 0 seq_num 48) dirty exists 2026-02-20T11:15:27.175 INFO:tasks.rados.rados.0.trial043.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2026-02-20T11:15:27.175 INFO:tasks.rados.rados.0.trial043.stdout:28: finishing write tid 4 to trial04342147-28 2026-02-20T11:15:27.175 INFO:tasks.rados.rados.0.trial043.stdout:28: oid 28 updating version 0 to 7 2026-02-20T11:15:27.175 INFO:tasks.rados.rados.0.trial043.stdout:28: oid 28 updating version 7 to 9 2026-02-20T11:15:27.175 INFO:tasks.rados.rados.0.trial043.stdout:28: oid 28 version 9 is already newer than 9 2026-02-20T11:15:27.175 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 28 v 9 (ObjNum 27 snap 0 seq_num 27) dirty exists 2026-02-20T11:15:27.175 INFO:tasks.rados.rados.0.trial043.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2026-02-20T11:15:27.175 INFO:tasks.rados.rados.0.trial043.stdout:44: finishing write tid 4 to trial04342147-44 2026-02-20T11:15:27.175 INFO:tasks.rados.rados.0.trial043.stdout:44: oid 44 updating version 0 to 10 2026-02-20T11:15:27.175 INFO:tasks.rados.rados.0.trial043.stdout:44: oid 44 updating version 10 to 12 2026-02-20T11:15:27.175 INFO:tasks.rados.rados.0.trial043.stdout:44: oid 44 version 12 is already newer than 12 2026-02-20T11:15:27.176 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 44 v 12 (ObjNum 43 snap 0 seq_num 43) dirty exists 2026-02-20T11:15:27.176 INFO:tasks.rados.rados.0.trial043.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2026-02-20T11:15:27.176 INFO:tasks.rados.rados.0.trial043.stdout:27: finishing write tid 1 to trial04342147-27 2026-02-20T11:15:27.176 INFO:tasks.rados.rados.0.trial043.stdout:28: done (15 left) 2026-02-20T11:15:27.176 INFO:tasks.rados.rados.0.trial043.stdout:44: done (14 left) 2026-02-20T11:15:27.176 INFO:tasks.rados.rados.0.trial043.stdout:49: done (13 left) 2026-02-20T11:15:27.176 INFO:tasks.rados.rados.0.trial043.stdout:57: read oid 10 snap -1 2026-02-20T11:15:27.176 INFO:tasks.rados.rados.0.trial043.stdout:57: expect (ObjNum 9 snap 0 seq_num 9) 2026-02-20T11:15:27.176 INFO:tasks.rados.rados.0.trial043.stdout:58: delete oid 28 current snap is 0 2026-02-20T11:15:27.176 INFO:tasks.rados.rados.0.trial043.stdout:37: finishing write tid 1 to trial04342147-37 2026-02-20T11:15:27.176 INFO:tasks.rados.rados.0.trial043.stdout:55: expect (ObjNum 24 snap 0 seq_num 24) 2026-02-20T11:15:27.190 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1233906 2026-02-20T11:15:27.202 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:15:27.205 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:15:27.205 INFO:teuthology.orchestra.run.trial043.stdout:{"fsid":"dfee6dd4-7cac-4c06-aaed-6868a4b5b9a0","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":23,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":1},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1771586112,"num_in_osds":4,"osd_in_since":1771586105,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":16},{"state_name":"active+clean","count":9}],"num_pgs":25,"num_pools":3,"num_objects":4,"data_bytes":590387,"bytes_used":85903482880,"bytes_avail":128844816384,"bytes_total":214748299264,"unknown_pgs_ratio":0.63999998569488525},"fsmap":{"epoch":1,"btime":"2026-02-20T11:15:03:729055+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":1,"modified":"2026-02-20T11:15:03.730334+0000","services":{}},"progress_events":{}} 2026-02-20T11:15:27.217 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-20T11:15:27.217 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-20T11:15:27.223 INFO:tasks.rados.rados.0.trial043.stdout:50: finishing write tid 2 to trial04342147-50 2026-02-20T11:15:27.223 INFO:tasks.rados.rados.0.trial043.stdout:47: finishing write tid 1 to trial04342147-47 2026-02-20T11:15:27.223 INFO:tasks.rados.rados.0.trial043.stdout:56: expect (ObjNum 29 snap 0 seq_num 29) 2026-02-20T11:15:27.229 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1741781 2026-02-20T11:15:27.237 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1487452 2026-02-20T11:15:27.271 INFO:tasks.rados.rados.0.trial043.stdout:37: finishing write tid 2 to trial04342147-37 2026-02-20T11:15:27.271 INFO:tasks.rados.rados.0.trial043.stdout:27: finishing write tid 2 to trial04342147-27 2026-02-20T11:15:27.271 INFO:tasks.rados.rados.0.trial043.stdout:55: done (14 left) 2026-02-20T11:15:27.271 INFO:tasks.rados.rados.0.trial043.stdout:56: done (13 left) 2026-02-20T11:15:27.271 INFO:tasks.rados.rados.0.trial043.stdout:58: done (12 left) 2026-02-20T11:15:27.271 INFO:tasks.rados.rados.0.trial043.stdout:59: write oid 12 current snap is 0 2026-02-20T11:15:27.271 INFO:tasks.rados.rados.0.trial043.stdout:47: finishing write tid 2 to trial04342147-47 2026-02-20T11:15:27.271 INFO:tasks.rados.rados.0.trial043.stdout:59: seq_num 53 ranges {490771=686315,1885075=406346} 2026-02-20T11:15:27.275 INFO:tasks.rados.rados.0.trial043.stdout:59: writing trial04342147-12 from 490771 to 1177086 tid 1 2026-02-20T11:15:27.280 INFO:tasks.rados.rados.0.trial043.stdout:59: writing trial04342147-12 from 1885075 to 2291421 tid 2 2026-02-20T11:15:27.280 INFO:tasks.rados.rados.0.trial043.stdout:60: write oid 29 current snap is 0 2026-02-20T11:15:27.280 INFO:tasks.rados.rados.0.trial043.stdout:37: finishing write tid 3 to trial04342147-37 2026-02-20T11:15:27.280 INFO:tasks.rados.rados.0.trial043.stdout:60: seq_num 54 ranges {507578=477917,1543557=550004,2833665=73913} 2026-02-20T11:15:27.283 INFO:tasks.rados.rados.0.trial043.stdout:60: writing trial04342147-29 from 507578 to 985495 tid 1 2026-02-20T11:15:27.287 INFO:tasks.rados.rados.0.trial043.stdout:60: writing trial04342147-29 from 1543557 to 2093561 tid 2 2026-02-20T11:15:27.291 INFO:tasks.rados.rados.0.trial043.stdout:60: writing trial04342147-29 from 2833665 to 2907578 tid 3 2026-02-20T11:15:27.291 INFO:tasks.rados.rados.0.trial043.stdout:61: read oid 38 snap -1 2026-02-20T11:15:27.292 INFO:tasks.rados.rados.0.trial043.stdout:61: expect (ObjNum 37 snap 0 seq_num 37) 2026-02-20T11:15:27.292 INFO:tasks.rados.rados.0.trial043.stdout:62: snap_create 2026-02-20T11:15:27.292 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 15 2026-02-20T11:15:27.292 INFO:tasks.rados.rados.0.trial043.stdout:47: finishing write tid 3 to trial04342147-47 2026-02-20T11:15:27.292 INFO:tasks.rados.rados.0.trial043.stdout:37: finishing write tid 4 to trial04342147-37 2026-02-20T11:15:27.292 INFO:tasks.rados.rados.0.trial043.stdout:37: oid 37 updating version 0 to 13 2026-02-20T11:15:27.292 INFO:tasks.rados.rados.0.trial043.stdout:37: oid 37 updating version 13 to 15 2026-02-20T11:15:27.292 INFO:tasks.rados.rados.0.trial043.stdout:37: oid 37 version 15 is already newer than 15 2026-02-20T11:15:27.292 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 37 v 15 (ObjNum 36 snap 0 seq_num 36) dirty exists 2026-02-20T11:15:27.292 INFO:tasks.rados.rados.0.trial043.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2026-02-20T11:15:27.292 INFO:tasks.rados.rados.0.trial043.stdout:47: finishing write tid 4 to trial04342147-47 2026-02-20T11:15:27.293 INFO:tasks.rados.rados.0.trial043.stdout:47: oid 47 updating version 0 to 15 2026-02-20T11:15:27.293 INFO:tasks.rados.rados.0.trial043.stdout:47: oid 47 version 15 is already newer than 13 2026-02-20T11:15:27.293 INFO:tasks.rados.rados.0.trial043.stdout:47: oid 47 version 15 is already newer than 15 2026-02-20T11:15:27.293 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 47 v 15 (ObjNum 46 snap 0 seq_num 46) dirty exists 2026-02-20T11:15:27.293 INFO:tasks.rados.rados.0.trial043.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2026-02-20T11:15:27.293 INFO:tasks.rados.rados.0.trial043.stdout:27: finishing write tid 3 to trial04342147-27 2026-02-20T11:15:27.293 INFO:tasks.rados.rados.0.trial043.stdout:51: finishing write tid 1 to trial04342147-2 2026-02-20T11:15:27.293 INFO:tasks.rados.rados.0.trial043.stdout:45: finishing write tid 1 to trial04342147-45 2026-02-20T11:15:27.293 INFO:tasks.rados.rados.0.trial043.stdout:27: finishing write tid 4 to trial04342147-27 2026-02-20T11:15:27.293 INFO:tasks.rados.rados.0.trial043.stdout:27: oid 27 updating version 0 to 12 2026-02-20T11:15:27.293 INFO:tasks.rados.rados.0.trial043.stdout:27: oid 27 version 12 is already newer than 12 2026-02-20T11:15:27.294 INFO:tasks.rados.rados.0.trial043.stdout:27: oid 27 version 12 is already newer than 10 2026-02-20T11:15:27.294 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 27 v 12 (ObjNum 26 snap 0 seq_num 26) dirty exists 2026-02-20T11:15:27.294 INFO:tasks.rados.rados.0.trial043.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2026-02-20T11:15:27.294 INFO:tasks.rados.rados.0.trial043.stdout:27: done (14 left) 2026-02-20T11:15:27.294 INFO:tasks.rados.rados.0.trial043.stdout:37: done (13 left) 2026-02-20T11:15:27.294 INFO:tasks.rados.rados.0.trial043.stdout:47: done (12 left) 2026-02-20T11:15:27.294 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 12 2026-02-20T11:15:27.294 INFO:tasks.rados.rados.0.trial043.stdout:57: expect (ObjNum 9 snap 0 seq_num 9) 2026-02-20T11:15:27.300 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:15:27.301 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:27.298+0000 7f7b1801a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:27.303 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:27.303+0000 7f7b1801a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:27.336 INFO:tasks.rados.rados.0.trial043.stdout:51: finishing write tid 2 to trial04342147-2 2026-02-20T11:15:27.337 INFO:tasks.rados.rados.0.trial043.stdout:24: finishing write tid 3 to trial04342147-24 2026-02-20T11:15:27.337 INFO:tasks.rados.rados.0.trial043.stdout:24: finishing write tid 4 to trial04342147-24 2026-02-20T11:15:27.337 INFO:tasks.rados.rados.0.trial043.stdout:24: oid 24 updating version 0 to 6 2026-02-20T11:15:27.337 INFO:tasks.rados.rados.0.trial043.stdout:24: oid 24 version 6 is already newer than 4 2026-02-20T11:15:27.337 INFO:tasks.rados.rados.0.trial043.stdout:24: oid 24 version 6 is already newer than 6 2026-02-20T11:15:27.337 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 24 v 6 (ObjNum 23 snap 0 seq_num 23) dirty exists 2026-02-20T11:15:27.337 INFO:tasks.rados.rados.0.trial043.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2026-02-20T11:15:27.337 INFO:tasks.rados.rados.0.trial043.stdout:24: done (11 left) 2026-02-20T11:15:27.337 INFO:tasks.rados.rados.0.trial043.stdout:57: done (10 left) 2026-02-20T11:15:27.338 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 10 2026-02-20T11:15:27.338 INFO:tasks.rados.rados.0.trial043.stdout:51: finishing write tid 3 to trial04342147-2 2026-02-20T11:15:27.338 INFO:tasks.rados.rados.0.trial043.stdout:45: finishing write tid 2 to trial04342147-45 2026-02-20T11:15:27.338 INFO:tasks.rados.rados.0.trial043.stdout:51: finishing write tid 4 to trial04342147-2 2026-02-20T11:15:27.338 INFO:tasks.rados.rados.0.trial043.stdout:53: expect (ObjNum 19 snap 0 seq_num 19) 2026-02-20T11:15:27.338 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:15:27.356 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 57176 2026-02-20T11:15:27.376 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:15:27.399 INFO:tasks.rados.rados.0.trial043.stdout:45: finishing write tid 3 to trial04342147-45 2026-02-20T11:15:27.399 INFO:tasks.rados.rados.0.trial043.stdout:53: done (9 left) 2026-02-20T11:15:27.399 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 9 2026-02-20T11:15:27.399 INFO:tasks.rados.rados.0.trial043.stdout:51: finishing write tid 5 to trial04342147-2 2026-02-20T11:15:27.399 INFO:tasks.rados.rados.0.trial043.stdout:51: oid 2 updating version 0 to 16 2026-02-20T11:15:27.399 INFO:tasks.rados.rados.0.trial043.stdout:51: oid 2 updating version 16 to 19 2026-02-20T11:15:27.400 INFO:tasks.rados.rados.0.trial043.stdout:51: oid 2 version 19 is already newer than 19 2026-02-20T11:15:27.400 INFO:tasks.rados.rados.0.trial043.stdout:51: oid 2 version 19 is already newer than 19 2026-02-20T11:15:27.400 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 2 v 19 (ObjNum 50 snap 0 seq_num 50) dirty exists 2026-02-20T11:15:27.400 INFO:tasks.rados.rados.0.trial043.stdout:51: left oid 2 (ObjNum 50 snap 0 seq_num 50) 2026-02-20T11:15:27.400 INFO:tasks.rados.rados.0.trial043.stdout:45: finishing write tid 4 to trial04342147-45 2026-02-20T11:15:27.400 INFO:tasks.rados.rados.0.trial043.stdout:45: oid 45 updating version 0 to 18 2026-02-20T11:15:27.400 INFO:tasks.rados.rados.0.trial043.stdout:45: oid 45 version 18 is already newer than 16 2026-02-20T11:15:27.400 INFO:tasks.rados.rados.0.trial043.stdout:45: oid 45 version 18 is already newer than 18 2026-02-20T11:15:27.400 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 45 v 18 (ObjNum 44 snap 0 seq_num 44) dirty exists 2026-02-20T11:15:27.400 INFO:tasks.rados.rados.0.trial043.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2026-02-20T11:15:27.400 INFO:tasks.rados.rados.0.trial043.stdout:52: finishing write tid 1 to trial04342147-14 2026-02-20T11:15:27.401 INFO:tasks.rados.rados.0.trial043.stdout:50: finishing write tid 3 to trial04342147-50 2026-02-20T11:15:27.401 INFO:tasks.rados.rados.0.trial043.stdout:45: done (8 left) 2026-02-20T11:15:27.401 INFO:tasks.rados.rados.0.trial043.stdout:51: done (7 left) 2026-02-20T11:15:27.401 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 7 2026-02-20T11:15:27.401 INFO:tasks.rados.rados.0.trial043.stdout:50: finishing write tid 4 to trial04342147-50 2026-02-20T11:15:27.401 INFO:tasks.rados.rados.0.trial043.stdout:50: oid 50 updating version 0 to 12 2026-02-20T11:15:27.401 INFO:tasks.rados.rados.0.trial043.stdout:50: oid 50 version 12 is already newer than 12 2026-02-20T11:15:27.401 INFO:tasks.rados.rados.0.trial043.stdout:50: oid 50 version 12 is already newer than 10 2026-02-20T11:15:27.401 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 50 v 12 (ObjNum 49 snap 0 seq_num 49) dirty exists 2026-02-20T11:15:27.401 INFO:tasks.rados.rados.0.trial043.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2026-02-20T11:15:27.401 INFO:tasks.rados.rados.0.trial043.stdout:52: finishing write tid 2 to trial04342147-14 2026-02-20T11:15:27.401 INFO:tasks.rados.rados.0.trial043.stdout:52: finishing write tid 3 to trial04342147-14 2026-02-20T11:15:27.402 INFO:tasks.rados.rados.0.trial043.stdout:50: done (6 left) 2026-02-20T11:15:27.402 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 6 2026-02-20T11:15:27.402 INFO:tasks.rados.rados.0.trial043.stdout:52: finishing write tid 4 to trial04342147-14 2026-02-20T11:15:27.402 INFO:tasks.rados.rados.0.trial043.stdout:52: oid 14 updating version 0 to 13 2026-02-20T11:15:27.402 INFO:tasks.rados.rados.0.trial043.stdout:52: oid 14 updating version 13 to 15 2026-02-20T11:15:27.402 INFO:tasks.rados.rados.0.trial043.stdout:52: oid 14 version 15 is already newer than 15 2026-02-20T11:15:27.402 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 14 v 15 (ObjNum 51 snap 0 seq_num 51) dirty exists 2026-02-20T11:15:27.402 INFO:tasks.rados.rados.0.trial043.stdout:52: left oid 14 (ObjNum 51 snap 0 seq_num 51) 2026-02-20T11:15:27.402 INFO:tasks.rados.rados.0.trial043.stdout:33: finishing write tid 3 to trial04342147-33 2026-02-20T11:15:27.402 INFO:tasks.rados.rados.0.trial043.stdout:33: finishing write tid 4 to trial04342147-33 2026-02-20T11:15:27.402 INFO:tasks.rados.rados.0.trial043.stdout:33: oid 33 updating version 0 to 12 2026-02-20T11:15:27.402 INFO:tasks.rados.rados.0.trial043.stdout:33: oid 33 version 12 is already newer than 12 2026-02-20T11:15:27.403 INFO:tasks.rados.rados.0.trial043.stdout:33: oid 33 version 12 is already newer than 10 2026-02-20T11:15:27.403 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 33 v 12 (ObjNum 32 snap 0 seq_num 32) dirty exists 2026-02-20T11:15:27.403 INFO:tasks.rados.rados.0.trial043.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2026-02-20T11:15:27.403 INFO:tasks.rados.rados.0.trial043.stdout:59: finishing write tid 1 to trial04342147-12 2026-02-20T11:15:27.403 INFO:tasks.rados.rados.0.trial043.stdout:43: finishing write tid 1 to trial04342147-43 2026-02-20T11:15:27.403 INFO:tasks.rados.rados.0.trial043.stdout:43: finishing write tid 2 to trial04342147-43 2026-02-20T11:15:27.403 INFO:tasks.rados.rados.0.trial043.stdout:33: done (5 left) 2026-02-20T11:15:27.403 INFO:tasks.rados.rados.0.trial043.stdout:52: done (4 left) 2026-02-20T11:15:27.403 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 4 2026-02-20T11:15:27.403 INFO:tasks.rados.rados.0.trial043.stdout:59: finishing write tid 2 to trial04342147-12 2026-02-20T11:15:27.403 INFO:tasks.rados.rados.0.trial043.stdout:43: finishing write tid 3 to trial04342147-43 2026-02-20T11:15:27.404 INFO:tasks.rados.rados.0.trial043.stdout:43: finishing write tid 4 to trial04342147-43 2026-02-20T11:15:27.404 INFO:tasks.rados.rados.0.trial043.stdout:43: oid 43 updating version 0 to 15 2026-02-20T11:15:27.404 INFO:tasks.rados.rados.0.trial043.stdout:43: oid 43 version 15 is already newer than 15 2026-02-20T11:15:27.404 INFO:tasks.rados.rados.0.trial043.stdout:43: oid 43 version 15 is already newer than 13 2026-02-20T11:15:27.404 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 43 v 15 (ObjNum 42 snap 0 seq_num 42) dirty exists 2026-02-20T11:15:27.404 INFO:tasks.rados.rados.0.trial043.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2026-02-20T11:15:27.404 INFO:tasks.rados.rados.0.trial043.stdout:60: finishing write tid 1 to trial04342147-29 2026-02-20T11:15:27.404 INFO:tasks.rados.rados.0.trial043.stdout:59: finishing write tid 3 to trial04342147-12 2026-02-20T11:15:27.404 INFO:tasks.rados.rados.0.trial043.stdout:43: done (3 left) 2026-02-20T11:15:27.404 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:15:27.404 INFO:tasks.rados.rados.0.trial043.stdout:61: expect (ObjNum 37 snap 0 seq_num 37) 2026-02-20T11:15:27.417 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:15:27.418 INFO:teuthology.orchestra.run.trial043.stderr:dumped all 2026-02-20T11:15:27.427 INFO:teuthology.orchestra.run.trial043.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2026-02-20T11:15:26.500322+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209715136,"kb_used":83890120,"kb_used_data":83890120,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":125825016,"statfs":{"total":214748299264,"available":128844816384,"internally_reserved":0,"allocated":85903482880,"data_stored":85903482880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.561720"},"pg_stats":[{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.981411+0000","last_change":"2026-02-20T11:15:16.984535+0000","last_active":"2026-02-20T11:15:16.981411+0000","last_peered":"2026-02-20T11:15:16.981411+0000","last_clean":"2026-02-20T11:15:16.981411+0000","last_became_active":"2026-02-20T11:15:15.006596+0000","last_became_peered":"2026-02-20T11:15:15.006596+0000","last_unstale":"2026-02-20T11:15:16.981411+0000","last_undegraded":"2026-02-20T11:15:16.981411+0000","last_fullsized":"2026-02-20T11:15:16.981411+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.983008+0000","last_change":"2026-02-20T11:15:16.985758+0000","last_active":"2026-02-20T11:15:16.983008+0000","last_peered":"2026-02-20T11:15:16.983008+0000","last_clean":"2026-02-20T11:15:16.983008+0000","last_became_active":"2026-02-20T11:15:15.019055+0000","last_became_peered":"2026-02-20T11:15:15.019055+0000","last_unstale":"2026-02-20T11:15:16.983008+0000","last_undegraded":"2026-02-20T11:15:16.983008+0000","last_fullsized":"2026-02-20T11:15:16.983008+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.2","version":"14'1","reported_seq":18,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.983646+0000","last_change":"2026-02-20T11:15:16.983646+0000","last_active":"2026-02-20T11:15:16.983646+0000","last_peered":"2026-02-20T11:15:16.983646+0000","last_clean":"2026-02-20T11:15:16.983646+0000","last_became_active":"2026-02-20T11:15:15.012729+0000","last_became_peered":"2026-02-20T11:15:15.012729+0000","last_unstale":"2026-02-20T11:15:16.983646+0000","last_undegraded":"2026-02-20T11:15:16.983646+0000","last_fullsized":"2026-02-20T11:15:16.983646+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.980370+0000","last_change":"2026-02-20T11:15:16.983291+0000","last_active":"2026-02-20T11:15:16.980370+0000","last_peered":"2026-02-20T11:15:16.980370+0000","last_clean":"2026-02-20T11:15:16.980370+0000","last_became_active":"2026-02-20T11:15:15.009983+0000","last_became_peered":"2026-02-20T11:15:15.009983+0000","last_unstale":"2026-02-20T11:15:16.980370+0000","last_undegraded":"2026-02-20T11:15:16.980370+0000","last_fullsized":"2026-02-20T11:15:16.980370+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'52","reported_seq":82,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.980870+0000","last_change":"2026-02-20T11:15:15.035494+0000","last_active":"2026-02-20T11:15:16.980870+0000","last_peered":"2026-02-20T11:15:16.980870+0000","last_clean":"2026-02-20T11:15:16.980870+0000","last_became_active":"2026-02-20T11:15:15.012645+0000","last_became_peered":"2026-02-20T11:15:15.012645+0000","last_unstale":"2026-02-20T11:15:16.980870+0000","last_undegraded":"2026-02-20T11:15:16.980870+0000","last_fullsized":"2026-02-20T11:15:16.980870+0000","mapping_epoch":12,"log_start":"13'48","ondisk_log_start":"13'48","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":48,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.982312+0000","last_change":"2026-02-20T11:15:16.984208+0000","last_active":"2026-02-20T11:15:16.982312+0000","last_peered":"2026-02-20T11:15:16.982312+0000","last_clean":"2026-02-20T11:15:16.982312+0000","last_became_active":"2026-02-20T11:15:15.027750+0000","last_became_peered":"2026-02-20T11:15:15.027750+0000","last_unstale":"2026-02-20T11:15:16.982312+0000","last_undegraded":"2026-02-20T11:15:16.982312+0000","last_fullsized":"2026-02-20T11:15:16.982312+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.983812+0000","last_change":"2026-02-20T11:15:16.985671+0000","last_active":"2026-02-20T11:15:16.983812+0000","last_peered":"2026-02-20T11:15:16.983812+0000","last_clean":"2026-02-20T11:15:16.983812+0000","last_became_active":"2026-02-20T11:15:15.003549+0000","last_became_peered":"2026-02-20T11:15:15.003549+0000","last_unstale":"2026-02-20T11:15:16.983812+0000","last_undegraded":"2026-02-20T11:15:16.983812+0000","last_fullsized":"2026-02-20T11:15:16.983812+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.982620+0000","last_change":"2026-02-20T11:15:16.986873+0000","last_active":"2026-02-20T11:15:16.982620+0000","last_peered":"2026-02-20T11:15:16.982620+0000","last_clean":"2026-02-20T11:15:16.982620+0000","last_became_active":"2026-02-20T11:15:15.018131+0000","last_became_peered":"2026-02-20T11:15:15.018131+0000","last_unstale":"2026-02-20T11:15:16.982620+0000","last_undegraded":"2026-02-20T11:15:16.982620+0000","last_fullsized":"2026-02-20T11:15:16.982620+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2026-02-20T11:15:23.488435+0000","last_change":"2026-02-20T11:15:23.488435+0000","last_active":"2026-02-20T11:15:23.488435+0000","last_peered":"2026-02-20T11:15:23.488435+0000","last_clean":"2026-02-20T11:15:23.488435+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-20T11:15:23.488435+0000","last_undegraded":"2026-02-20T11:15:23.488435+0000","last_fullsized":"2026-02-20T11:15:23.488435+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-20T11:15:16.983722+0000","last_change":"2026-02-20T11:15:16.986570+0000","last_active":"2026-02-20T11:15:16.983722+0000","last_peered":"2026-02-20T11:15:16.983722+0000","last_clean":"2026-02-20T11:15:16.983722+0000","last_became_active":"2026-02-20T11:15:15.057391+0000","last_became_peered":"2026-02-20T11:15:15.057391+0000","last_unstale":"2026-02-20T11:15:16.983722+0000","last_undegraded":"2026-02-20T11:15:16.983722+0000","last_fullsized":"2026-02-20T11:15:16.983722+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32210939904,"internally_reserved":0,"allocated":21476134912,"data_stored":21476134912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96633778176,"internally_reserved":0,"allocated":64427446272,"data_stored":64427446272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":24,"acting":24,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640265,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972788,"kb_used_data":20972788,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31455996,"statfs":{"total":53687074816,"available":32210939904,"internally_reserved":0,"allocated":21476134912,"data_stored":21476134912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640266,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20971984,"kb_used_data":20971984,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456800,"statfs":{"total":53687074816,"available":32211763200,"internally_reserved":0,"allocated":21475311616,"data_stored":21475311616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972656,"kb_used_data":20972656,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456128,"statfs":{"total":53687074816,"available":32211075072,"internally_reserved":0,"allocated":21475999744,"data_stored":21475999744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640266,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972692,"kb_used_data":20972692,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456092,"statfs":{"total":53687074816,"available":32211038208,"internally_reserved":0,"allocated":21476036608,"data_stored":21476036608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32211075072,"internally_reserved":0,"allocated":21475999744,"data_stored":21475999744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53687074816,"available":32211763200,"internally_reserved":0,"allocated":21475311616,"data_stored":21475311616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":53687074816,"available":32210939904,"internally_reserved":0,"allocated":21476134912,"data_stored":21476134912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53687074816,"available":32210939904,"internally_reserved":0,"allocated":21476134912,"data_stored":21476134912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-20T11:15:27.446 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1042256 2026-02-20T11:15:27.459 INFO:tasks.rados.rados.0.trial043.stdout:59: finishing write tid 4 to trial04342147-12 2026-02-20T11:15:27.459 INFO:tasks.rados.rados.0.trial043.stdout:59: oid 12 updating version 0 to 22 2026-02-20T11:15:27.459 INFO:tasks.rados.rados.0.trial043.stdout:59: oid 12 version 22 is already newer than 22 2026-02-20T11:15:27.460 INFO:tasks.rados.rados.0.trial043.stdout:59: oid 12 version 22 is already newer than 22 2026-02-20T11:15:27.460 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 12 v 22 (ObjNum 53 snap 0 seq_num 53) dirty exists 2026-02-20T11:15:27.460 INFO:tasks.rados.rados.0.trial043.stdout:59: left oid 12 (ObjNum 53 snap 0 seq_num 53) 2026-02-20T11:15:27.460 INFO:tasks.rados.rados.0.trial043.stdout:60: finishing write tid 2 to trial04342147-29 2026-02-20T11:15:27.460 INFO:tasks.rados.rados.0.trial043.stdout:60: finishing write tid 3 to trial04342147-29 2026-02-20T11:15:27.460 INFO:tasks.rados.rados.0.trial043.stdout:60: finishing write tid 4 to trial04342147-29 2026-02-20T11:15:27.460 INFO:tasks.rados.rados.0.trial043.stdout:60: finishing write tid 5 to trial04342147-29 2026-02-20T11:15:27.460 INFO:tasks.rados.rados.0.trial043.stdout:60: oid 29 updating version 0 to 19 2026-02-20T11:15:27.460 INFO:tasks.rados.rados.0.trial043.stdout:60: oid 29 version 19 is already newer than 19 2026-02-20T11:15:27.460 INFO:tasks.rados.rados.0.trial043.stdout:60: oid 29 version 19 is already newer than 19 2026-02-20T11:15:27.460 INFO:tasks.rados.rados.0.trial043.stdout:60: oid 29 version 19 is already newer than 19 2026-02-20T11:15:27.460 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 29 v 19 (ObjNum 54 snap 0 seq_num 54) dirty exists 2026-02-20T11:15:27.460 INFO:tasks.rados.rados.0.trial043.stdout:60: left oid 29 (ObjNum 54 snap 0 seq_num 54) 2026-02-20T11:15:27.460 INFO:tasks.rados.rados.0.trial043.stdout:59: done (2 left) 2026-02-20T11:15:27.460 INFO:tasks.rados.rados.0.trial043.stdout:60: done (1 left) 2026-02-20T11:15:27.460 INFO:tasks.rados.rados.0.trial043.stdout:61: done (0 left) 2026-02-20T11:15:27.503 INFO:tasks.rados.rados.0.trial043.stdout:62: done (0 left) 2026-02-20T11:15:27.504 INFO:tasks.rados.rados.0.trial043.stdout:63: read oid 49 snap -1 2026-02-20T11:15:27.504 INFO:tasks.rados.rados.0.trial043.stdout:63: expect (ObjNum 48 snap 0 seq_num 48) 2026-02-20T11:15:27.504 INFO:tasks.rados.rados.0.trial043.stdout:64: delete oid 24 current snap is 1 2026-02-20T11:15:27.528 INFO:tasks.rados.rados.0.trial043.stdout:63: expect (ObjNum 48 snap 0 seq_num 48) 2026-02-20T11:15:27.599 INFO:tasks.rados.rados.0.trial043.stdout:63: done (1 left) 2026-02-20T11:15:27.599 INFO:tasks.rados.rados.0.trial043.stdout:64: done (0 left) 2026-02-20T11:15:27.599 INFO:tasks.rados.rados.0.trial043.stdout:65: read oid 39 snap 0 2026-02-20T11:15:27.599 INFO:tasks.rados.rados.0.trial043.stdout:65: expect (ObjNum 38 snap 0 seq_num 38) 2026-02-20T11:15:27.599 INFO:tasks.rados.rados.0.trial043.stdout:66: read oid 16 snap -1 2026-02-20T11:15:27.599 INFO:tasks.rados.rados.0.trial043.stdout:66: expect (ObjNum 15 snap 0 seq_num 15) 2026-02-20T11:15:27.599 INFO:tasks.rados.rados.0.trial043.stdout:67: write oid 45 current snap is 1 2026-02-20T11:15:27.599 INFO:tasks.rados.rados.0.trial043.stdout:67: seq_num 56 ranges {541192=628415,1660521=553673,2888511=52681} 2026-02-20T11:15:27.603 INFO:tasks.rados.rados.0.trial043.stdout:67: writing trial04342147-45 from 541192 to 1169607 tid 1 2026-02-20T11:15:27.607 INFO:tasks.rados.rados.0.trial043.stdout:67: writing trial04342147-45 from 1660521 to 2214194 tid 2 2026-02-20T11:15:27.610 INFO:tasks.rados.rados.0.trial043.stdout:67: writing trial04342147-45 from 2888511 to 2941192 tid 3 2026-02-20T11:15:27.610 INFO:tasks.rados.rados.0.trial043.stdout:68: read oid 48 snap -1 2026-02-20T11:15:27.610 INFO:tasks.rados.rados.0.trial043.stdout:68: expect (ObjNum 47 snap 0 seq_num 47) 2026-02-20T11:15:27.610 INFO:tasks.rados.rados.0.trial043.stdout:69: snap_create 2026-02-20T11:15:27.611 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 4 2026-02-20T11:15:27.614 INFO:tasks.rados.rados.0.trial043.stdout:66: expect (ObjNum 15 snap 0 seq_num 15) 2026-02-20T11:15:27.614 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:15:27.669 INFO:tasks.rados.rados.0.trial043.stdout:65: expect (ObjNum 38 snap 0 seq_num 38) 2026-02-20T11:15:27.670 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:15:27.711 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:15:27.733 INFO:tasks.rados.rados.0.trial043.stdout:68: expect (ObjNum 47 snap 0 seq_num 47) 2026-02-20T11:15:27.733 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:15:27.771 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 995082 2026-02-20T11:15:27.786 INFO:tasks.rados.rados.0.trial043.stdout:67: finishing write tid 1 to trial04342147-45 2026-02-20T11:15:27.786 INFO:tasks.rados.rados.0.trial043.stdout:67: finishing write tid 2 to trial04342147-45 2026-02-20T11:15:27.786 INFO:tasks.rados.rados.0.trial043.stdout:65: done (3 left) 2026-02-20T11:15:27.786 INFO:tasks.rados.rados.0.trial043.stdout:66: done (2 left) 2026-02-20T11:15:27.786 INFO:tasks.rados.rados.0.trial043.stdout:68: done (1 left) 2026-02-20T11:15:27.786 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:15:27.786 INFO:tasks.rados.rados.0.trial043.stdout:67: finishing write tid 3 to trial04342147-45 2026-02-20T11:15:27.786 INFO:tasks.rados.rados.0.trial043.stdout:67: finishing write tid 4 to trial04342147-45 2026-02-20T11:15:27.787 INFO:tasks.rados.rados.0.trial043.stdout:67: finishing write tid 5 to trial04342147-45 2026-02-20T11:15:27.787 INFO:tasks.rados.rados.0.trial043.stdout:67: oid 45 updating version 0 to 23 2026-02-20T11:15:27.787 INFO:tasks.rados.rados.0.trial043.stdout:67: oid 45 version 23 is already newer than 23 2026-02-20T11:15:27.787 INFO:tasks.rados.rados.0.trial043.stdout:67: oid 45 version 23 is already newer than 23 2026-02-20T11:15:27.787 INFO:tasks.rados.rados.0.trial043.stdout:67: oid 45 version 23 is already newer than 23 2026-02-20T11:15:27.787 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 45 v 23 (ObjNum 56 snap 1 seq_num 56) dirty exists 2026-02-20T11:15:27.787 INFO:tasks.rados.rados.0.trial043.stdout:67: left oid 45 (ObjNum 56 snap 1 seq_num 56) 2026-02-20T11:15:27.787 INFO:tasks.rados.rados.0.trial043.stdout:67: done (0 left) 2026-02-20T11:15:28.511 INFO:tasks.rados.rados.0.trial043.stdout:69: done (0 left) 2026-02-20T11:15:28.511 INFO:tasks.rados.rados.0.trial043.stdout:70: delete oid 38 current snap is 2 2026-02-20T11:15:28.545 INFO:tasks.rados.rados.0.trial043.stdout:70: done (0 left) 2026-02-20T11:15:28.545 INFO:tasks.rados.rados.0.trial043.stdout:71: snap_remove snap 1 2026-02-20T11:15:28.932 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:15:29.009 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:29.007+0000 7f8915ba4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:29.011 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:29.011+0000 7f8915ba4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:29.514 INFO:tasks.rados.rados.0.trial043.stdout:71: done (0 left) 2026-02-20T11:15:29.514 INFO:tasks.rados.rados.0.trial043.stdout:72: read oid 15 snap -1 2026-02-20T11:15:29.514 INFO:tasks.rados.rados.0.trial043.stdout:72: expect (ObjNum 14 snap 0 seq_num 14) 2026-02-20T11:15:29.514 INFO:tasks.rados.rados.0.trial043.stdout:73: write oid 42 current snap is 2 2026-02-20T11:15:29.514 INFO:tasks.rados.rados.0.trial043.stdout:73: seq_num 58 ranges {574806=778913,2061132=557342} 2026-02-20T11:15:29.519 INFO:tasks.rados.rados.0.trial043.stdout:73: writing trial04342147-42 from 574806 to 1353719 tid 1 2026-02-20T11:15:29.524 INFO:tasks.rados.rados.0.trial043.stdout:73: writing trial04342147-42 from 2061132 to 2618474 tid 2 2026-02-20T11:15:29.524 INFO:tasks.rados.rados.0.trial043.stdout:74: write (excl) oid 29 current snap is 2 2026-02-20T11:15:29.524 INFO:tasks.rados.rados.0.trial043.stdout:74: seq_num 59 ranges {591613=454162,1719614=417353,2570780=420833} 2026-02-20T11:15:29.528 INFO:tasks.rados.rados.0.trial043.stdout:74: writing trial04342147-29 from 591613 to 1045775 tid 1 2026-02-20T11:15:29.532 INFO:tasks.rados.rados.0.trial043.stdout:74: writing trial04342147-29 from 1719614 to 2136967 tid 2 2026-02-20T11:15:29.536 INFO:tasks.rados.rados.0.trial043.stdout:74: writing trial04342147-29 from 2570780 to 2991613 tid 3 2026-02-20T11:15:29.536 INFO:tasks.rados.rados.0.trial043.stdout:75: write (excl) oid 10 current snap is 2 2026-02-20T11:15:29.536 INFO:tasks.rados.rados.0.trial043.stdout:75: seq_num 60 ranges {608420=529411,1778096=561011,2881850=126570} 2026-02-20T11:15:29.540 INFO:tasks.rados.rados.0.trial043.stdout:75: writing trial04342147-10 from 608420 to 1137831 tid 1 2026-02-20T11:15:29.545 INFO:tasks.rados.rados.0.trial043.stdout:75: writing trial04342147-10 from 1778096 to 2339107 tid 2 2026-02-20T11:15:29.548 INFO:tasks.rados.rados.0.trial043.stdout:75: writing trial04342147-10 from 2881850 to 3008420 tid 3 2026-02-20T11:15:29.548 INFO:tasks.rados.rados.0.trial043.stdout:76: snap_remove snap 0 2026-02-20T11:15:30.431 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-20T11:15:30.510 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:30.507+0000 7fd491a35640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:30.514 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:30.514+0000 7fd491a35640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:30.515 INFO:tasks.rados.rados.0.trial043.stdout:76: done (4 left) 2026-02-20T11:15:30.515 INFO:tasks.rados.rados.0.trial043.stdout:77: write oid 40 current snap is 2 2026-02-20T11:15:30.515 INFO:tasks.rados.rados.0.trial043.stdout:77: seq_num 61 ranges {625227=721013,1952931=704669} 2026-02-20T11:15:30.520 INFO:tasks.rados.rados.0.trial043.stdout:77: writing trial04342147-40 from 625227 to 1346240 tid 1 2026-02-20T11:15:30.526 INFO:tasks.rados.rados.0.trial043.stdout:77: writing trial04342147-40 from 1952931 to 2657600 tid 2 2026-02-20T11:15:30.526 INFO:tasks.rados.rados.0.trial043.stdout:78: snap_create 2026-02-20T11:15:30.526 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 5 2026-02-20T11:15:30.526 INFO:tasks.rados.rados.0.trial043.stdout:72: expect (ObjNum 14 snap 0 seq_num 14) 2026-02-20T11:15:30.564 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:15:30.586 INFO:tasks.rados.rados.0.trial043.stdout:74: finishing write tid 1 to trial04342147-29 2026-02-20T11:15:30.586 INFO:tasks.rados.rados.0.trial043.stdout:75: finishing write tid 1 to trial04342147-10 2026-02-20T11:15:30.586 INFO:tasks.rados.rados.0.trial043.stdout:72: done (4 left) 2026-02-20T11:15:30.587 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 4 2026-02-20T11:15:30.587 INFO:tasks.rados.rados.0.trial043.stdout:74: finishing write tid 2 to trial04342147-29 2026-02-20T11:15:30.587 INFO:tasks.rados.rados.0.trial043.stdout:74: finishing write tid 3 to trial04342147-29 2026-02-20T11:15:30.587 INFO:tasks.rados.rados.0.trial043.stdout:73: finishing write tid 1 to trial04342147-42 2026-02-20T11:15:30.587 INFO:tasks.rados.rados.0.trial043.stdout:75: finishing write tid 2 to trial04342147-10 2026-02-20T11:15:30.587 INFO:tasks.rados.rados.0.trial043.stdout:74: finishing write tid 4 to trial04342147-29 2026-02-20T11:15:30.587 INFO:tasks.rados.rados.0.trial043.stdout:73: finishing write tid 2 to trial04342147-42 2026-02-20T11:15:30.587 INFO:tasks.rados.rados.0.trial043.stdout:74: finishing write tid 5 to trial04342147-29 2026-02-20T11:15:30.588 INFO:tasks.rados.rados.0.trial043.stdout:74: oid 29 updating version 0 to 21 2026-02-20T11:15:30.588 INFO:tasks.rados.rados.0.trial043.stdout:74: oid 29 updating version 21 to 24 2026-02-20T11:15:30.588 INFO:tasks.rados.rados.0.trial043.stdout:74: oid 29 version 24 is already newer than 24 2026-02-20T11:15:30.588 INFO:tasks.rados.rados.0.trial043.stdout:74: oid 29 version 24 is already newer than 24 2026-02-20T11:15:30.588 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 29 v 24 (ObjNum 59 snap 2 seq_num 59) dirty exists 2026-02-20T11:15:30.588 INFO:tasks.rados.rados.0.trial043.stdout:74: left oid 29 (ObjNum 59 snap 2 seq_num 59) 2026-02-20T11:15:30.588 INFO:tasks.rados.rados.0.trial043.stdout:73: finishing write tid 3 to trial04342147-42 2026-02-20T11:15:30.588 INFO:tasks.rados.rados.0.trial043.stdout:74: done (3 left) 2026-02-20T11:15:30.589 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:15:30.589 INFO:tasks.rados.rados.0.trial043.stdout:75: finishing write tid 3 to trial04342147-10 2026-02-20T11:15:30.589 INFO:tasks.rados.rados.0.trial043.stdout:73: finishing write tid 4 to trial04342147-42 2026-02-20T11:15:30.589 INFO:tasks.rados.rados.0.trial043.stdout:73: oid 42 updating version 0 to 26 2026-02-20T11:15:30.589 INFO:tasks.rados.rados.0.trial043.stdout:73: oid 42 version 26 is already newer than 26 2026-02-20T11:15:30.589 INFO:tasks.rados.rados.0.trial043.stdout:73: oid 42 version 26 is already newer than 26 2026-02-20T11:15:30.589 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 42 v 26 (ObjNum 58 snap 2 seq_num 58) dirty exists 2026-02-20T11:15:30.589 INFO:tasks.rados.rados.0.trial043.stdout:73: left oid 42 (ObjNum 58 snap 2 seq_num 58) 2026-02-20T11:15:30.589 INFO:tasks.rados.rados.0.trial043.stdout:75: finishing write tid 4 to trial04342147-10 2026-02-20T11:15:30.590 INFO:tasks.rados.rados.0.trial043.stdout:75: finishing write tid 5 to trial04342147-10 2026-02-20T11:15:30.590 INFO:tasks.rados.rados.0.trial043.stdout:75: oid 10 updating version 0 to 17 2026-02-20T11:15:30.590 INFO:tasks.rados.rados.0.trial043.stdout:75: oid 10 version 17 is already newer than 14 2026-02-20T11:15:30.590 INFO:tasks.rados.rados.0.trial043.stdout:75: oid 10 version 17 is already newer than 17 2026-02-20T11:15:30.590 INFO:tasks.rados.rados.0.trial043.stdout:75: oid 10 version 17 is already newer than 17 2026-02-20T11:15:30.590 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 10 v 17 (ObjNum 60 snap 2 seq_num 60) dirty exists 2026-02-20T11:15:30.590 INFO:tasks.rados.rados.0.trial043.stdout:75: left oid 10 (ObjNum 60 snap 2 seq_num 60) 2026-02-20T11:15:30.590 INFO:tasks.rados.rados.0.trial043.stdout:73: done (2 left) 2026-02-20T11:15:30.591 INFO:tasks.rados.rados.0.trial043.stdout:75: done (1 left) 2026-02-20T11:15:30.591 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:15:30.591 INFO:tasks.rados.rados.0.trial043.stdout:77: finishing write tid 1 to trial04342147-40 2026-02-20T11:15:30.591 INFO:tasks.rados.rados.0.trial043.stdout:77: finishing write tid 2 to trial04342147-40 2026-02-20T11:15:30.591 INFO:tasks.rados.rados.0.trial043.stdout:77: finishing write tid 3 to trial04342147-40 2026-02-20T11:15:30.591 INFO:tasks.rados.rados.0.trial043.stdout:77: finishing write tid 4 to trial04342147-40 2026-02-20T11:15:30.591 INFO:tasks.rados.rados.0.trial043.stdout:77: oid 40 updating version 0 to 12 2026-02-20T11:15:30.591 INFO:tasks.rados.rados.0.trial043.stdout:77: oid 40 version 12 is already newer than 12 2026-02-20T11:15:30.592 INFO:tasks.rados.rados.0.trial043.stdout:77: oid 40 version 12 is already newer than 12 2026-02-20T11:15:30.592 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 40 v 12 (ObjNum 61 snap 2 seq_num 61) dirty exists 2026-02-20T11:15:30.592 INFO:tasks.rados.rados.0.trial043.stdout:77: left oid 40 (ObjNum 61 snap 2 seq_num 61) 2026-02-20T11:15:30.592 INFO:tasks.rados.rados.0.trial043.stdout:77: done (0 left) 2026-02-20T11:15:30.635 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:15:30.636 INFO:teuthology.orchestra.run.trial043.stderr:dumped all 2026-02-20T11:15:30.644 INFO:teuthology.orchestra.run.trial043.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2026-02-20T11:15:30.512919+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109483507,"num_objects":51,"num_object_clones":1,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":193,"num_read_kb":32189,"num_write":286,"num_write_kb":55782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209715136,"kb_used":84005596,"kb_used_data":84005596,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":125709540,"statfs":{"total":214748299264,"available":128726568960,"internally_reserved":0,"allocated":86021730304,"data_stored":86021730304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":108893120,"num_objects":47,"num_object_clones":1,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":140,"num_read_kb":32143,"num_write":205,"num_write_kb":54674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.012596"},"pg_stats":[{"pgid":"3.f","version":"19'6","reported_seq":27,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-20T11:15:27.512614+0000","last_change":"2026-02-20T11:15:24.596244+0000","last_active":"2026-02-20T11:15:27.512614+0000","last_peered":"2026-02-20T11:15:27.512614+0000","last_clean":"2026-02-20T11:15:27.512614+0000","last_became_active":"2026-02-20T11:15:24.592285+0000","last_became_peered":"2026-02-20T11:15:24.592285+0000","last_unstale":"2026-02-20T11:15:27.512614+0000","last_undegraded":"2026-02-20T11:15:27.512614+0000","last_fullsized":"2026-02-20T11:15:27.512614+0000","mapping_epoch":16,"log_start":"19'2","ondisk_log_start":"19'2","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":2,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":4,"num_read_kb":2,"num_write":9,"num_write_kb":1840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"19'6","reported_seq":29,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-20T11:15:27.511471+0000","last_change":"2026-02-20T11:15:24.564588+0000","last_active":"2026-02-20T11:15:27.511471+0000","last_peered":"2026-02-20T11:15:27.511471+0000","last_clean":"2026-02-20T11:15:27.511471+0000","last_became_active":"2026-02-20T11:15:24.548806+0000","last_became_peered":"2026-02-20T11:15:24.548806+0000","last_unstale":"2026-02-20T11:15:27.511471+0000","last_undegraded":"2026-02-20T11:15:27.511471+0000","last_fullsized":"2026-02-20T11:15:27.511471+0000","mapping_epoch":16,"log_start":"19'2","ondisk_log_start":"19'2","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":2,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4705894,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":3915,"num_write":8,"num_write_kb":1962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"19'10","reported_seq":34,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-20T11:15:27.510005+0000","last_change":"2026-02-20T11:15:24.557251+0000","last_active":"2026-02-20T11:15:27.510005+0000","last_peered":"2026-02-20T11:15:27.510005+0000","last_clean":"2026-02-20T11:15:27.510005+0000","last_became_active":"2026-02-20T11:15:24.549642+0000","last_became_peered":"2026-02-20T11:15:24.549642+0000","last_unstale":"2026-02-20T11:15:27.510005+0000","last_undegraded":"2026-02-20T11:15:27.510005+0000","last_fullsized":"2026-02-20T11:15:27.510005+0000","mapping_epoch":16,"log_start":"19'6","ondisk_log_start":"19'6","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":6,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4420175,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":4389,"num_write":13,"num_write_kb":3410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"19'19","reported_seq":46,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-20T11:15:27.609887+0000","last_change":"2026-02-20T11:15:24.572253+0000","last_active":"2026-02-20T11:15:27.609887+0000","last_peered":"2026-02-20T11:15:27.609887+0000","last_clean":"2026-02-20T11:15:27.609887+0000","last_became_active":"2026-02-20T11:15:24.554048+0000","last_became_peered":"2026-02-20T11:15:24.554048+0000","last_unstale":"2026-02-20T11:15:27.609887+0000","last_undegraded":"2026-02-20T11:15:27.609887+0000","last_fullsized":"2026-02-20T11:15:27.609887+0000","mapping_epoch":16,"log_start":"19'13","ondisk_log_start":"19'13","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":13,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12571471,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":3952,"num_write":25,"num_write_kb":6302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2026-02-20T11:15:26.515125+0000","last_change":"2026-02-20T11:15:24.609041+0000","last_active":"2026-02-20T11:15:26.515125+0000","last_peered":"2026-02-20T11:15:26.515125+0000","last_clean":"2026-02-20T11:15:26.515125+0000","last_became_active":"2026-02-20T11:15:24.599300+0000","last_became_peered":"2026-02-20T11:15:24.599300+0000","last_unstale":"2026-02-20T11:15:26.515125+0000","last_undegraded":"2026-02-20T11:15:26.515125+0000","last_fullsized":"2026-02-20T11:15:26.515125+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"19'12","reported_seq":37,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-20T11:15:27.510935+0000","last_change":"2026-02-20T11:15:24.590824+0000","last_active":"2026-02-20T11:15:27.510935+0000","last_peered":"2026-02-20T11:15:27.510935+0000","last_clean":"2026-02-20T11:15:27.510935+0000","last_became_active":"2026-02-20T11:15:24.577320+0000","last_became_peered":"2026-02-20T11:15:24.577320+0000","last_unstale":"2026-02-20T11:15:27.510935+0000","last_undegraded":"2026-02-20T11:15:27.510935+0000","last_fullsized":"2026-02-20T11:15:27.510935+0000","mapping_epoch":16,"log_start":"19'8","ondisk_log_start":"19'8","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":8,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8689087,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":3391,"num_write":16,"num_write_kb":3883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"19'9","reported_seq":30,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-20T11:15:27.511356+0000","last_change":"2026-02-20T11:15:24.570138+0000","last_active":"2026-02-20T11:15:27.511356+0000","last_peered":"2026-02-20T11:15:27.511356+0000","last_clean":"2026-02-20T11:15:27.511356+0000","last_became_active":"2026-02-20T11:15:24.555345+0000","last_became_peered":"2026-02-20T11:15:24.555345+0000","last_unstale":"2026-02-20T11:15:27.511356+0000","last_undegraded":"2026-02-20T11:15:27.511356+0000","last_fullsized":"2026-02-20T11:15:27.511356+0000","mapping_epoch":16,"log_start":"19'5","ondisk_log_start":"19'5","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":5,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7462209,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"19'6","reported_seq":23,"reported_epoch":19,"state":"active+clean","last_fresh":"2026-02-20T11:15:26.818696+0000","last_change":"2026-02-20T11:15:24.563825+0000","last_active":"2026-02-20T11:15:26.818696+0000","last_peered":"2026-02-20T11:15:26.818696+0000","last_clean":"2026-02-20T11:15:26.818696+0000","last_became_active":"2026-02-20T11:15:24.556605+0000","last_became_peered":"2026-02-20T11:15:24.556605+0000","last_unstale":"2026-02-20T11:15:26.818696+0000","last_undegraded":"2026-02-20T11:15:26.818696+0000","last_fullsized":"2026-02-20T11:15:26.818696+0000","mapping_epoch":16,"log_start":"19'2","ondisk_log_start":"19'2","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":2,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4789929,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":31,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-20T11:15:27.511469+0000","last_change":"2026-02-20T11:15:16.984535+0000","last_active":"2026-02-20T11:15:27.511469+0000","last_peered":"2026-02-20T11:15:27.511469+0000","last_clean":"2026-02-20T11:15:27.511469+0000","last_became_active":"2026-02-20T11:15:15.006596+0000","last_became_peered":"2026-02-20T11:15:15.006596+0000","last_unstale":"2026-02-20T11:15:27.511469+0000","last_undegraded":"2026-02-20T11:15:27.511469+0000","last_fullsized":"2026-02-20T11:15:27.511469+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"19'10","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2026-02-20T11:15:26.879889+0000","last_change":"2026-02-20T11:15:24.554281+0000","last_active":"2026-02-20T11:15:26.879889+0000","last_peered":"2026-02-20T11:15:26.879889+0000","last_clean":"2026-02-20T11:15:26.879889+0000","last_became_active":"2026-02-20T11:15:24.550094+0000","last_became_peered":"2026-02-20T11:15:24.550094+0000","last_unstale":"2026-02-20T11:15:26.879889+0000","last_undegraded":"2026-02-20T11:15:26.879889+0000","last_fullsized":"2026-02-20T11:15:26.879889+0000","mapping_epoch":16,"log_start":"19'6","ondisk_log_start":"19'6","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":6,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7749947,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":13,"num_write_kb":3498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":31,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-20T11:15:27.511867+0000","last_change":"2026-02-20T11:15:16.985758+0000","last_active":"2026-02-20T11:15:27.511867+0000","last_peered":"2026-02-20T11:15:27.511867+0000","last_clean":"2026-02-20T11:15:27.511867+0000","last_became_active":"2026-02-20T11:15:15.019055+0000","last_became_peered":"2026-02-20T11:15:15.019055+0000","last_unstale":"2026-02-20T11:15:27.511867+0000","last_undegraded":"2026-02-20T11:15:27.511867+0000","last_fullsized":"2026-02-20T11:15:27.511867+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"19'9","reported_seq":36,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-20T11:15:27.620353+0000","last_change":"2026-02-20T11:15:24.594862+0000","last_active":"2026-02-20T11:15:27.620353+0000","last_peered":"2026-02-20T11:15:27.620353+0000","last_clean":"2026-02-20T11:15:27.620353+0000","last_became_active":"2026-02-20T11:15:24.580572+0000","last_became_peered":"2026-02-20T11:15:24.580572+0000","last_unstale":"2026-02-20T11:15:27.620353+0000","last_undegraded":"2026-02-20T11:15:27.620353+0000","last_fullsized":"2026-02-20T11:15:27.620353+0000","mapping_epoch":16,"log_start":"19'5","ondisk_log_start":"19'5","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":5,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8100875,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":8525,"num_write":12,"num_write_kb":3878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'2","reported_seq":33,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-20T11:15:27.511727+0000","last_change":"2026-02-20T11:15:16.983646+0000","last_active":"2026-02-20T11:15:27.511727+0000","last_peered":"2026-02-20T11:15:27.511727+0000","last_clean":"2026-02-20T11:15:27.511727+0000","last_became_active":"2026-02-20T11:15:15.012729+0000","last_became_peered":"2026-02-20T11:15:15.012729+0000","last_unstale":"2026-02-20T11:15:27.511727+0000","last_undegraded":"2026-02-20T11:15:27.511727+0000","last_fullsized":"2026-02-20T11:15:27.511727+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"19'9","reported_seq":30,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-20T11:15:27.512609+0000","last_change":"2026-02-20T11:15:24.597593+0000","last_active":"2026-02-20T11:15:27.512609+0000","last_peered":"2026-02-20T11:15:27.512609+0000","last_clean":"2026-02-20T11:15:27.512609+0000","last_became_active":"2026-02-20T11:15:24.590844+0000","last_became_peered":"2026-02-20T11:15:24.590844+0000","last_unstale":"2026-02-20T11:15:27.512609+0000","last_undegraded":"2026-02-20T11:15:27.512609+0000","last_fullsized":"2026-02-20T11:15:27.512609+0000","mapping_epoch":16,"log_start":"19'5","ondisk_log_start":"19'5","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":5,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7176490,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":2973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":29,"reported_epoch":19,"state":"active+clean","last_fresh":"2026-02-20T11:15:26.514125+0000","last_change":"2026-02-20T11:15:16.983291+0000","last_active":"2026-02-20T11:15:26.514125+0000","last_peered":"2026-02-20T11:15:26.514125+0000","last_clean":"2026-02-20T11:15:26.514125+0000","last_became_active":"2026-02-20T11:15:15.009983+0000","last_became_peered":"2026-02-20T11:15:15.009983+0000","last_unstale":"2026-02-20T11:15:26.514125+0000","last_undegraded":"2026-02-20T11:15:26.514125+0000","last_fullsized":"2026-02-20T11:15:26.514125+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'52","reported_seq":97,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-20T11:15:27.511234+0000","last_change":"2026-02-20T11:15:15.035494+0000","last_active":"2026-02-20T11:15:27.511234+0000","last_peered":"2026-02-20T11:15:27.511234+0000","last_clean":"2026-02-20T11:15:27.511234+0000","last_became_active":"2026-02-20T11:15:15.012645+0000","last_became_peered":"2026-02-20T11:15:15.012645+0000","last_unstale":"2026-02-20T11:15:27.511234+0000","last_undegraded":"2026-02-20T11:15:27.511234+0000","last_fullsized":"2026-02-20T11:15:27.511234+0000","mapping_epoch":12,"log_start":"13'48","ondisk_log_start":"13'48","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":48,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"19'3","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-20T11:15:27.513238+0000","last_change":"2026-02-20T11:15:24.583239+0000","last_active":"2026-02-20T11:15:27.513238+0000","last_peered":"2026-02-20T11:15:27.513238+0000","last_clean":"2026-02-20T11:15:27.513238+0000","last_became_active":"2026-02-20T11:15:24.575441+0000","last_became_peered":"2026-02-20T11:15:24.575441+0000","last_unstale":"2026-02-20T11:15:27.513238+0000","last_undegraded":"2026-02-20T11:15:27.513238+0000","last_fullsized":"2026-02-20T11:15:27.513238+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2521017,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"19'6","reported_seq":32,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-20T11:15:27.612512+0000","last_change":"2026-02-20T11:15:24.617223+0000","last_active":"2026-02-20T11:15:27.612512+0000","last_peered":"2026-02-20T11:15:27.612512+0000","last_clean":"2026-02-20T11:15:27.612512+0000","last_became_active":"2026-02-20T11:15:24.612337+0000","last_became_peered":"2026-02-20T11:15:24.612337+0000","last_unstale":"2026-02-20T11:15:27.612512+0000","last_undegraded":"2026-02-20T11:15:27.612512+0000","last_fullsized":"2026-02-20T11:15:27.612512+0000","mapping_epoch":16,"log_start":"19'2","ondisk_log_start":"19'2","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":2,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":7941,"num_write":8,"num_write_kb":2298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":28,"reported_epoch":19,"state":"active+clean","last_fresh":"2026-02-20T11:15:26.514995+0000","last_change":"2026-02-20T11:15:16.984208+0000","last_active":"2026-02-20T11:15:26.514995+0000","last_peered":"2026-02-20T11:15:26.514995+0000","last_clean":"2026-02-20T11:15:26.514995+0000","last_became_active":"2026-02-20T11:15:15.027750+0000","last_became_peered":"2026-02-20T11:15:15.027750+0000","last_unstale":"2026-02-20T11:15:26.514995+0000","last_undegraded":"2026-02-20T11:15:26.514995+0000","last_fullsized":"2026-02-20T11:15:26.514995+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"19'15","reported_seq":38,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-20T11:15:27.512155+0000","last_change":"2026-02-20T11:15:24.575271+0000","last_active":"2026-02-20T11:15:27.512155+0000","last_peered":"2026-02-20T11:15:27.512155+0000","last_clean":"2026-02-20T11:15:27.512155+0000","last_became_active":"2026-02-20T11:15:24.559859+0000","last_became_peered":"2026-02-20T11:15:24.559859+0000","last_unstale":"2026-02-20T11:15:27.512155+0000","last_undegraded":"2026-02-20T11:15:27.512155+0000","last_fullsized":"2026-02-20T11:15:27.512155+0000","mapping_epoch":16,"log_start":"19'11","ondisk_log_start":"19'11","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":11,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9983226,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-20T11:15:27.512747+0000","last_change":"2026-02-20T11:15:16.985671+0000","last_active":"2026-02-20T11:15:27.512747+0000","last_peered":"2026-02-20T11:15:27.512747+0000","last_clean":"2026-02-20T11:15:27.512747+0000","last_became_active":"2026-02-20T11:15:15.003549+0000","last_became_peered":"2026-02-20T11:15:15.003549+0000","last_unstale":"2026-02-20T11:15:27.512747+0000","last_undegraded":"2026-02-20T11:15:27.512747+0000","last_fullsized":"2026-02-20T11:15:27.512747+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"19'12","reported_seq":34,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-20T11:15:27.511066+0000","last_change":"2026-02-20T11:15:24.560282+0000","last_active":"2026-02-20T11:15:27.511066+0000","last_peered":"2026-02-20T11:15:27.511066+0000","last_clean":"2026-02-20T11:15:27.511066+0000","last_became_active":"2026-02-20T11:15:24.543822+0000","last_became_peered":"2026-02-20T11:15:24.543822+0000","last_unstale":"2026-02-20T11:15:27.511066+0000","last_undegraded":"2026-02-20T11:15:27.511066+0000","last_fullsized":"2026-02-20T11:15:27.511066+0000","mapping_epoch":16,"log_start":"19'8","ondisk_log_start":"19'8","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":8,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9344560,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":31,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-20T11:15:27.512728+0000","last_change":"2026-02-20T11:15:16.986873+0000","last_active":"2026-02-20T11:15:27.512728+0000","last_peered":"2026-02-20T11:15:27.512728+0000","last_clean":"2026-02-20T11:15:27.512728+0000","last_became_active":"2026-02-20T11:15:15.018131+0000","last_became_peered":"2026-02-20T11:15:15.018131+0000","last_unstale":"2026-02-20T11:15:27.512728+0000","last_undegraded":"2026-02-20T11:15:27.512728+0000","last_fullsized":"2026-02-20T11:15:27.512728+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"19'22","reported_seq":47,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-20T11:15:27.510365+0000","last_change":"2026-02-20T11:15:24.590252+0000","last_active":"2026-02-20T11:15:27.510365+0000","last_peered":"2026-02-20T11:15:27.510365+0000","last_clean":"2026-02-20T11:15:27.510365+0000","last_became_active":"2026-02-20T11:15:24.578162+0000","last_became_peered":"2026-02-20T11:15:24.578162+0000","last_unstale":"2026-02-20T11:15:27.510365+0000","last_undegraded":"2026-02-20T11:15:27.510365+0000","last_fullsized":"2026-02-20T11:15:27.510365+0000","mapping_epoch":16,"log_start":"19'18","ondisk_log_start":"19'18","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":18,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13916031,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":7,"num_write":29,"num_write_kb":8343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":31,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-20T11:15:27.512145+0000","last_change":"2026-02-20T11:15:16.986570+0000","last_active":"2026-02-20T11:15:27.512145+0000","last_peered":"2026-02-20T11:15:27.512145+0000","last_clean":"2026-02-20T11:15:27.512145+0000","last_became_active":"2026-02-20T11:15:15.057391+0000","last_became_peered":"2026-02-20T11:15:15.057391+0000","last_unstale":"2026-02-20T11:15:27.512145+0000","last_undegraded":"2026-02-20T11:15:27.512145+0000","last_fullsized":"2026-02-20T11:15:27.512145+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":108893120,"num_objects":47,"num_object_clones":1,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":140,"num_read_kb":32143,"num_write":205,"num_write_kb":54674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214748299264,"available":128654495744,"internally_reserved":0,"allocated":86093803520,"data_stored":86093803520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61,"ondisk_log_size":61,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32138956800,"internally_reserved":0,"allocated":21548118016,"data_stored":21548118016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96480100352,"internally_reserved":0,"allocated":64581124096,"data_stored":64581124096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640270,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":21016508,"kb_used_data":21016508,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31412276,"statfs":{"total":53687074816,"available":32166170624,"internally_reserved":0,"allocated":21520904192,"data_stored":21520904192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640270,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":21007524,"kb_used_data":21007524,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31421260,"statfs":{"total":53687074816,"available":32175370240,"internally_reserved":0,"allocated":21511704576,"data_stored":21511704576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20973088,"kb_used_data":20973088,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31455696,"statfs":{"total":53687074816,"available":32210632704,"internally_reserved":0,"allocated":21476442112,"data_stored":21476442112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640271,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":21008476,"kb_used_data":21008476,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31420308,"statfs":{"total":53687074816,"available":32174395392,"internally_reserved":0,"allocated":21512679424,"data_stored":21512679424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32180969472,"internally_reserved":0,"allocated":21506105344,"data_stored":21506105344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53687074816,"available":32160174080,"internally_reserved":0,"allocated":21526900736,"data_stored":21526900736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":53687074816,"available":32138956800,"internally_reserved":0,"allocated":21548118016,"data_stored":21548118016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53687074816,"available":32138956800,"internally_reserved":0,"allocated":21548118016,"data_stored":21548118016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53687074816,"available":32174395392,"internally_reserved":0,"allocated":21512679424,"data_stored":21512679424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53687074816,"available":32180969472,"internally_reserved":0,"allocated":21506105344,"data_stored":21506105344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53687074816,"available":32160174080,"internally_reserved":0,"allocated":21526900736,"data_stored":21526900736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53687074816,"available":32138956800,"internally_reserved":0,"allocated":21548118016,"data_stored":21548118016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-20T11:15:30.649 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2026-02-20T11:15:30.649 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-20T11:15:30.650 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [0, 1, 2, 3] 2026-02-20T11:15:30.650 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-20T11:15:30.650 INFO:teuthology.orchestra.run:waiting for 300 2026-02-20T11:15:30.707 INFO:tasks.ceph.osd.2:Stopped 2026-02-20T11:15:31.070 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:15:31.144 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:31.143+0000 7fd63cc51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:31.147 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:31.147+0000 7fd63cc51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:31.518 INFO:tasks.rados.rados.0.trial043.stdout:78: done (0 left) 2026-02-20T11:15:31.518 INFO:tasks.rados.rados.0.trial043.stdout:79: write oid 6 current snap is 3 2026-02-20T11:15:31.519 INFO:tasks.rados.rados.0.trial043.stdout:79: seq_num 62 ranges {642034=796262,1895060=564680} 2026-02-20T11:15:31.524 INFO:tasks.rados.rados.0.trial043.stdout:79: writing trial04342147-6 from 642034 to 1438296 tid 1 2026-02-20T11:15:31.528 INFO:tasks.rados.rados.0.trial043.stdout:79: writing trial04342147-6 from 1895060 to 2459740 tid 2 2026-02-20T11:15:31.528 INFO:tasks.rados.rados.0.trial043.stdout:80: write oid 7 current snap is 3 2026-02-20T11:15:31.528 INFO:tasks.rados.rados.0.trial043.stdout:80: seq_num 63 ranges {658841=471511,1553542=708338,2964119=94722} 2026-02-20T11:15:31.532 INFO:tasks.rados.rados.0.trial043.stdout:80: writing trial04342147-7 from 658841 to 1130352 tid 1 2026-02-20T11:15:31.536 INFO:tasks.rados.rados.0.trial043.stdout:80: writing trial04342147-7 from 1553542 to 2261880 tid 2 2026-02-20T11:15:31.539 INFO:tasks.rados.rados.0.trial043.stdout:80: writing trial04342147-7 from 2964119 to 3058841 tid 3 2026-02-20T11:15:31.539 INFO:tasks.rados.rados.0.trial043.stdout:81: write oid 22 current snap is 3 2026-02-20T11:15:31.540 INFO:tasks.rados.rados.0.trial043.stdout:81: seq_num 64 ranges {675648=546760,2012024=568349} 2026-02-20T11:15:31.544 INFO:tasks.rados.rados.0.trial043.stdout:81: writing trial04342147-22 from 675648 to 1222408 tid 1 2026-02-20T11:15:31.549 INFO:tasks.rados.rados.0.trial043.stdout:81: writing trial04342147-22 from 2012024 to 2580373 tid 2 2026-02-20T11:15:31.550 INFO:tasks.rados.rados.0.trial043.stdout:82: delete oid 14 current snap is 3 2026-02-20T11:15:31.594 INFO:tasks.rados.rados.0.trial043.stdout:82: done (3 left) 2026-02-20T11:15:31.594 INFO:tasks.rados.rados.0.trial043.stdout:83: write (excl) oid 45 current snap is 3 2026-02-20T11:15:31.594 INFO:tasks.rados.rados.0.trial043.stdout:83: seq_num 66 ranges {709262=697258,2012635=572018,3046388=62874} 2026-02-20T11:15:31.599 INFO:tasks.rados.rados.0.trial043.stdout:83: writing trial04342147-45 from 709262 to 1406520 tid 1 2026-02-20T11:15:31.604 INFO:tasks.rados.rados.0.trial043.stdout:83: writing trial04342147-45 from 2012635 to 2584653 tid 2 2026-02-20T11:15:31.606 INFO:tasks.rados.rados.0.trial043.stdout:83: writing trial04342147-45 from 3046388 to 3109262 tid 3 2026-02-20T11:15:31.606 INFO:tasks.rados.rados.0.trial043.stdout:84: write oid 27 current snap is 3 2026-02-20T11:15:31.607 INFO:tasks.rados.rados.0.trial043.stdout:84: seq_num 67 ranges {726069=772507,2071117=715676} 2026-02-20T11:15:31.612 INFO:tasks.rados.rados.0.trial043.stdout:84: writing trial04342147-27 from 726069 to 1498576 tid 1 2026-02-20T11:15:31.617 INFO:tasks.rados.rados.0.trial043.stdout:84: writing trial04342147-27 from 2071117 to 2786793 tid 2 2026-02-20T11:15:31.617 INFO:tasks.rados.rados.0.trial043.stdout:85: read oid 14 snap -1 2026-02-20T11:15:31.617 INFO:tasks.rados.rados.0.trial043.stdout:85: expect deleted 2026-02-20T11:15:31.618 INFO:tasks.rados.rados.0.trial043.stdout:86: write oid 49 current snap is 3 2026-02-20T11:15:31.618 INFO:tasks.rados.rados.0.trial043.stdout:86: seq_num 68 ranges {742876=447756,1729599=575687,3101234=41642} 2026-02-20T11:15:31.621 INFO:tasks.rados.rados.0.trial043.stdout:86: writing trial04342147-49 from 742876 to 1190632 tid 1 2026-02-20T11:15:31.625 INFO:tasks.rados.rados.0.trial043.stdout:86: writing trial04342147-49 from 1729599 to 2305286 tid 2 2026-02-20T11:15:31.629 INFO:tasks.rados.rados.0.trial043.stdout:86: writing trial04342147-49 from 3101234 to 3142876 tid 3 2026-02-20T11:15:31.629 INFO:tasks.rados.rados.0.trial043.stdout:87: read oid 33 snap -1 2026-02-20T11:15:31.629 INFO:tasks.rados.rados.0.trial043.stdout:87: expect (ObjNum 32 snap 0 seq_num 32) 2026-02-20T11:15:31.629 INFO:tasks.rados.rados.0.trial043.stdout:88: write oid 10 current snap is 3 2026-02-20T11:15:31.629 INFO:tasks.rados.rados.0.trial043.stdout:88: seq_num 69 ranges {759683=639358,1904434=719345,3128657=31026} 2026-02-20T11:15:31.634 INFO:tasks.rados.rados.0.trial043.stdout:88: writing trial04342147-10 from 759683 to 1399041 tid 1 2026-02-20T11:15:31.639 INFO:tasks.rados.rados.0.trial043.stdout:88: writing trial04342147-10 from 1904434 to 2623779 tid 2 2026-02-20T11:15:31.641 INFO:tasks.rados.rados.0.trial043.stdout:88: writing trial04342147-10 from 3128657 to 3159683 tid 3 2026-02-20T11:15:31.641 INFO:tasks.rados.rados.0.trial043.stdout:89: snap_remove snap 2 2026-02-20T11:15:32.523 INFO:tasks.rados.rados.0.trial043.stdout:89: done (9 left) 2026-02-20T11:15:32.523 INFO:tasks.rados.rados.0.trial043.stdout:90: write oid 34 current snap is 3 2026-02-20T11:15:32.523 INFO:tasks.rados.rados.0.trial043.stdout:90: seq_num 70 ranges {776490=714607,2246563=579356} 2026-02-20T11:15:32.530 INFO:tasks.rados.rados.0.trial043.stdout:90: writing trial04342147-34 from 776490 to 1491097 tid 1 2026-02-20T11:15:32.536 INFO:tasks.rados.rados.0.trial043.stdout:90: writing trial04342147-34 from 2246563 to 2825919 tid 2 2026-02-20T11:15:32.536 INFO:tasks.rados.rados.0.trial043.stdout:91: read oid 3 snap -1 2026-02-20T11:15:32.536 INFO:tasks.rados.rados.0.trial043.stdout:91: expect (ObjNum 2 snap 0 seq_num 2) 2026-02-20T11:15:32.536 INFO:tasks.rados.rados.0.trial043.stdout:92: read oid 41 snap -1 2026-02-20T11:15:32.536 INFO:tasks.rados.rados.0.trial043.stdout:92: expect (ObjNum 40 snap 0 seq_num 40) 2026-02-20T11:15:32.536 INFO:tasks.rados.rados.0.trial043.stdout:93: read oid 26 snap -1 2026-02-20T11:15:32.536 INFO:tasks.rados.rados.0.trial043.stdout:93: expect (ObjNum 25 snap 0 seq_num 25) 2026-02-20T11:15:32.536 INFO:tasks.rados.rados.0.trial043.stdout:94: read oid 31 snap -1 2026-02-20T11:15:32.536 INFO:tasks.rados.rados.0.trial043.stdout:94: expect (ObjNum 30 snap 0 seq_num 30) 2026-02-20T11:15:32.536 INFO:tasks.rados.rados.0.trial043.stdout:95: write oid 39 current snap is 3 2026-02-20T11:15:32.536 INFO:tasks.rados.rados.0.trial043.stdout:84: finishing write tid 1 to trial04342147-27 2026-02-20T11:15:32.536 INFO:tasks.rados.rados.0.trial043.stdout:87: expect (ObjNum 32 snap 0 seq_num 32) 2026-02-20T11:15:32.560 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1288743 2026-02-20T11:15:32.575 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1677531 2026-02-20T11:15:32.586 INFO:tasks.rados.rados.0.trial043.stdout:84: finishing write tid 2 to trial04342147-27 2026-02-20T11:15:32.586 INFO:tasks.rados.rados.0.trial043.stdout:84: finishing write tid 3 to trial04342147-27 2026-02-20T11:15:32.586 INFO:tasks.rados.rados.0.trial043.stdout:84: finishing write tid 4 to trial04342147-27 2026-02-20T11:15:32.586 INFO:tasks.rados.rados.0.trial043.stdout:84: oid 27 updating version 0 to 21 2026-02-20T11:15:32.586 INFO:tasks.rados.rados.0.trial043.stdout:84: oid 27 version 21 is already newer than 21 2026-02-20T11:15:32.586 INFO:tasks.rados.rados.0.trial043.stdout:84: oid 27 version 21 is already newer than 21 2026-02-20T11:15:32.586 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 27 v 21 (ObjNum 67 snap 3 seq_num 67) dirty exists 2026-02-20T11:15:32.587 INFO:tasks.rados.rados.0.trial043.stdout:84: left oid 27 (ObjNum 67 snap 3 seq_num 67) 2026-02-20T11:15:32.587 INFO:tasks.rados.rados.0.trial043.stdout:95: seq_num 71 ranges {793297=789856,2305045=723014} 2026-02-20T11:15:32.593 INFO:tasks.rados.rados.0.trial043.stdout:95: writing trial04342147-39 from 793297 to 1583153 tid 1 2026-02-20T11:15:32.599 INFO:tasks.rados.rados.0.trial043.stdout:95: writing trial04342147-39 from 2305045 to 3028059 tid 2 2026-02-20T11:15:32.599 INFO:tasks.rados.rados.0.trial043.stdout:84: done (14 left) 2026-02-20T11:15:32.599 INFO:tasks.rados.rados.0.trial043.stdout:85: done (13 left) 2026-02-20T11:15:32.599 INFO:tasks.rados.rados.0.trial043.stdout:87: done (12 left) 2026-02-20T11:15:32.599 INFO:tasks.rados.rados.0.trial043.stdout:96: delete oid 21 current snap is 3 2026-02-20T11:15:32.599 INFO:tasks.rados.rados.0.trial043.stdout:93: expect (ObjNum 25 snap 0 seq_num 25) 2026-02-20T11:15:32.649 INFO:tasks.rados.rados.0.trial043.stdout:92: expect (ObjNum 40 snap 0 seq_num 40) 2026-02-20T11:15:32.670 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:15:35.707 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3] 2026-02-20T11:15:35.708 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-20T11:15:35.708 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-20T11:15:35.785 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:35.783+0000 7eff52d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:35.787 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:15:35.788+0000 7eff52d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:15:35.898 INFO:teuthology.orchestra.run.trial043.stdout: 2026-02-20T11:15:35.899 INFO:teuthology.orchestra.run.trial043.stderr:dumped all 2026-02-20T11:15:35.907 INFO:teuthology.orchestra.run.trial043.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2026-02-20T11:15:33.940213+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":110411537,"num_objects":55,"num_object_clones":4,"num_object_copies":165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":234,"num_read_kb":51191,"num_write":341,"num_write_kb":71665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84,"ondisk_log_size":84,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209715136,"kb_used":84086912,"kb_used_data":84086912,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":125628224,"statfs":{"total":214748299264,"available":128643301376,"internally_reserved":0,"allocated":86104997888,"data_stored":86104997888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":928030,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":19002,"num_write":55,"num_write_kb":15883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.432678"},"pg_stats":[{"pgid":"3.f","version":"19'6","reported_seq":27,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-20T11:15:27.512614+0000","last_change":"2026-02-20T11:15:24.596244+0000","last_active":"2026-02-20T11:15:27.512614+0000","last_peered":"2026-02-20T11:15:27.512614+0000","last_clean":"2026-02-20T11:15:27.512614+0000","last_became_active":"2026-02-20T11:15:24.592285+0000","last_became_peered":"2026-02-20T11:15:24.592285+0000","last_unstale":"2026-02-20T11:15:27.512614+0000","last_undegraded":"2026-02-20T11:15:27.512614+0000","last_fullsized":"2026-02-20T11:15:27.512614+0000","mapping_epoch":16,"log_start":"19'2","ondisk_log_start":"19'2","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":2,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":4,"num_read_kb":2,"num_write":9,"num_write_kb":1840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"19'6","reported_seq":29,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-20T11:15:27.511471+0000","last_change":"2026-02-20T11:15:24.564588+0000","last_active":"2026-02-20T11:15:27.511471+0000","last_peered":"2026-02-20T11:15:27.511471+0000","last_clean":"2026-02-20T11:15:27.511471+0000","last_became_active":"2026-02-20T11:15:24.548806+0000","last_became_peered":"2026-02-20T11:15:24.548806+0000","last_unstale":"2026-02-20T11:15:27.511471+0000","last_undegraded":"2026-02-20T11:15:27.511471+0000","last_fullsized":"2026-02-20T11:15:27.511471+0000","mapping_epoch":16,"log_start":"19'2","ondisk_log_start":"19'2","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":2,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4705894,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":3915,"num_write":8,"num_write_kb":1962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"19'10","reported_seq":52,"reported_epoch":25,"state":"active+clean","last_fresh":"2026-02-20T11:15:32.532556+0000","last_change":"2026-02-20T11:15:32.538070+0000","last_active":"2026-02-20T11:15:32.532556+0000","last_peered":"2026-02-20T11:15:32.532556+0000","last_clean":"2026-02-20T11:15:32.532556+0000","last_became_active":"2026-02-20T11:15:24.549642+0000","last_became_peered":"2026-02-20T11:15:24.549642+0000","last_unstale":"2026-02-20T11:15:32.532556+0000","last_undegraded":"2026-02-20T11:15:32.532556+0000","last_fullsized":"2026-02-20T11:15:32.532556+0000","mapping_epoch":16,"log_start":"19'6","ondisk_log_start":"19'6","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":6,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4420175,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":4389,"num_write":13,"num_write_kb":3410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"23'26","reported_seq":71,"reported_epoch":25,"state":"active+clean","last_fresh":"2026-02-20T11:15:32.565501+0000","last_change":"2026-02-20T11:15:32.571449+0000","last_active":"2026-02-20T11:15:32.565501+0000","last_peered":"2026-02-20T11:15:32.565501+0000","last_clean":"2026-02-20T11:15:32.565501+0000","last_became_active":"2026-02-20T11:15:24.554048+0000","last_became_peered":"2026-02-20T11:15:24.554048+0000","last_unstale":"2026-02-20T11:15:32.565501+0000","last_undegraded":"2026-02-20T11:15:32.565501+0000","last_fullsized":"2026-02-20T11:15:32.565501+0000","mapping_epoch":16,"log_start":"22'20","ondisk_log_start":"22'20","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":20,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":11447193,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":7094,"num_write":30,"num_write_kb":7565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"19'3","reported_seq":36,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-20T11:15:31.529546+0000","last_change":"2026-02-20T11:15:30.529422+0000","last_active":"2026-02-20T11:15:31.529546+0000","last_peered":"2026-02-20T11:15:31.529546+0000","last_clean":"2026-02-20T11:15:31.529546+0000","last_became_active":"2026-02-20T11:15:24.599300+0000","last_became_peered":"2026-02-20T11:15:24.599300+0000","last_unstale":"2026-02-20T11:15:31.529546+0000","last_undegraded":"2026-02-20T11:15:31.529546+0000","last_fullsized":"2026-02-20T11:15:31.529546+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2756315,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"24'29","reported_seq":66,"reported_epoch":25,"state":"active+clean+snaptrim","last_fresh":"2026-02-20T11:15:32.532813+0000","last_change":"2026-02-20T11:15:30.526771+0000","last_active":"2026-02-20T11:15:32.532813+0000","last_peered":"2026-02-20T11:15:32.532813+0000","last_clean":"2026-02-20T11:15:32.532813+0000","last_became_active":"2026-02-20T11:15:24.577320+0000","last_became_peered":"2026-02-20T11:15:24.577320+0000","last_unstale":"2026-02-20T11:15:32.532813+0000","last_undegraded":"2026-02-20T11:15:32.532813+0000","last_fullsized":"2026-02-20T11:15:32.532813+0000","mapping_epoch":16,"log_start":"23'19","ondisk_log_start":"23'19","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":19,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":9752928,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":3392,"num_write":31,"num_write_kb":7678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"23'12","reported_seq":55,"reported_epoch":25,"state":"active+clean","last_fresh":"2026-02-20T11:15:32.571521+0000","last_change":"2026-02-20T11:15:32.545136+0000","last_active":"2026-02-20T11:15:32.571521+0000","last_peered":"2026-02-20T11:15:32.571521+0000","last_clean":"2026-02-20T11:15:32.571521+0000","last_became_active":"2026-02-20T11:15:24.555345+0000","last_became_peered":"2026-02-20T11:15:24.555345+0000","last_unstale":"2026-02-20T11:15:32.571521+0000","last_undegraded":"2026-02-20T11:15:32.571521+0000","last_fullsized":"2026-02-20T11:15:32.571521+0000","mapping_epoch":16,"log_start":"19'8","ondisk_log_start":"19'8","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":8,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7831963,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":4402,"num_write":16,"num_write_kb":4796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"19'6","reported_seq":43,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-20T11:15:31.530151+0000","last_change":"2026-02-20T11:15:30.529377+0000","last_active":"2026-02-20T11:15:31.530151+0000","last_peered":"2026-02-20T11:15:31.530151+0000","last_clean":"2026-02-20T11:15:31.530151+0000","last_became_active":"2026-02-20T11:15:24.556605+0000","last_became_peered":"2026-02-20T11:15:24.556605+0000","last_unstale":"2026-02-20T11:15:31.530151+0000","last_undegraded":"2026-02-20T11:15:31.530151+0000","last_fullsized":"2026-02-20T11:15:31.530151+0000","mapping_epoch":16,"log_start":"19'2","ondisk_log_start":"19'2","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":2,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4789929,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":2704,"num_write":8,"num_write_kb":2146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":46,"reported_epoch":25,"state":"active+clean","last_fresh":"2026-02-20T11:15:32.533131+0000","last_change":"2026-02-20T11:15:16.984535+0000","last_active":"2026-02-20T11:15:32.533131+0000","last_peered":"2026-02-20T11:15:32.533131+0000","last_clean":"2026-02-20T11:15:32.533131+0000","last_became_active":"2026-02-20T11:15:15.006596+0000","last_became_peered":"2026-02-20T11:15:15.006596+0000","last_unstale":"2026-02-20T11:15:32.533131+0000","last_undegraded":"2026-02-20T11:15:32.533131+0000","last_fullsized":"2026-02-20T11:15:32.533131+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"23'25","reported_seq":64,"reported_epoch":24,"state":"active+clean+snaptrim","last_fresh":"2026-02-20T11:15:31.529873+0000","last_change":"2026-02-20T11:15:30.526177+0000","last_active":"2026-02-20T11:15:31.529873+0000","last_peered":"2026-02-20T11:15:31.529873+0000","last_clean":"2026-02-20T11:15:31.529873+0000","last_became_active":"2026-02-20T11:15:24.550094+0000","last_became_peered":"2026-02-20T11:15:24.550094+0000","last_unstale":"2026-02-20T11:15:31.529873+0000","last_undegraded":"2026-02-20T11:15:31.529873+0000","last_fullsized":"2026-02-20T11:15:31.529873+0000","mapping_epoch":16,"log_start":"19'18","ondisk_log_start":"19'18","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":18,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":13490225,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":6747,"num_write":30,"num_write_kb":8687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":46,"reported_epoch":25,"state":"active+clean","last_fresh":"2026-02-20T11:15:32.534957+0000","last_change":"2026-02-20T11:15:16.985758+0000","last_active":"2026-02-20T11:15:32.534957+0000","last_peered":"2026-02-20T11:15:32.534957+0000","last_clean":"2026-02-20T11:15:32.534957+0000","last_became_active":"2026-02-20T11:15:15.019055+0000","last_became_peered":"2026-02-20T11:15:15.019055+0000","last_unstale":"2026-02-20T11:15:32.534957+0000","last_undegraded":"2026-02-20T11:15:32.534957+0000","last_fullsized":"2026-02-20T11:15:32.534957+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"19'9","reported_seq":36,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-20T11:15:27.620353+0000","last_change":"2026-02-20T11:15:24.594862+0000","last_active":"2026-02-20T11:15:27.620353+0000","last_peered":"2026-02-20T11:15:27.620353+0000","last_clean":"2026-02-20T11:15:27.620353+0000","last_became_active":"2026-02-20T11:15:24.580572+0000","last_became_peered":"2026-02-20T11:15:24.580572+0000","last_unstale":"2026-02-20T11:15:27.620353+0000","last_undegraded":"2026-02-20T11:15:27.620353+0000","last_fullsized":"2026-02-20T11:15:27.620353+0000","mapping_epoch":16,"log_start":"19'5","ondisk_log_start":"19'5","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":5,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8100875,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":8525,"num_write":12,"num_write_kb":3878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'2","reported_seq":33,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-20T11:15:27.511727+0000","last_change":"2026-02-20T11:15:16.983646+0000","last_active":"2026-02-20T11:15:27.511727+0000","last_peered":"2026-02-20T11:15:27.511727+0000","last_clean":"2026-02-20T11:15:27.511727+0000","last_became_active":"2026-02-20T11:15:15.012729+0000","last_became_peered":"2026-02-20T11:15:15.012729+0000","last_unstale":"2026-02-20T11:15:27.511727+0000","last_undegraded":"2026-02-20T11:15:27.511727+0000","last_fullsized":"2026-02-20T11:15:27.511727+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"25'12","reported_seq":52,"reported_epoch":25,"state":"active+clean","last_fresh":"2026-02-20T11:15:32.548905+0000","last_change":"2026-02-20T11:15:32.540936+0000","last_active":"2026-02-20T11:15:32.548905+0000","last_peered":"2026-02-20T11:15:32.548905+0000","last_clean":"2026-02-20T11:15:32.548905+0000","last_became_active":"2026-02-20T11:15:24.590844+0000","last_became_peered":"2026-02-20T11:15:24.590844+0000","last_unstale":"2026-02-20T11:15:32.548905+0000","last_undegraded":"2026-02-20T11:15:32.548905+0000","last_fullsized":"2026-02-20T11:15:32.548905+0000","mapping_epoch":16,"log_start":"19'8","ondisk_log_start":"19'8","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":8,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5462209,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":17,"num_write_kb":4237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":44,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-20T11:15:31.529115+0000","last_change":"2026-02-20T11:15:16.983291+0000","last_active":"2026-02-20T11:15:31.529115+0000","last_peered":"2026-02-20T11:15:31.529115+0000","last_clean":"2026-02-20T11:15:31.529115+0000","last_became_active":"2026-02-20T11:15:15.009983+0000","last_became_peered":"2026-02-20T11:15:15.009983+0000","last_unstale":"2026-02-20T11:15:31.529115+0000","last_undegraded":"2026-02-20T11:15:31.529115+0000","last_fullsized":"2026-02-20T11:15:31.529115+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'52","reported_seq":112,"reported_epoch":25,"state":"active+clean","last_fresh":"2026-02-20T11:15:32.531880+0000","last_change":"2026-02-20T11:15:15.035494+0000","last_active":"2026-02-20T11:15:32.531880+0000","last_peered":"2026-02-20T11:15:32.531880+0000","last_clean":"2026-02-20T11:15:32.531880+0000","last_became_active":"2026-02-20T11:15:15.012645+0000","last_became_peered":"2026-02-20T11:15:15.012645+0000","last_unstale":"2026-02-20T11:15:32.531880+0000","last_undegraded":"2026-02-20T11:15:32.531880+0000","last_fullsized":"2026-02-20T11:15:32.531880+0000","mapping_epoch":12,"log_start":"13'48","ondisk_log_start":"13'48","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":48,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"19'3","reported_seq":40,"reported_epoch":25,"state":"active+clean","last_fresh":"2026-02-20T11:15:32.534354+0000","last_change":"2026-02-20T11:15:32.537444+0000","last_active":"2026-02-20T11:15:32.534354+0000","last_peered":"2026-02-20T11:15:32.534354+0000","last_clean":"2026-02-20T11:15:32.534354+0000","last_became_active":"2026-02-20T11:15:24.575441+0000","last_became_peered":"2026-02-20T11:15:24.575441+0000","last_unstale":"2026-02-20T11:15:32.534354+0000","last_undegraded":"2026-02-20T11:15:32.534354+0000","last_fullsized":"2026-02-20T11:15:32.534354+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2521017,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"19'6","reported_seq":32,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-20T11:15:27.612512+0000","last_change":"2026-02-20T11:15:24.617223+0000","last_active":"2026-02-20T11:15:27.612512+0000","last_peered":"2026-02-20T11:15:27.612512+0000","last_clean":"2026-02-20T11:15:27.612512+0000","last_became_active":"2026-02-20T11:15:24.612337+0000","last_became_peered":"2026-02-20T11:15:24.612337+0000","last_unstale":"2026-02-20T11:15:27.612512+0000","last_undegraded":"2026-02-20T11:15:27.612512+0000","last_fullsized":"2026-02-20T11:15:27.612512+0000","mapping_epoch":16,"log_start":"19'2","ondisk_log_start":"19'2","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":2,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":7941,"num_write":8,"num_write_kb":2298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":43,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-20T11:15:31.530055+0000","last_change":"2026-02-20T11:15:16.984208+0000","last_active":"2026-02-20T11:15:31.530055+0000","last_peered":"2026-02-20T11:15:31.530055+0000","last_clean":"2026-02-20T11:15:31.530055+0000","last_became_active":"2026-02-20T11:15:15.027750+0000","last_became_peered":"2026-02-20T11:15:15.027750+0000","last_unstale":"2026-02-20T11:15:31.530055+0000","last_undegraded":"2026-02-20T11:15:31.530055+0000","last_fullsized":"2026-02-20T11:15:31.530055+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"25'25","reported_seq":65,"reported_epoch":25,"state":"active+clean+snaptrim","last_fresh":"2026-02-20T11:15:32.555117+0000","last_change":"2026-02-20T11:15:32.533801+0000","last_active":"2026-02-20T11:15:32.555117+0000","last_peered":"2026-02-20T11:15:32.555117+0000","last_clean":"2026-02-20T11:15:32.555117+0000","last_became_active":"2026-02-20T11:15:24.559859+0000","last_became_peered":"2026-02-20T11:15:24.559859+0000","last_unstale":"2026-02-20T11:15:32.555117+0000","last_undegraded":"2026-02-20T11:15:32.555117+0000","last_fullsized":"2026-02-20T11:15:32.555117+0000","mapping_epoch":16,"log_start":"24'17","ondisk_log_start":"24'17","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":17,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0,"stat_sum":{"num_bytes":3819627,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":17,"num_read_kb":2018,"num_write":25,"num_write_kb":6569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":31,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-20T11:15:27.512747+0000","last_change":"2026-02-20T11:15:16.985671+0000","last_active":"2026-02-20T11:15:27.512747+0000","last_peered":"2026-02-20T11:15:27.512747+0000","last_clean":"2026-02-20T11:15:27.512747+0000","last_became_active":"2026-02-20T11:15:15.003549+0000","last_became_peered":"2026-02-20T11:15:15.003549+0000","last_unstale":"2026-02-20T11:15:27.512747+0000","last_undegraded":"2026-02-20T11:15:27.512747+0000","last_fullsized":"2026-02-20T11:15:27.512747+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"19'12","reported_seq":34,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-20T11:15:27.511066+0000","last_change":"2026-02-20T11:15:24.560282+0000","last_active":"2026-02-20T11:15:27.511066+0000","last_peered":"2026-02-20T11:15:27.511066+0000","last_clean":"2026-02-20T11:15:27.511066+0000","last_became_active":"2026-02-20T11:15:24.543822+0000","last_became_peered":"2026-02-20T11:15:24.543822+0000","last_unstale":"2026-02-20T11:15:27.511066+0000","last_undegraded":"2026-02-20T11:15:27.511066+0000","last_fullsized":"2026-02-20T11:15:27.511066+0000","mapping_epoch":16,"log_start":"19'8","ondisk_log_start":"19'8","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":8,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9344560,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":46,"reported_epoch":25,"state":"active+clean","last_fresh":"2026-02-20T11:15:32.535222+0000","last_change":"2026-02-20T11:15:16.986873+0000","last_active":"2026-02-20T11:15:32.535222+0000","last_peered":"2026-02-20T11:15:32.535222+0000","last_clean":"2026-02-20T11:15:32.535222+0000","last_became_active":"2026-02-20T11:15:15.018131+0000","last_became_peered":"2026-02-20T11:15:15.018131+0000","last_unstale":"2026-02-20T11:15:32.535222+0000","last_undegraded":"2026-02-20T11:15:32.535222+0000","last_fullsized":"2026-02-20T11:15:32.535222+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"19'22","reported_seq":47,"reported_epoch":20,"state":"active+clean","last_fresh":"2026-02-20T11:15:27.510365+0000","last_change":"2026-02-20T11:15:24.590252+0000","last_active":"2026-02-20T11:15:27.510365+0000","last_peered":"2026-02-20T11:15:27.510365+0000","last_clean":"2026-02-20T11:15:27.510365+0000","last_became_active":"2026-02-20T11:15:24.578162+0000","last_became_peered":"2026-02-20T11:15:24.578162+0000","last_unstale":"2026-02-20T11:15:27.510365+0000","last_undegraded":"2026-02-20T11:15:27.510365+0000","last_fullsized":"2026-02-20T11:15:27.510365+0000","mapping_epoch":16,"log_start":"19'18","ondisk_log_start":"19'18","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:23.488435+0000","last_clean_scrub_stamp":"2026-02-20T11:15:23.488435+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":18,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13916031,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":7,"num_write":29,"num_write_kb":8343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":46,"reported_epoch":25,"state":"active+clean","last_fresh":"2026-02-20T11:15:32.535049+0000","last_change":"2026-02-20T11:15:16.986570+0000","last_active":"2026-02-20T11:15:32.535049+0000","last_peered":"2026-02-20T11:15:32.535049+0000","last_clean":"2026-02-20T11:15:32.535049+0000","last_became_active":"2026-02-20T11:15:15.057391+0000","last_became_peered":"2026-02-20T11:15:15.057391+0000","last_unstale":"2026-02-20T11:15:32.535049+0000","last_undegraded":"2026-02-20T11:15:32.535049+0000","last_fullsized":"2026-02-20T11:15:32.535049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:15:13.957789+0000","last_clean_scrub_stamp":"2026-02-20T11:15:13.957789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":109821150,"num_objects":51,"num_object_clones":4,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":181,"num_read_kb":51145,"num_write":260,"num_write_kb":70557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214748299264,"available":128624242688,"internally_reserved":0,"allocated":86124056576,"data_stored":86124056576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32132595712,"internally_reserved":0,"allocated":21554479104,"data_stored":21554479104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96452513792,"internally_reserved":0,"allocated":64608710656,"data_stored":64608710656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640275,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":21050712,"kb_used_data":21050712,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31378072,"statfs":{"total":53687074816,"available":32131145728,"internally_reserved":0,"allocated":21555929088,"data_stored":21555929088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640270,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":21007524,"kb_used_data":21007524,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31421260,"statfs":{"total":53687074816,"available":32175370240,"internally_reserved":0,"allocated":21511704576,"data_stored":21511704576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672979,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":21017596,"kb_used_data":21017596,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31411188,"statfs":{"total":53687074816,"available":32165056512,"internally_reserved":0,"allocated":21522018304,"data_stored":21522018304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640276,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":21011080,"kb_used_data":21011080,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31417704,"statfs":{"total":53687074816,"available":32171728896,"internally_reserved":0,"allocated":21515345920,"data_stored":21515345920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32159744000,"internally_reserved":0,"allocated":21527330816,"data_stored":21527330816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53687074816,"available":32160174080,"internally_reserved":0,"allocated":21526900736,"data_stored":21526900736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":53687074816,"available":32132595712,"internally_reserved":0,"allocated":21554479104,"data_stored":21554479104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53687074816,"available":32132595712,"internally_reserved":0,"allocated":21554479104,"data_stored":21554479104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53687074816,"available":32171728896,"internally_reserved":0,"allocated":21515345920,"data_stored":21515345920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53687074816,"available":32159744000,"internally_reserved":0,"allocated":21527330816,"data_stored":21527330816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53687074816,"available":32160174080,"internally_reserved":0,"allocated":21526900736,"data_stored":21526900736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53687074816,"available":32132595712,"internally_reserved":0,"allocated":21554479104,"data_stored":21554479104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-20T11:15:40.910 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3] 2026-02-20T11:15:40.910 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-20T11:15:40.911 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2026-02-20T11:15:40.911 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-20T11:15:40.911 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2026-02-20T11:15:40.914 INFO:tasks.ceph.osd.2:Started 2026-02-20T11:15:40.914 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:15:40.988 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:15:40.991 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:15:40.992 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:15:45.326 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2026-02-20T11:15:45.993 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:15:46.064 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:15:46.066 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:15:46.067 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:15:50.829 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~6s 2026-02-20T11:15:51.068 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:15:51.140 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:15:51.143 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:15:51.143 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:15:56.144 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:15:56.215 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:15:56.218 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:15:56.219 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:15:56.333 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~11s 2026-02-20T11:16:00.742 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:16:00.781 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:16:00.782 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:16:01.220 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:16:01.290 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:16:01.293 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:16:01.294 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:16:01.799 INFO:tasks.rados.rados.0.trial043.stdout:80: finishing write tid 1 to trial04342147-7 2026-02-20T11:16:01.801 INFO:tasks.rados.rados.0.trial043.stdout:80: finishing write tid 2 to trial04342147-7 2026-02-20T11:16:01.807 INFO:tasks.rados.rados.0.trial043.stdout:80: finishing write tid 3 to trial04342147-7 2026-02-20T11:16:01.807 INFO:tasks.rados.rados.0.trial043.stdout:90: finishing write tid 1 to trial04342147-34 2026-02-20T11:16:01.807 INFO:tasks.rados.rados.0.trial043.stdout:90: finishing write tid 2 to trial04342147-34 2026-02-20T11:16:01.807 INFO:tasks.rados.rados.0.trial043.stdout:80: finishing write tid 4 to trial04342147-7 2026-02-20T11:16:01.808 INFO:tasks.rados.rados.0.trial043.stdout:88: finishing write tid 1 to trial04342147-10 2026-02-20T11:16:01.808 INFO:tasks.rados.rados.0.trial043.stdout:90: finishing write tid 3 to trial04342147-34 2026-02-20T11:16:01.809 INFO:tasks.rados.rados.0.trial043.stdout:88: finishing write tid 2 to trial04342147-10 2026-02-20T11:16:01.809 INFO:tasks.rados.rados.0.trial043.stdout:92: done (12 left) 2026-02-20T11:16:01.809 INFO:tasks.rados.rados.0.trial043.stdout:93: done (11 left) 2026-02-20T11:16:01.809 INFO:tasks.rados.rados.0.trial043.stdout:96: done (10 left) 2026-02-20T11:16:01.809 INFO:tasks.rados.rados.0.trial043.stdout:97: read oid 33 snap -1 2026-02-20T11:16:01.809 INFO:tasks.rados.rados.0.trial043.stdout:97: expect (ObjNum 32 snap 0 seq_num 32) 2026-02-20T11:16:01.809 INFO:tasks.rados.rados.0.trial043.stdout:98: write oid 12 current snap is 3 2026-02-20T11:16:01.810 INFO:tasks.rados.rados.0.trial043.stdout:98: seq_num 73 ranges {426911=540354,1622009=726683,3121996=104915} 2026-02-20T11:16:01.812 INFO:tasks.rados.rados.0.trial043.stdout:98: writing trial04342147-12 from 426911 to 967265 tid 1 2026-02-20T11:16:01.817 INFO:tasks.rados.rados.0.trial043.stdout:98: writing trial04342147-12 from 1622009 to 2348692 tid 2 2026-02-20T11:16:01.821 INFO:tasks.rados.rados.0.trial043.stdout:98: writing trial04342147-12 from 3121996 to 3226911 tid 3 2026-02-20T11:16:01.821 INFO:tasks.rados.rados.0.trial043.stdout:99: read oid 27 snap -1 2026-02-20T11:16:01.821 INFO:tasks.rados.rados.0.trial043.stdout:99: expect (ObjNum 67 snap 3 seq_num 67) 2026-02-20T11:16:01.822 INFO:tasks.rados.rados.0.trial043.stdout:100: write oid 23 current snap is 3 2026-02-20T11:16:01.822 INFO:tasks.rados.rados.0.trial043.stdout:88: finishing write tid 3 to trial04342147-10 2026-02-20T11:16:01.822 INFO:tasks.rados.rados.0.trial043.stdout:100: seq_num 74 ranges {443718=615603,1564138=470341,2633066=610652} 2026-02-20T11:16:01.825 INFO:tasks.rados.rados.0.trial043.stdout:100: writing trial04342147-23 from 443718 to 1059321 tid 1 2026-02-20T11:16:01.829 INFO:tasks.rados.rados.0.trial043.stdout:100: writing trial04342147-23 from 1564138 to 2034479 tid 2 2026-02-20T11:16:01.834 INFO:tasks.rados.rados.0.trial043.stdout:100: writing trial04342147-23 from 2633066 to 3243718 tid 3 2026-02-20T11:16:01.834 INFO:tasks.rados.rados.0.trial043.stdout:101: write oid 4 current snap is 3 2026-02-20T11:16:01.834 INFO:tasks.rados.rados.0.trial043.stdout:88: finishing write tid 4 to trial04342147-10 2026-02-20T11:16:01.834 INFO:tasks.rados.rados.0.trial043.stdout:90: finishing write tid 4 to trial04342147-34 2026-02-20T11:16:01.834 INFO:tasks.rados.rados.0.trial043.stdout:90: oid 34 updating version 0 to 10 2026-02-20T11:16:01.834 INFO:tasks.rados.rados.0.trial043.stdout:90: oid 34 updating version 10 to 12 2026-02-20T11:16:01.834 INFO:tasks.rados.rados.0.trial043.stdout:90: oid 34 version 12 is already newer than 11 2026-02-20T11:16:01.834 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 34 v 12 (ObjNum 70 snap 3 seq_num 70) dirty exists 2026-02-20T11:16:01.835 INFO:tasks.rados.rados.0.trial043.stdout:90: left oid 34 (ObjNum 70 snap 3 seq_num 70) 2026-02-20T11:16:01.835 INFO:tasks.rados.rados.0.trial043.stdout:101: seq_num 75 ranges {460525=690852,1622620=730352,3060489=200036} 2026-02-20T11:16:01.838 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~17s 2026-02-20T11:16:01.838 INFO:tasks.rados.rados.0.trial043.stdout:101: writing trial04342147-4 from 460525 to 1151377 tid 1 2026-02-20T11:16:01.842 INFO:tasks.rados.rados.0.trial043.stdout:101: writing trial04342147-4 from 1622620 to 2352972 tid 2 2026-02-20T11:16:01.846 INFO:tasks.rados.rados.0.trial043.stdout:101: writing trial04342147-4 from 3060489 to 3260525 tid 3 2026-02-20T11:16:01.847 INFO:tasks.rados.rados.0.trial043.stdout:90: done (14 left) 2026-02-20T11:16:01.847 INFO:tasks.rados.rados.0.trial043.stdout:102: delete oid 20 current snap is 3 2026-02-20T11:16:01.847 INFO:tasks.rados.rados.0.trial043.stdout:80: finishing write tid 5 to trial04342147-7 2026-02-20T11:16:01.847 INFO:tasks.rados.rados.0.trial043.stdout:80: oid 7 updating version 0 to 21 2026-02-20T11:16:01.847 INFO:tasks.rados.rados.0.trial043.stdout:80: oid 7 updating version 21 to 24 2026-02-20T11:16:01.847 INFO:tasks.rados.rados.0.trial043.stdout:80: oid 7 version 24 is already newer than 23 2026-02-20T11:16:01.847 INFO:tasks.rados.rados.0.trial043.stdout:80: oid 7 version 24 is already newer than 22 2026-02-20T11:16:01.847 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 7 v 24 (ObjNum 63 snap 3 seq_num 63) dirty exists 2026-02-20T11:16:01.847 INFO:tasks.rados.rados.0.trial043.stdout:80: left oid 7 (ObjNum 63 snap 3 seq_num 63) 2026-02-20T11:16:01.848 INFO:tasks.rados.rados.0.trial043.stdout:88: finishing write tid 5 to trial04342147-10 2026-02-20T11:16:01.848 INFO:tasks.rados.rados.0.trial043.stdout:88: oid 10 updating version 0 to 28 2026-02-20T11:16:01.848 INFO:tasks.rados.rados.0.trial043.stdout:88: oid 10 version 28 is already newer than 26 2026-02-20T11:16:01.848 INFO:tasks.rados.rados.0.trial043.stdout:88: oid 10 version 28 is already newer than 27 2026-02-20T11:16:01.848 INFO:tasks.rados.rados.0.trial043.stdout:88: oid 10 updating version 28 to 29 2026-02-20T11:16:01.848 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 10 v 29 (ObjNum 69 snap 3 seq_num 69) dirty exists 2026-02-20T11:16:01.848 INFO:tasks.rados.rados.0.trial043.stdout:88: left oid 10 (ObjNum 69 snap 3 seq_num 69) 2026-02-20T11:16:01.856 INFO:tasks.rados.rados.0.trial043.stdout:83: finishing write tid 1 to trial04342147-45 2026-02-20T11:16:01.859 INFO:tasks.rados.rados.0.trial043.stdout:97: expect (ObjNum 32 snap 0 seq_num 32) 2026-02-20T11:16:01.862 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 707925 2026-02-20T11:16:01.899 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:16:01.921 INFO:tasks.rados.rados.0.trial043.stdout:94: expect (ObjNum 30 snap 0 seq_num 30) 2026-02-20T11:16:01.921 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:16:01.977 INFO:tasks.rados.rados.0.trial043.stdout:95: finishing write tid 1 to trial04342147-39 2026-02-20T11:16:01.977 INFO:tasks.rados.rados.0.trial043.stdout:81: finishing write tid 1 to trial04342147-22 2026-02-20T11:16:01.978 INFO:tasks.rados.rados.0.trial043.stdout:79: finishing write tid 1 to trial04342147-6 2026-02-20T11:16:01.978 INFO:tasks.rados.rados.0.trial043.stdout:80: done (14 left) 2026-02-20T11:16:01.978 INFO:tasks.rados.rados.0.trial043.stdout:88: done (13 left) 2026-02-20T11:16:01.978 INFO:tasks.rados.rados.0.trial043.stdout:94: done (12 left) 2026-02-20T11:16:01.978 INFO:tasks.rados.rados.0.trial043.stdout:97: done (11 left) 2026-02-20T11:16:01.978 INFO:tasks.rados.rados.0.trial043.stdout:102: done (10 left) 2026-02-20T11:16:01.978 INFO:tasks.rados.rados.0.trial043.stdout:103: read oid 24 snap -1 2026-02-20T11:16:01.978 INFO:tasks.rados.rados.0.trial043.stdout:103: expect deleted 2026-02-20T11:16:01.978 INFO:tasks.rados.rados.0.trial043.stdout:104: delete oid 47 current snap is 3 2026-02-20T11:16:01.978 INFO:tasks.rados.rados.0.trial043.stdout:81: finishing write tid 2 to trial04342147-22 2026-02-20T11:16:01.978 INFO:tasks.rados.rados.0.trial043.stdout:86: finishing write tid 1 to trial04342147-49 2026-02-20T11:16:01.979 INFO:tasks.rados.rados.0.trial043.stdout:79: finishing write tid 2 to trial04342147-6 2026-02-20T11:16:01.979 INFO:tasks.rados.rados.0.trial043.stdout:81: finishing write tid 3 to trial04342147-22 2026-02-20T11:16:01.979 INFO:tasks.rados.rados.0.trial043.stdout:99: expect (ObjNum 67 snap 3 seq_num 67) 2026-02-20T11:16:02.065 INFO:tasks.rados.rados.0.trial043.stdout:81: finishing write tid 4 to trial04342147-22 2026-02-20T11:16:02.065 INFO:tasks.rados.rados.0.trial043.stdout:81: oid 22 updating version 0 to 15 2026-02-20T11:16:02.065 INFO:tasks.rados.rados.0.trial043.stdout:81: oid 22 version 15 is already newer than 15 2026-02-20T11:16:02.065 INFO:tasks.rados.rados.0.trial043.stdout:81: oid 22 version 15 is already newer than 15 2026-02-20T11:16:02.066 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 22 v 15 (ObjNum 64 snap 3 seq_num 64) dirty exists 2026-02-20T11:16:02.066 INFO:tasks.rados.rados.0.trial043.stdout:81: left oid 22 (ObjNum 64 snap 3 seq_num 64) 2026-02-20T11:16:02.066 INFO:tasks.rados.rados.0.trial043.stdout:101: finishing write tid 1 to trial04342147-4 2026-02-20T11:16:02.066 INFO:tasks.rados.rados.0.trial043.stdout:100: finishing write tid 1 to trial04342147-23 2026-02-20T11:16:02.066 INFO:tasks.rados.rados.0.trial043.stdout:101: finishing write tid 2 to trial04342147-4 2026-02-20T11:16:02.066 INFO:tasks.rados.rados.0.trial043.stdout:101: finishing write tid 3 to trial04342147-4 2026-02-20T11:16:02.066 INFO:tasks.rados.rados.0.trial043.stdout:79: finishing write tid 3 to trial04342147-6 2026-02-20T11:16:02.066 INFO:tasks.rados.rados.0.trial043.stdout:95: finishing write tid 2 to trial04342147-39 2026-02-20T11:16:02.066 INFO:tasks.rados.rados.0.trial043.stdout:79: finishing write tid 4 to trial04342147-6 2026-02-20T11:16:02.066 INFO:tasks.rados.rados.0.trial043.stdout:79: oid 6 updating version 0 to 13 2026-02-20T11:16:02.066 INFO:tasks.rados.rados.0.trial043.stdout:79: oid 6 version 13 is already newer than 13 2026-02-20T11:16:02.066 INFO:tasks.rados.rados.0.trial043.stdout:79: oid 6 version 13 is already newer than 13 2026-02-20T11:16:02.066 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 6 v 13 (ObjNum 62 snap 3 seq_num 62) dirty exists 2026-02-20T11:16:02.067 INFO:tasks.rados.rados.0.trial043.stdout:79: left oid 6 (ObjNum 62 snap 3 seq_num 62) 2026-02-20T11:16:02.067 INFO:tasks.rados.rados.0.trial043.stdout:101: finishing write tid 4 to trial04342147-4 2026-02-20T11:16:02.067 INFO:tasks.rados.rados.0.trial043.stdout:79: done (11 left) 2026-02-20T11:16:02.067 INFO:tasks.rados.rados.0.trial043.stdout:81: done (10 left) 2026-02-20T11:16:02.067 INFO:tasks.rados.rados.0.trial043.stdout:99: done (9 left) 2026-02-20T11:16:02.067 INFO:tasks.rados.rados.0.trial043.stdout:104: done (8 left) 2026-02-20T11:16:02.067 INFO:tasks.rados.rados.0.trial043.stdout:105: read oid 13 snap -1 2026-02-20T11:16:02.067 INFO:tasks.rados.rados.0.trial043.stdout:105: expect (ObjNum 12 snap 0 seq_num 12) 2026-02-20T11:16:02.067 INFO:tasks.rados.rados.0.trial043.stdout:106: snap_create 2026-02-20T11:16:02.067 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 9 2026-02-20T11:16:02.067 INFO:tasks.rados.rados.0.trial043.stdout:98: finishing write tid 1 to trial04342147-12 2026-02-20T11:16:02.067 INFO:tasks.rados.rados.0.trial043.stdout:101: finishing write tid 5 to trial04342147-4 2026-02-20T11:16:02.067 INFO:tasks.rados.rados.0.trial043.stdout:101: oid 4 updating version 0 to 19 2026-02-20T11:16:02.067 INFO:tasks.rados.rados.0.trial043.stdout:101: oid 4 version 19 is already newer than 19 2026-02-20T11:16:02.068 INFO:tasks.rados.rados.0.trial043.stdout:101: oid 4 version 19 is already newer than 19 2026-02-20T11:16:02.068 INFO:tasks.rados.rados.0.trial043.stdout:101: oid 4 version 19 is already newer than 19 2026-02-20T11:16:02.068 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 4 v 19 (ObjNum 75 snap 3 seq_num 75) dirty exists 2026-02-20T11:16:02.068 INFO:tasks.rados.rados.0.trial043.stdout:101: left oid 4 (ObjNum 75 snap 3 seq_num 75) 2026-02-20T11:16:02.068 INFO:tasks.rados.rados.0.trial043.stdout:98: finishing write tid 2 to trial04342147-12 2026-02-20T11:16:02.068 INFO:tasks.rados.rados.0.trial043.stdout:83: finishing write tid 2 to trial04342147-45 2026-02-20T11:16:02.068 INFO:tasks.rados.rados.0.trial043.stdout:95: finishing write tid 3 to trial04342147-39 2026-02-20T11:16:02.068 INFO:tasks.rados.rados.0.trial043.stdout:95: finishing write tid 4 to trial04342147-39 2026-02-20T11:16:02.068 INFO:tasks.rados.rados.0.trial043.stdout:95: oid 39 updating version 0 to 9 2026-02-20T11:16:02.068 INFO:tasks.rados.rados.0.trial043.stdout:95: oid 39 version 9 is already newer than 9 2026-02-20T11:16:02.068 INFO:tasks.rados.rados.0.trial043.stdout:95: oid 39 version 9 is already newer than 9 2026-02-20T11:16:02.068 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 39 v 9 (ObjNum 71 snap 3 seq_num 71) dirty exists 2026-02-20T11:16:02.068 INFO:tasks.rados.rados.0.trial043.stdout:95: left oid 39 (ObjNum 71 snap 3 seq_num 71) 2026-02-20T11:16:02.068 INFO:tasks.rados.rados.0.trial043.stdout:98: finishing write tid 3 to trial04342147-12 2026-02-20T11:16:02.069 INFO:tasks.rados.rados.0.trial043.stdout:86: finishing write tid 2 to trial04342147-49 2026-02-20T11:16:02.069 INFO:tasks.rados.rados.0.trial043.stdout:98: finishing write tid 4 to trial04342147-12 2026-02-20T11:16:02.069 INFO:tasks.rados.rados.0.trial043.stdout:86: finishing write tid 3 to trial04342147-49 2026-02-20T11:16:02.069 INFO:tasks.rados.rados.0.trial043.stdout:95: done (8 left) 2026-02-20T11:16:02.069 INFO:tasks.rados.rados.0.trial043.stdout:101: done (7 left) 2026-02-20T11:16:02.069 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 7 2026-02-20T11:16:02.069 INFO:tasks.rados.rados.0.trial043.stdout:98: finishing write tid 5 to trial04342147-12 2026-02-20T11:16:02.069 INFO:tasks.rados.rados.0.trial043.stdout:98: oid 12 updating version 0 to 32 2026-02-20T11:16:02.069 INFO:tasks.rados.rados.0.trial043.stdout:98: oid 12 version 32 is already newer than 32 2026-02-20T11:16:02.069 INFO:tasks.rados.rados.0.trial043.stdout:98: oid 12 version 32 is already newer than 32 2026-02-20T11:16:02.069 INFO:tasks.rados.rados.0.trial043.stdout:98: oid 12 version 32 is already newer than 32 2026-02-20T11:16:02.069 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 12 v 32 (ObjNum 73 snap 3 seq_num 73) dirty exists 2026-02-20T11:16:02.069 INFO:tasks.rados.rados.0.trial043.stdout:98: left oid 12 (ObjNum 73 snap 3 seq_num 73) 2026-02-20T11:16:02.069 INFO:tasks.rados.rados.0.trial043.stdout:86: finishing write tid 4 to trial04342147-49 2026-02-20T11:16:02.070 INFO:tasks.rados.rados.0.trial043.stdout:91: expect (ObjNum 2 snap 0 seq_num 2) 2026-02-20T11:16:02.094 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1364431 2026-02-20T11:16:02.104 INFO:tasks.rados.rados.0.trial043.stdout:86: finishing write tid 5 to trial04342147-49 2026-02-20T11:16:02.105 INFO:tasks.rados.rados.0.trial043.stdout:86: oid 49 updating version 0 to 13 2026-02-20T11:16:02.105 INFO:tasks.rados.rados.0.trial043.stdout:86: oid 49 version 13 is already newer than 13 2026-02-20T11:16:02.105 INFO:tasks.rados.rados.0.trial043.stdout:86: oid 49 version 13 is already newer than 13 2026-02-20T11:16:02.105 INFO:tasks.rados.rados.0.trial043.stdout:86: oid 49 version 13 is already newer than 13 2026-02-20T11:16:02.105 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 49 v 13 (ObjNum 68 snap 3 seq_num 68) dirty exists 2026-02-20T11:16:02.105 INFO:tasks.rados.rados.0.trial043.stdout:86: left oid 49 (ObjNum 68 snap 3 seq_num 68) 2026-02-20T11:16:02.105 INFO:tasks.rados.rados.0.trial043.stdout:86: done (6 left) 2026-02-20T11:16:02.105 INFO:tasks.rados.rados.0.trial043.stdout:91: done (5 left) 2026-02-20T11:16:02.105 INFO:tasks.rados.rados.0.trial043.stdout:98: done (4 left) 2026-02-20T11:16:02.105 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 4 2026-02-20T11:16:02.106 INFO:tasks.rados.rados.0.trial043.stdout:100: finishing write tid 2 to trial04342147-23 2026-02-20T11:16:02.106 INFO:tasks.rados.rados.0.trial043.stdout:100: finishing write tid 3 to trial04342147-23 2026-02-20T11:16:02.106 INFO:tasks.rados.rados.0.trial043.stdout:100: finishing write tid 4 to trial04342147-23 2026-02-20T11:16:02.106 INFO:tasks.rados.rados.0.trial043.stdout:100: finishing write tid 5 to trial04342147-23 2026-02-20T11:16:02.106 INFO:tasks.rados.rados.0.trial043.stdout:100: oid 23 updating version 0 to 33 2026-02-20T11:16:02.106 INFO:tasks.rados.rados.0.trial043.stdout:100: oid 23 version 33 is already newer than 33 2026-02-20T11:16:02.106 INFO:tasks.rados.rados.0.trial043.stdout:100: oid 23 version 33 is already newer than 33 2026-02-20T11:16:02.106 INFO:tasks.rados.rados.0.trial043.stdout:100: oid 23 version 33 is already newer than 33 2026-02-20T11:16:02.106 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 23 v 33 (ObjNum 74 snap 3 seq_num 74) dirty exists 2026-02-20T11:16:02.106 INFO:tasks.rados.rados.0.trial043.stdout:100: left oid 23 (ObjNum 74 snap 3 seq_num 74) 2026-02-20T11:16:02.106 INFO:tasks.rados.rados.0.trial043.stdout:83: finishing write tid 3 to trial04342147-45 2026-02-20T11:16:02.106 INFO:tasks.rados.rados.0.trial043.stdout:83: finishing write tid 4 to trial04342147-45 2026-02-20T11:16:02.106 INFO:tasks.rados.rados.0.trial043.stdout:83: finishing write tid 5 to trial04342147-45 2026-02-20T11:16:02.107 INFO:tasks.rados.rados.0.trial043.stdout:83: oid 45 updating version 0 to 27 2026-02-20T11:16:02.107 INFO:tasks.rados.rados.0.trial043.stdout:83: oid 45 updating version 27 to 30 2026-02-20T11:16:02.107 INFO:tasks.rados.rados.0.trial043.stdout:83: oid 45 version 30 is already newer than 30 2026-02-20T11:16:02.107 INFO:tasks.rados.rados.0.trial043.stdout:83: oid 45 version 30 is already newer than 30 2026-02-20T11:16:02.107 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 45 v 30 (ObjNum 66 snap 3 seq_num 66) dirty exists 2026-02-20T11:16:02.107 INFO:tasks.rados.rados.0.trial043.stdout:83: left oid 45 (ObjNum 66 snap 3 seq_num 66) 2026-02-20T11:16:02.107 INFO:tasks.rados.rados.0.trial043.stdout:83: done (3 left) 2026-02-20T11:16:02.107 INFO:tasks.rados.rados.0.trial043.stdout:100: done (2 left) 2026-02-20T11:16:02.107 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:16:02.107 INFO:tasks.rados.rados.0.trial043.stdout:103: done (1 left) 2026-02-20T11:16:02.107 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:16:02.107 INFO:tasks.rados.rados.0.trial043.stdout:105: expect (ObjNum 12 snap 0 seq_num 12) 2026-02-20T11:16:02.156 INFO:tasks.rados.rados.0.trial043.stdout:105: done (0 left) 2026-02-20T11:16:02.746 INFO:tasks.rados.rados.0.trial043.stdout:106: done (0 left) 2026-02-20T11:16:02.747 INFO:tasks.rados.rados.0.trial043.stdout:107: read oid 40 snap 3 2026-02-20T11:16:02.747 INFO:tasks.rados.rados.0.trial043.stdout:107: expect (ObjNum 61 snap 2 seq_num 61) 2026-02-20T11:16:02.747 INFO:tasks.rados.rados.0.trial043.stdout:108: write oid 37 current snap is 4 2026-02-20T11:16:02.747 INFO:tasks.rados.rados.0.trial043.stdout:108: seq_num 78 ranges {510946=632952,1798066=477679,2742758=457217} 2026-02-20T11:16:02.751 INFO:tasks.rados.rados.0.trial043.stdout:108: writing trial04342147-37 from 510946 to 1143898 tid 1 2026-02-20T11:16:02.755 INFO:tasks.rados.rados.0.trial043.stdout:108: writing trial04342147-37 from 1798066 to 2275745 tid 2 2026-02-20T11:16:02.759 INFO:tasks.rados.rados.0.trial043.stdout:108: writing trial04342147-37 from 2742758 to 3199975 tid 3 2026-02-20T11:16:02.759 INFO:tasks.rados.rados.0.trial043.stdout:109: snap_create 2026-02-20T11:16:02.759 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:16:02.776 INFO:tasks.rados.rados.0.trial043.stdout:107: expect (ObjNum 61 snap 2 seq_num 61) 2026-02-20T11:16:02.776 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 61264 2026-02-20T11:16:02.783 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:16:02.812 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:16:02.858 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:02.857+0000 7feddae5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:02.860 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:02.861+0000 7feddae5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:02.881 INFO:tasks.rados.rados.0.trial043.stdout:108: finishing write tid 1 to trial04342147-37 2026-02-20T11:16:02.881 INFO:tasks.rados.rados.0.trial043.stdout:108: finishing write tid 2 to trial04342147-37 2026-02-20T11:16:02.881 INFO:tasks.rados.rados.0.trial043.stdout:107: done (1 left) 2026-02-20T11:16:02.881 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:16:02.881 INFO:tasks.rados.rados.0.trial043.stdout:108: finishing write tid 3 to trial04342147-37 2026-02-20T11:16:02.881 INFO:tasks.rados.rados.0.trial043.stdout:108: finishing write tid 4 to trial04342147-37 2026-02-20T11:16:02.881 INFO:tasks.rados.rados.0.trial043.stdout:108: finishing write tid 5 to trial04342147-37 2026-02-20T11:16:02.881 INFO:tasks.rados.rados.0.trial043.stdout:108: oid 37 updating version 0 to 36 2026-02-20T11:16:02.881 INFO:tasks.rados.rados.0.trial043.stdout:108: oid 37 version 36 is already newer than 36 2026-02-20T11:16:02.881 INFO:tasks.rados.rados.0.trial043.stdout:108: oid 37 version 36 is already newer than 36 2026-02-20T11:16:02.882 INFO:tasks.rados.rados.0.trial043.stdout:108: oid 37 version 36 is already newer than 36 2026-02-20T11:16:02.882 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 37 v 36 (ObjNum 78 snap 4 seq_num 78) dirty exists 2026-02-20T11:16:02.882 INFO:tasks.rados.rados.0.trial043.stdout:108: left oid 37 (ObjNum 78 snap 4 seq_num 78) 2026-02-20T11:16:02.882 INFO:tasks.rados.rados.0.trial043.stdout:108: done (0 left) 2026-02-20T11:16:02.887 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:16:02.902 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:16:02.903 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:16:03.749 INFO:tasks.rados.rados.0.trial043.stdout:109: done (0 left) 2026-02-20T11:16:03.749 INFO:tasks.rados.rados.0.trial043.stdout:110: write (excl) oid 42 current snap is 5 2026-02-20T11:16:03.749 INFO:tasks.rados.rados.0.trial043.stdout:110: seq_num 79 ranges {527753=708201,1856548=737690,3286534=41219} 2026-02-20T11:16:03.753 INFO:tasks.rados.rados.0.trial043.stdout:110: writing trial04342147-42 from 527753 to 1235954 tid 1 2026-02-20T11:16:03.759 INFO:tasks.rados.rados.0.trial043.stdout:110: writing trial04342147-42 from 1856548 to 2594238 tid 2 2026-02-20T11:16:03.761 INFO:tasks.rados.rados.0.trial043.stdout:110: writing trial04342147-42 from 3286534 to 3327753 tid 3 2026-02-20T11:16:03.762 INFO:tasks.rados.rados.0.trial043.stdout:111: snap_remove snap 3 2026-02-20T11:16:04.753 INFO:tasks.rados.rados.0.trial043.stdout:111: done (1 left) 2026-02-20T11:16:04.754 INFO:tasks.rados.rados.0.trial043.stdout:112: snap_remove snap 4 2026-02-20T11:16:04.904 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:16:04.981 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:04.979+0000 7f5456dad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:04.983 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:04.982+0000 7f5455b24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:05.008 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:16:05.023 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:16:05.023 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:16:05.756 INFO:tasks.rados.rados.0.trial043.stdout:112: done (1 left) 2026-02-20T11:16:05.756 INFO:tasks.rados.rados.0.trial043.stdout:113: write (excl) oid 28 current snap is 5 2026-02-20T11:16:05.756 INFO:tasks.rados.rados.0.trial043.stdout:110: finishing write tid 1 to trial04342147-42 2026-02-20T11:16:05.756 INFO:tasks.rados.rados.0.trial043.stdout:113: seq_num 80 ranges {544560=783450,1915030=481348,2797604=479761} 2026-02-20T11:16:05.761 INFO:tasks.rados.rados.0.trial043.stdout:113: writing trial04342147-28 from 544560 to 1328010 tid 1 2026-02-20T11:16:05.766 INFO:tasks.rados.rados.0.trial043.stdout:113: writing trial04342147-28 from 1915030 to 2396378 tid 2 2026-02-20T11:16:05.769 INFO:tasks.rados.rados.0.trial043.stdout:113: writing trial04342147-28 from 2797604 to 3277365 tid 3 2026-02-20T11:16:05.769 INFO:tasks.rados.rados.0.trial043.stdout:114: snap_create 2026-02-20T11:16:05.769 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:16:05.769 INFO:tasks.rados.rados.0.trial043.stdout:110: finishing write tid 2 to trial04342147-42 2026-02-20T11:16:05.769 INFO:tasks.rados.rados.0.trial043.stdout:110: finishing write tid 3 to trial04342147-42 2026-02-20T11:16:05.770 INFO:tasks.rados.rados.0.trial043.stdout:110: finishing write tid 4 to trial04342147-42 2026-02-20T11:16:05.770 INFO:tasks.rados.rados.0.trial043.stdout:110: finishing write tid 5 to trial04342147-42 2026-02-20T11:16:05.770 INFO:tasks.rados.rados.0.trial043.stdout:110: oid 42 updating version 0 to 38 2026-02-20T11:16:05.770 INFO:tasks.rados.rados.0.trial043.stdout:110: oid 42 version 38 is already newer than 35 2026-02-20T11:16:05.770 INFO:tasks.rados.rados.0.trial043.stdout:110: oid 42 version 38 is already newer than 38 2026-02-20T11:16:05.770 INFO:tasks.rados.rados.0.trial043.stdout:110: oid 42 version 38 is already newer than 38 2026-02-20T11:16:05.770 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 42 v 38 (ObjNum 79 snap 5 seq_num 79) dirty exists 2026-02-20T11:16:05.770 INFO:tasks.rados.rados.0.trial043.stdout:110: left oid 42 (ObjNum 79 snap 5 seq_num 79) 2026-02-20T11:16:05.770 INFO:tasks.rados.rados.0.trial043.stdout:110: done (1 left) 2026-02-20T11:16:05.770 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:16:05.784 INFO:tasks.rados.rados.0.trial043.stdout:113: finishing write tid 1 to trial04342147-28 2026-02-20T11:16:05.796 INFO:tasks.rados.rados.0.trial043.stdout:113: finishing write tid 2 to trial04342147-28 2026-02-20T11:16:05.801 INFO:tasks.rados.rados.0.trial043.stdout:113: finishing write tid 3 to trial04342147-28 2026-02-20T11:16:05.807 INFO:tasks.rados.rados.0.trial043.stdout:113: finishing write tid 4 to trial04342147-28 2026-02-20T11:16:05.809 INFO:tasks.rados.rados.0.trial043.stdout:113: finishing write tid 5 to trial04342147-28 2026-02-20T11:16:05.809 INFO:tasks.rados.rados.0.trial043.stdout:113: oid 28 updating version 0 to 14 2026-02-20T11:16:05.809 INFO:tasks.rados.rados.0.trial043.stdout:113: oid 28 version 14 is already newer than 14 2026-02-20T11:16:05.809 INFO:tasks.rados.rados.0.trial043.stdout:113: oid 28 version 14 is already newer than 14 2026-02-20T11:16:05.809 INFO:tasks.rados.rados.0.trial043.stdout:113: oid 28 version 14 is already newer than 11 2026-02-20T11:16:05.809 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 28 v 14 (ObjNum 80 snap 5 seq_num 80) dirty exists 2026-02-20T11:16:05.809 INFO:tasks.rados.rados.0.trial043.stdout:113: left oid 28 (ObjNum 80 snap 5 seq_num 80) 2026-02-20T11:16:05.809 INFO:tasks.rados.rados.0.trial043.stdout:113: done (0 left) 2026-02-20T11:16:06.295 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:16:06.367 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:16:06.370 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:16:06.370 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:16:06.762 INFO:tasks.rados.rados.0.trial043.stdout:114: done (0 left) 2026-02-20T11:16:06.763 INFO:tasks.rados.rados.0.trial043.stdout:115: delete oid 14 current snap is 6 2026-02-20T11:16:06.786 INFO:tasks.rados.rados.0.trial043.stdout:115: done (0 left) 2026-02-20T11:16:06.786 INFO:tasks.rados.rados.0.trial043.stdout:116: snap_remove snap 5 2026-02-20T11:16:07.025 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:16:07.101 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:07.099+0000 7fe2f4f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:07.103 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:07.102+0000 7fe2f4f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:07.129 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:16:07.145 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:16:07.145 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:16:07.341 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~22s 2026-02-20T11:16:07.764 INFO:tasks.rados.rados.0.trial043.stdout:116: done (0 left) 2026-02-20T11:16:07.765 INFO:tasks.rados.rados.0.trial043.stdout:117: snap_create 2026-02-20T11:16:08.768 INFO:tasks.rados.rados.0.trial043.stdout:117: done (0 left) 2026-02-20T11:16:08.768 INFO:tasks.rados.rados.0.trial043.stdout:118: snap_create 2026-02-20T11:16:09.147 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:16:09.224 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:09.222+0000 7f56fa1e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:09.227 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:09.226+0000 7f56fa1e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:09.254 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:16:09.270 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:16:09.271 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:16:09.773 INFO:tasks.rados.rados.0.trial043.stdout:118: done (0 left) 2026-02-20T11:16:09.774 INFO:tasks.rados.rados.0.trial043.stdout:119: snap_remove snap 6 2026-02-20T11:16:10.776 INFO:tasks.rados.rados.0.trial043.stdout:119: done (0 left) 2026-02-20T11:16:10.776 INFO:tasks.rados.rados.0.trial043.stdout:120: read oid 14 snap 7 2026-02-20T11:16:10.777 INFO:tasks.rados.rados.0.trial043.stdout:120: expect deleted 2026-02-20T11:16:10.777 INFO:tasks.rados.rados.0.trial043.stdout:121: write oid 34 current snap is 8 2026-02-20T11:16:10.777 INFO:tasks.rados.rados.0.trial043.stdout:121: seq_num 82 ranges {578174=533948,1515641=485017,2736097=502305} 2026-02-20T11:16:10.780 INFO:tasks.rados.rados.0.trial043.stdout:121: writing trial04342147-34 from 578174 to 1112122 tid 1 2026-02-20T11:16:10.784 INFO:tasks.rados.rados.0.trial043.stdout:121: writing trial04342147-34 from 1515641 to 2000658 tid 2 2026-02-20T11:16:10.789 INFO:tasks.rados.rados.0.trial043.stdout:121: writing trial04342147-34 from 2736097 to 3238402 tid 3 2026-02-20T11:16:10.790 INFO:tasks.rados.rados.0.trial043.stdout:122: read oid 29 snap 7 2026-02-20T11:16:10.790 INFO:tasks.rados.rados.0.trial043.stdout:122: expect (ObjNum 59 snap 2 seq_num 59) 2026-02-20T11:16:10.790 INFO:tasks.rados.rados.0.trial043.stdout:123: write oid 36 current snap is 8 2026-02-20T11:16:10.790 INFO:tasks.rados.rados.0.trial043.stdout:123: seq_num 83 ranges {594981=609197,1974123=745028,3279873=115108} 2026-02-20T11:16:10.793 INFO:tasks.rados.rados.0.trial043.stdout:123: writing trial04342147-36 from 594981 to 1204178 tid 1 2026-02-20T11:16:10.798 INFO:tasks.rados.rados.0.trial043.stdout:123: writing trial04342147-36 from 1974123 to 2719151 tid 2 2026-02-20T11:16:10.801 INFO:tasks.rados.rados.0.trial043.stdout:123: writing trial04342147-36 from 3279873 to 3394981 tid 3 2026-02-20T11:16:10.801 INFO:tasks.rados.rados.0.trial043.stdout:124: read oid 46 snap -1 2026-02-20T11:16:10.801 INFO:tasks.rados.rados.0.trial043.stdout:124: expect (ObjNum 45 snap 0 seq_num 45) 2026-02-20T11:16:10.801 INFO:tasks.rados.rados.0.trial043.stdout:125: write oid 28 current snap is 8 2026-02-20T11:16:10.801 INFO:tasks.rados.rados.0.trial043.stdout:125: seq_num 84 ranges {611788=400799,1748958=488686,2907296=504492} 2026-02-20T11:16:10.805 INFO:tasks.rados.rados.0.trial043.stdout:125: writing trial04342147-28 from 611788 to 1012587 tid 1 2026-02-20T11:16:10.809 INFO:tasks.rados.rados.0.trial043.stdout:125: writing trial04342147-28 from 1748958 to 2237644 tid 2 2026-02-20T11:16:10.814 INFO:tasks.rados.rados.0.trial043.stdout:125: writing trial04342147-28 from 2907296 to 3411788 tid 3 2026-02-20T11:16:10.814 INFO:tasks.rados.rados.0.trial043.stdout:126: read oid 10 snap -1 2026-02-20T11:16:10.814 INFO:tasks.rados.rados.0.trial043.stdout:126: expect (ObjNum 69 snap 3 seq_num 69) 2026-02-20T11:16:10.815 INFO:tasks.rados.rados.0.trial043.stdout:127: read oid 11 snap -1 2026-02-20T11:16:10.815 INFO:tasks.rados.rados.0.trial043.stdout:127: expect (ObjNum 10 snap 0 seq_num 10) 2026-02-20T11:16:10.815 INFO:tasks.rados.rados.0.trial043.stdout:128: delete oid 41 current snap is 8 2026-02-20T11:16:10.819 INFO:tasks.rados.rados.0.trial043.stdout:122: expect (ObjNum 59 snap 2 seq_num 59) 2026-02-20T11:16:10.929 INFO:tasks.rados.rados.0.trial043.stdout:120: done (8 left) 2026-02-20T11:16:10.929 INFO:tasks.rados.rados.0.trial043.stdout:122: done (7 left) 2026-02-20T11:16:10.929 INFO:tasks.rados.rados.0.trial043.stdout:128: done (6 left) 2026-02-20T11:16:10.929 INFO:tasks.rados.rados.0.trial043.stdout:129: snap_create 2026-02-20T11:16:10.929 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 6 2026-02-20T11:16:10.929 INFO:tasks.rados.rados.0.trial043.stdout:124: expect (ObjNum 45 snap 0 seq_num 45) 2026-02-20T11:16:10.929 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 753885 2026-02-20T11:16:10.948 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:16:10.973 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 844127 2026-02-20T11:16:10.988 INFO:tasks.rados.rados.0.trial043.stdout:123: finishing write tid 1 to trial04342147-36 2026-02-20T11:16:10.989 INFO:tasks.rados.rados.0.trial043.stdout:127: expect (ObjNum 10 snap 0 seq_num 10) 2026-02-20T11:16:11.028 INFO:tasks.rados.rados.0.trial043.stdout:121: finishing write tid 1 to trial04342147-34 2026-02-20T11:16:11.028 INFO:tasks.rados.rados.0.trial043.stdout:124: done (5 left) 2026-02-20T11:16:11.028 INFO:tasks.rados.rados.0.trial043.stdout:127: done (4 left) 2026-02-20T11:16:11.028 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 4 2026-02-20T11:16:11.028 INFO:tasks.rados.rados.0.trial043.stdout:125: finishing write tid 1 to trial04342147-28 2026-02-20T11:16:11.029 INFO:tasks.rados.rados.0.trial043.stdout:123: finishing write tid 2 to trial04342147-36 2026-02-20T11:16:11.029 INFO:tasks.rados.rados.0.trial043.stdout:121: finishing write tid 2 to trial04342147-34 2026-02-20T11:16:11.029 INFO:tasks.rados.rados.0.trial043.stdout:123: finishing write tid 3 to trial04342147-36 2026-02-20T11:16:11.029 INFO:tasks.rados.rados.0.trial043.stdout:123: finishing write tid 4 to trial04342147-36 2026-02-20T11:16:11.029 INFO:tasks.rados.rados.0.trial043.stdout:123: finishing write tid 5 to trial04342147-36 2026-02-20T11:16:11.029 INFO:tasks.rados.rados.0.trial043.stdout:123: oid 36 updating version 0 to 14 2026-02-20T11:16:11.029 INFO:tasks.rados.rados.0.trial043.stdout:123: oid 36 version 14 is already newer than 14 2026-02-20T11:16:11.029 INFO:tasks.rados.rados.0.trial043.stdout:123: oid 36 version 14 is already newer than 14 2026-02-20T11:16:11.029 INFO:tasks.rados.rados.0.trial043.stdout:123: oid 36 version 14 is already newer than 14 2026-02-20T11:16:11.029 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 36 v 14 (ObjNum 83 snap 8 seq_num 83) dirty exists 2026-02-20T11:16:11.029 INFO:tasks.rados.rados.0.trial043.stdout:123: left oid 36 (ObjNum 83 snap 8 seq_num 83) 2026-02-20T11:16:11.030 INFO:tasks.rados.rados.0.trial043.stdout:121: finishing write tid 3 to trial04342147-34 2026-02-20T11:16:11.030 INFO:tasks.rados.rados.0.trial043.stdout:123: done (3 left) 2026-02-20T11:16:11.030 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:16:11.030 INFO:tasks.rados.rados.0.trial043.stdout:125: finishing write tid 2 to trial04342147-28 2026-02-20T11:16:11.030 INFO:tasks.rados.rados.0.trial043.stdout:121: finishing write tid 4 to trial04342147-34 2026-02-20T11:16:11.030 INFO:tasks.rados.rados.0.trial043.stdout:121: finishing write tid 5 to trial04342147-34 2026-02-20T11:16:11.030 INFO:tasks.rados.rados.0.trial043.stdout:121: oid 34 updating version 0 to 18 2026-02-20T11:16:11.030 INFO:tasks.rados.rados.0.trial043.stdout:121: oid 34 version 18 is already newer than 18 2026-02-20T11:16:11.030 INFO:tasks.rados.rados.0.trial043.stdout:121: oid 34 version 18 is already newer than 18 2026-02-20T11:16:11.030 INFO:tasks.rados.rados.0.trial043.stdout:121: oid 34 version 18 is already newer than 18 2026-02-20T11:16:11.030 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 34 v 18 (ObjNum 82 snap 8 seq_num 82) dirty exists 2026-02-20T11:16:11.031 INFO:tasks.rados.rados.0.trial043.stdout:121: left oid 34 (ObjNum 82 snap 8 seq_num 82) 2026-02-20T11:16:11.031 INFO:tasks.rados.rados.0.trial043.stdout:125: finishing write tid 3 to trial04342147-28 2026-02-20T11:16:11.031 INFO:tasks.rados.rados.0.trial043.stdout:125: finishing write tid 4 to trial04342147-28 2026-02-20T11:16:11.031 INFO:tasks.rados.rados.0.trial043.stdout:121: done (2 left) 2026-02-20T11:16:11.031 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:16:11.031 INFO:tasks.rados.rados.0.trial043.stdout:125: finishing write tid 5 to trial04342147-28 2026-02-20T11:16:11.031 INFO:tasks.rados.rados.0.trial043.stdout:125: oid 28 updating version 0 to 19 2026-02-20T11:16:11.031 INFO:tasks.rados.rados.0.trial043.stdout:125: oid 28 version 19 is already newer than 19 2026-02-20T11:16:11.031 INFO:tasks.rados.rados.0.trial043.stdout:125: oid 28 version 19 is already newer than 19 2026-02-20T11:16:11.031 INFO:tasks.rados.rados.0.trial043.stdout:125: oid 28 version 19 is already newer than 19 2026-02-20T11:16:11.031 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 28 v 19 (ObjNum 84 snap 8 seq_num 84) dirty exists 2026-02-20T11:16:11.031 INFO:tasks.rados.rados.0.trial043.stdout:125: left oid 28 (ObjNum 84 snap 8 seq_num 84) 2026-02-20T11:16:11.032 INFO:tasks.rados.rados.0.trial043.stdout:126: expect (ObjNum 69 snap 3 seq_num 69) 2026-02-20T11:16:11.167 INFO:tasks.rados.rados.0.trial043.stdout:125: done (1 left) 2026-02-20T11:16:11.167 INFO:tasks.rados.rados.0.trial043.stdout:126: done (0 left) 2026-02-20T11:16:11.272 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:16:11.347 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:11.345+0000 7fb44a3c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:11.349 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:11.349+0000 7fb44a3c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:11.371 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:16:11.376 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:16:11.391 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:16:11.391 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:16:11.439 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:16:11.442 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:16:11.443 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:16:11.781 INFO:tasks.rados.rados.0.trial043.stdout:129: done (0 left) 2026-02-20T11:16:11.781 INFO:tasks.rados.rados.0.trial043.stdout:130: delete oid 48 current snap is 9 2026-02-20T11:16:11.816 INFO:tasks.rados.rados.0.trial043.stdout:130: done (0 left) 2026-02-20T11:16:11.816 INFO:tasks.rados.rados.0.trial043.stdout:131: read oid 45 snap 7 2026-02-20T11:16:11.816 INFO:tasks.rados.rados.0.trial043.stdout:131: expect (ObjNum 66 snap 3 seq_num 66) 2026-02-20T11:16:11.816 INFO:tasks.rados.rados.0.trial043.stdout:132: write oid 2 current snap is 9 2026-02-20T11:16:11.816 INFO:tasks.rados.rados.0.trial043.stdout:132: seq_num 87 ranges {662209=626546,1808051=636013,2873212=588997} 2026-02-20T11:16:11.820 INFO:tasks.rados.rados.0.trial043.stdout:132: writing trial04342147-2 from 662209 to 1288755 tid 1 2026-02-20T11:16:11.825 INFO:tasks.rados.rados.0.trial043.stdout:132: writing trial04342147-2 from 1808051 to 2444064 tid 2 2026-02-20T11:16:11.829 INFO:tasks.rados.rados.0.trial043.stdout:132: writing trial04342147-2 from 2873212 to 3462209 tid 3 2026-02-20T11:16:11.829 INFO:tasks.rados.rados.0.trial043.stdout:133: snap_create 2026-02-20T11:16:11.829 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:16:11.837 INFO:tasks.rados.rados.0.trial043.stdout:131: expect (ObjNum 66 snap 3 seq_num 66) 2026-02-20T11:16:11.905 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2541202 2026-02-20T11:16:11.919 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 384001 2026-02-20T11:16:11.964 INFO:tasks.rados.rados.0.trial043.stdout:132: finishing write tid 1 to trial04342147-2 2026-02-20T11:16:11.965 INFO:tasks.rados.rados.0.trial043.stdout:132: finishing write tid 2 to trial04342147-2 2026-02-20T11:16:11.965 INFO:tasks.rados.rados.0.trial043.stdout:131: done (1 left) 2026-02-20T11:16:11.965 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:16:11.965 INFO:tasks.rados.rados.0.trial043.stdout:132: finishing write tid 3 to trial04342147-2 2026-02-20T11:16:11.965 INFO:tasks.rados.rados.0.trial043.stdout:132: finishing write tid 4 to trial04342147-2 2026-02-20T11:16:11.965 INFO:tasks.rados.rados.0.trial043.stdout:132: finishing write tid 5 to trial04342147-2 2026-02-20T11:16:11.965 INFO:tasks.rados.rados.0.trial043.stdout:132: oid 2 updating version 0 to 43 2026-02-20T11:16:11.965 INFO:tasks.rados.rados.0.trial043.stdout:132: oid 2 version 43 is already newer than 43 2026-02-20T11:16:11.965 INFO:tasks.rados.rados.0.trial043.stdout:132: oid 2 version 43 is already newer than 43 2026-02-20T11:16:11.965 INFO:tasks.rados.rados.0.trial043.stdout:132: oid 2 version 43 is already newer than 43 2026-02-20T11:16:11.966 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 2 v 43 (ObjNum 87 snap 9 seq_num 87) dirty exists 2026-02-20T11:16:11.966 INFO:tasks.rados.rados.0.trial043.stdout:132: left oid 2 (ObjNum 87 snap 9 seq_num 87) 2026-02-20T11:16:11.966 INFO:tasks.rados.rados.0.trial043.stdout:132: done (0 left) 2026-02-20T11:16:12.786 INFO:tasks.rados.rados.0.trial043.stdout:133: done (0 left) 2026-02-20T11:16:12.786 INFO:tasks.rados.rados.0.trial043.stdout:134: snap_remove snap 8 2026-02-20T11:16:12.846 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~28s 2026-02-20T11:16:13.392 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:16:13.469 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:13.467+0000 7f62707c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:13.472 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:13.471+0000 7f62707c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:13.498 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:16:13.513 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:16:13.513 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:16:13.787 INFO:tasks.rados.rados.0.trial043.stdout:134: done (0 left) 2026-02-20T11:16:13.787 INFO:tasks.rados.rados.0.trial043.stdout:135: write (excl) oid 26 current snap is 10 2026-02-20T11:16:13.787 INFO:tasks.rados.rados.0.trial043.stdout:135: seq_num 88 ranges {679016=701795,1866533=496024,2900635=402643} 2026-02-20T11:16:13.792 INFO:tasks.rados.rados.0.trial043.stdout:135: writing trial04342147-26 from 679016 to 1380811 tid 1 2026-02-20T11:16:13.796 INFO:tasks.rados.rados.0.trial043.stdout:135: writing trial04342147-26 from 1866533 to 2362557 tid 2 2026-02-20T11:16:13.800 INFO:tasks.rados.rados.0.trial043.stdout:135: writing trial04342147-26 from 2900635 to 3303278 tid 3 2026-02-20T11:16:13.800 INFO:tasks.rados.rados.0.trial043.stdout:136: snap_remove snap 7 2026-02-20T11:16:14.792 INFO:tasks.rados.rados.0.trial043.stdout:136: done (1 left) 2026-02-20T11:16:14.792 INFO:tasks.rados.rados.0.trial043.stdout:137: read oid 7 snap -1 2026-02-20T11:16:14.792 INFO:tasks.rados.rados.0.trial043.stdout:137: expect (ObjNum 63 snap 3 seq_num 63) 2026-02-20T11:16:14.792 INFO:tasks.rados.rados.0.trial043.stdout:138: delete oid 3 current snap is 10 2026-02-20T11:16:14.792 INFO:tasks.rados.rados.0.trial043.stdout:135: finishing write tid 1 to trial04342147-26 2026-02-20T11:16:14.793 INFO:tasks.rados.rados.0.trial043.stdout:135: finishing write tid 2 to trial04342147-26 2026-02-20T11:16:14.793 INFO:tasks.rados.rados.0.trial043.stdout:135: finishing write tid 3 to trial04342147-26 2026-02-20T11:16:14.793 INFO:tasks.rados.rados.0.trial043.stdout:135: finishing write tid 4 to trial04342147-26 2026-02-20T11:16:14.793 INFO:tasks.rados.rados.0.trial043.stdout:135: finishing write tid 5 to trial04342147-26 2026-02-20T11:16:14.793 INFO:tasks.rados.rados.0.trial043.stdout:135: oid 26 updating version 0 to 31 2026-02-20T11:16:14.793 INFO:tasks.rados.rados.0.trial043.stdout:135: oid 26 version 31 is already newer than 31 2026-02-20T11:16:14.793 INFO:tasks.rados.rados.0.trial043.stdout:135: oid 26 version 31 is already newer than 31 2026-02-20T11:16:14.793 INFO:tasks.rados.rados.0.trial043.stdout:135: oid 26 version 31 is already newer than 28 2026-02-20T11:16:14.793 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 26 v 31 (ObjNum 88 snap 10 seq_num 88) dirty exists 2026-02-20T11:16:14.793 INFO:tasks.rados.rados.0.trial043.stdout:135: left oid 26 (ObjNum 88 snap 10 seq_num 88) 2026-02-20T11:16:14.838 INFO:tasks.rados.rados.0.trial043.stdout:137: expect (ObjNum 63 snap 3 seq_num 63) 2026-02-20T11:16:14.848 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1840934 2026-02-20T11:16:14.879 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2320821 2026-02-20T11:16:14.925 INFO:tasks.rados.rados.0.trial043.stdout:135: done (2 left) 2026-02-20T11:16:14.925 INFO:tasks.rados.rados.0.trial043.stdout:137: done (1 left) 2026-02-20T11:16:14.925 INFO:tasks.rados.rados.0.trial043.stdout:138: done (0 left) 2026-02-20T11:16:14.925 INFO:tasks.rados.rados.0.trial043.stdout:139: snap_remove snap 9 2026-02-20T11:16:15.515 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:16:15.591 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:15.589+0000 7f839905a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:15.594 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:15.593+0000 7f839905a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:15.621 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:16:15.637 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:16:15.638 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:16:15.796 INFO:tasks.rados.rados.0.trial043.stdout:139: done (0 left) 2026-02-20T11:16:15.796 INFO:tasks.rados.rados.0.trial043.stdout:140: write (excl) oid 16 current snap is 10 2026-02-20T11:16:15.796 INFO:tasks.rados.rados.0.trial043.stdout:140: seq_num 90 ranges {712630=452293,1583497=499693,2555481=425187} 2026-02-20T11:16:15.800 INFO:tasks.rados.rados.0.trial043.stdout:140: writing trial04342147-16 from 712630 to 1164923 tid 1 2026-02-20T11:16:15.803 INFO:tasks.rados.rados.0.trial043.stdout:140: writing trial04342147-16 from 1583497 to 2083190 tid 2 2026-02-20T11:16:15.807 INFO:tasks.rados.rados.0.trial043.stdout:140: writing trial04342147-16 from 2555481 to 2980668 tid 3 2026-02-20T11:16:15.807 INFO:tasks.rados.rados.0.trial043.stdout:141: write (excl) oid 25 current snap is 10 2026-02-20T11:16:15.807 INFO:tasks.rados.rados.0.trial043.stdout:141: seq_num 91 ranges {729437=527542,1925626=643351,3266551=262886} 2026-02-20T11:16:15.811 INFO:tasks.rados.rados.0.trial043.stdout:141: writing trial04342147-25 from 729437 to 1256979 tid 1 2026-02-20T11:16:15.816 INFO:tasks.rados.rados.0.trial043.stdout:141: writing trial04342147-25 from 1925626 to 2568977 tid 2 2026-02-20T11:16:15.820 INFO:tasks.rados.rados.0.trial043.stdout:141: writing trial04342147-25 from 3266551 to 3529437 tid 3 2026-02-20T11:16:15.821 INFO:tasks.rados.rados.0.trial043.stdout:142: write (excl) oid 46 current snap is 10 2026-02-20T11:16:15.821 INFO:tasks.rados.rados.0.trial043.stdout:142: seq_num 92 ranges {746244=719144,2100461=503362,3126680=419564} 2026-02-20T11:16:15.826 INFO:tasks.rados.rados.0.trial043.stdout:142: writing trial04342147-46 from 746244 to 1465388 tid 1 2026-02-20T11:16:15.830 INFO:tasks.rados.rados.0.trial043.stdout:142: writing trial04342147-46 from 2100461 to 2603823 tid 2 2026-02-20T11:16:15.834 INFO:tasks.rados.rados.0.trial043.stdout:142: writing trial04342147-46 from 3126680 to 3546244 tid 3 2026-02-20T11:16:15.835 INFO:tasks.rados.rados.0.trial043.stdout:143: read oid 43 snap -1 2026-02-20T11:16:15.835 INFO:tasks.rados.rados.0.trial043.stdout:143: expect (ObjNum 42 snap 0 seq_num 42) 2026-02-20T11:16:15.835 INFO:tasks.rados.rados.0.trial043.stdout:144: delete oid 20 current snap is 10 2026-02-20T11:16:15.835 INFO:tasks.rados.rados.0.trial043.stdout:141: finishing write tid 1 to trial04342147-25 2026-02-20T11:16:15.835 INFO:tasks.rados.rados.0.trial043.stdout:140: finishing write tid 1 to trial04342147-16 2026-02-20T11:16:15.843 INFO:tasks.rados.rados.0.trial043.stdout:142: finishing write tid 1 to trial04342147-46 2026-02-20T11:16:15.846 INFO:tasks.rados.rados.0.trial043.stdout:141: finishing write tid 2 to trial04342147-25 2026-02-20T11:16:15.856 INFO:tasks.rados.rados.0.trial043.stdout:144: done (4 left) 2026-02-20T11:16:15.856 INFO:tasks.rados.rados.0.trial043.stdout:145: snap_create 2026-02-20T11:16:15.856 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 4 2026-02-20T11:16:15.857 INFO:tasks.rados.rados.0.trial043.stdout:141: finishing write tid 3 to trial04342147-25 2026-02-20T11:16:15.863 INFO:tasks.rados.rados.0.trial043.stdout:140: finishing write tid 2 to trial04342147-16 2026-02-20T11:16:15.863 INFO:tasks.rados.rados.0.trial043.stdout:141: finishing write tid 4 to trial04342147-25 2026-02-20T11:16:15.865 INFO:tasks.rados.rados.0.trial043.stdout:141: finishing write tid 5 to trial04342147-25 2026-02-20T11:16:15.865 INFO:tasks.rados.rados.0.trial043.stdout:141: oid 25 updating version 0 to 23 2026-02-20T11:16:15.865 INFO:tasks.rados.rados.0.trial043.stdout:141: oid 25 version 23 is already newer than 20 2026-02-20T11:16:15.865 INFO:tasks.rados.rados.0.trial043.stdout:141: oid 25 version 23 is already newer than 23 2026-02-20T11:16:15.866 INFO:tasks.rados.rados.0.trial043.stdout:141: oid 25 version 23 is already newer than 23 2026-02-20T11:16:15.866 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 25 v 23 (ObjNum 91 snap 10 seq_num 91) dirty exists 2026-02-20T11:16:15.866 INFO:tasks.rados.rados.0.trial043.stdout:141: left oid 25 (ObjNum 91 snap 10 seq_num 91) 2026-02-20T11:16:15.866 INFO:tasks.rados.rados.0.trial043.stdout:141: done (3 left) 2026-02-20T11:16:15.866 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:16:15.867 INFO:tasks.rados.rados.0.trial043.stdout:143: expect (ObjNum 42 snap 0 seq_num 42) 2026-02-20T11:16:15.902 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:16:15.925 INFO:tasks.rados.rados.0.trial043.stdout:142: finishing write tid 2 to trial04342147-46 2026-02-20T11:16:15.925 INFO:tasks.rados.rados.0.trial043.stdout:142: finishing write tid 3 to trial04342147-46 2026-02-20T11:16:15.926 INFO:tasks.rados.rados.0.trial043.stdout:143: done (2 left) 2026-02-20T11:16:15.926 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:16:15.926 INFO:tasks.rados.rados.0.trial043.stdout:142: finishing write tid 4 to trial04342147-46 2026-02-20T11:16:15.926 INFO:tasks.rados.rados.0.trial043.stdout:142: finishing write tid 5 to trial04342147-46 2026-02-20T11:16:15.926 INFO:tasks.rados.rados.0.trial043.stdout:142: oid 46 updating version 0 to 7 2026-02-20T11:16:15.926 INFO:tasks.rados.rados.0.trial043.stdout:142: oid 46 version 7 is already newer than 7 2026-02-20T11:16:15.926 INFO:tasks.rados.rados.0.trial043.stdout:142: oid 46 version 7 is already newer than 4 2026-02-20T11:16:15.926 INFO:tasks.rados.rados.0.trial043.stdout:142: oid 46 version 7 is already newer than 7 2026-02-20T11:16:15.926 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 46 v 7 (ObjNum 92 snap 10 seq_num 92) dirty exists 2026-02-20T11:16:15.926 INFO:tasks.rados.rados.0.trial043.stdout:142: left oid 46 (ObjNum 92 snap 10 seq_num 92) 2026-02-20T11:16:15.926 INFO:tasks.rados.rados.0.trial043.stdout:140: finishing write tid 3 to trial04342147-16 2026-02-20T11:16:15.927 INFO:tasks.rados.rados.0.trial043.stdout:140: finishing write tid 4 to trial04342147-16 2026-02-20T11:16:15.927 INFO:tasks.rados.rados.0.trial043.stdout:140: finishing write tid 5 to trial04342147-16 2026-02-20T11:16:15.927 INFO:tasks.rados.rados.0.trial043.stdout:140: oid 16 updating version 0 to 30 2026-02-20T11:16:15.927 INFO:tasks.rados.rados.0.trial043.stdout:140: oid 16 version 30 is already newer than 30 2026-02-20T11:16:15.927 INFO:tasks.rados.rados.0.trial043.stdout:140: oid 16 version 30 is already newer than 27 2026-02-20T11:16:15.927 INFO:tasks.rados.rados.0.trial043.stdout:140: oid 16 version 30 is already newer than 30 2026-02-20T11:16:15.927 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 16 v 30 (ObjNum 90 snap 10 seq_num 90) dirty exists 2026-02-20T11:16:15.927 INFO:tasks.rados.rados.0.trial043.stdout:140: left oid 16 (ObjNum 90 snap 10 seq_num 90) 2026-02-20T11:16:15.927 INFO:tasks.rados.rados.0.trial043.stdout:140: done (1 left) 2026-02-20T11:16:15.927 INFO:tasks.rados.rados.0.trial043.stdout:142: done (0 left) 2026-02-20T11:16:16.444 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:16:16.516 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:16:16.518 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:16:16.519 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:16:16.800 INFO:tasks.rados.rados.0.trial043.stdout:145: done (0 left) 2026-02-20T11:16:16.800 INFO:tasks.rados.rados.0.trial043.stdout:146: read oid 1 snap -1 2026-02-20T11:16:16.800 INFO:tasks.rados.rados.0.trial043.stdout:146: expect (ObjNum 0 snap 0 seq_num 0) 2026-02-20T11:16:16.800 INFO:tasks.rados.rados.0.trial043.stdout:147: snap_remove snap 10 2026-02-20T11:16:17.639 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:16:17.717 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:17.715+0000 7fbe24b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:17.720 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:17.719+0000 7fbe24b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:17.746 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:16:17.762 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:16:17.763 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:16:17.800 INFO:tasks.rados.rados.0.trial043.stdout:147: done (1 left) 2026-02-20T11:16:17.801 INFO:tasks.rados.rados.0.trial043.stdout:148: delete oid 6 current snap is 11 2026-02-20T11:16:17.801 INFO:tasks.rados.rados.0.trial043.stdout:146: expect (ObjNum 0 snap 0 seq_num 0) 2026-02-20T11:16:17.801 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 328614 2026-02-20T11:16:17.816 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:16:17.845 INFO:tasks.rados.rados.0.trial043.stdout:146: done (1 left) 2026-02-20T11:16:17.845 INFO:tasks.rados.rados.0.trial043.stdout:148: done (0 left) 2026-02-20T11:16:17.845 INFO:tasks.rados.rados.0.trial043.stdout:149: write oid 41 current snap is 11 2026-02-20T11:16:17.845 INFO:tasks.rados.rados.0.trial043.stdout:149: seq_num 95 ranges {796665=544891,1759554=650689,2976243=620422} 2026-02-20T11:16:17.850 INFO:tasks.rados.rados.0.trial043.stdout:149: writing trial04342147-41 from 796665 to 1341556 tid 1 2026-02-20T11:16:17.854 INFO:tasks.rados.rados.0.trial043.stdout:149: writing trial04342147-41 from 1759554 to 2410243 tid 2 2026-02-20T11:16:17.858 INFO:tasks.rados.rados.0.trial043.stdout:149: writing trial04342147-41 from 2976243 to 3596665 tid 3 2026-02-20T11:16:17.859 INFO:tasks.rados.rados.0.trial043.stdout:150: write oid 20 current snap is 11 2026-02-20T11:16:17.859 INFO:tasks.rados.rados.0.trial043.stdout:150: seq_num 96 ranges {413472=620140,1818036=510700,3120019=493453} 2026-02-20T11:16:17.862 INFO:tasks.rados.rados.0.trial043.stdout:150: writing trial04342147-20 from 413472 to 1033612 tid 1 2026-02-20T11:16:17.867 INFO:tasks.rados.rados.0.trial043.stdout:150: writing trial04342147-20 from 1818036 to 2328736 tid 2 2026-02-20T11:16:17.872 INFO:tasks.rados.rados.0.trial043.stdout:150: writing trial04342147-20 from 3120019 to 3613472 tid 3 2026-02-20T11:16:17.872 INFO:tasks.rados.rados.0.trial043.stdout:151: delete oid 30 current snap is 11 2026-02-20T11:16:17.872 INFO:tasks.rados.rados.0.trial043.stdout:149: finishing write tid 1 to trial04342147-41 2026-02-20T11:16:17.875 INFO:tasks.rados.rados.0.trial043.stdout:149: finishing write tid 2 to trial04342147-41 2026-02-20T11:16:17.886 INFO:tasks.rados.rados.0.trial043.stdout:149: finishing write tid 3 to trial04342147-41 2026-02-20T11:16:17.887 INFO:tasks.rados.rados.0.trial043.stdout:150: finishing write tid 1 to trial04342147-20 2026-02-20T11:16:17.890 INFO:tasks.rados.rados.0.trial043.stdout:151: done (2 left) 2026-02-20T11:16:17.890 INFO:tasks.rados.rados.0.trial043.stdout:152: write (excl) oid 40 current snap is 11 2026-02-20T11:16:17.890 INFO:tasks.rados.rados.0.trial043.stdout:152: seq_num 98 ranges {447086=770638,1935000=798016,3458512=188574} 2026-02-20T11:16:17.895 INFO:tasks.rados.rados.0.trial043.stdout:152: writing trial04342147-40 from 447086 to 1217724 tid 1 2026-02-20T11:16:17.900 INFO:tasks.rados.rados.0.trial043.stdout:152: writing trial04342147-40 from 1935000 to 2733016 tid 2 2026-02-20T11:16:17.904 INFO:tasks.rados.rados.0.trial043.stdout:152: writing trial04342147-40 from 3458512 to 3647086 tid 3 2026-02-20T11:16:17.904 INFO:tasks.rados.rados.0.trial043.stdout:153: write (excl) oid 49 current snap is 11 2026-02-20T11:16:17.905 INFO:tasks.rados.rados.0.trial043.stdout:150: finishing write tid 2 to trial04342147-20 2026-02-20T11:16:17.905 INFO:tasks.rados.rados.0.trial043.stdout:153: seq_num 99 ranges {463893=562240,1593482=658027,2685935=559341} 2026-02-20T11:16:17.908 INFO:tasks.rados.rados.0.trial043.stdout:153: writing trial04342147-49 from 463893 to 1026133 tid 1 2026-02-20T11:16:17.913 INFO:tasks.rados.rados.0.trial043.stdout:153: writing trial04342147-49 from 1593482 to 2251509 tid 2 2026-02-20T11:16:17.918 INFO:tasks.rados.rados.0.trial043.stdout:153: writing trial04342147-49 from 2685935 to 3245276 tid 3 2026-02-20T11:16:17.919 INFO:tasks.rados.rados.0.trial043.stdout:154: delete oid 3 current snap is 11 2026-02-20T11:16:17.919 INFO:tasks.rados.rados.0.trial043.stdout:149: finishing write tid 4 to trial04342147-41 2026-02-20T11:16:17.919 INFO:tasks.rados.rados.0.trial043.stdout:149: finishing write tid 5 to trial04342147-41 2026-02-20T11:16:17.919 INFO:tasks.rados.rados.0.trial043.stdout:149: oid 41 updating version 0 to 20 2026-02-20T11:16:17.919 INFO:tasks.rados.rados.0.trial043.stdout:149: oid 41 version 20 is already newer than 20 2026-02-20T11:16:17.919 INFO:tasks.rados.rados.0.trial043.stdout:149: oid 41 version 20 is already newer than 20 2026-02-20T11:16:17.920 INFO:tasks.rados.rados.0.trial043.stdout:149: oid 41 version 20 is already newer than 20 2026-02-20T11:16:17.920 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 41 v 20 (ObjNum 95 snap 11 seq_num 95) dirty exists 2026-02-20T11:16:17.920 INFO:tasks.rados.rados.0.trial043.stdout:149: left oid 41 (ObjNum 95 snap 11 seq_num 95) 2026-02-20T11:16:17.920 INFO:tasks.rados.rados.0.trial043.stdout:150: finishing write tid 3 to trial04342147-20 2026-02-20T11:16:17.920 INFO:tasks.rados.rados.0.trial043.stdout:150: finishing write tid 4 to trial04342147-20 2026-02-20T11:16:17.920 INFO:tasks.rados.rados.0.trial043.stdout:150: finishing write tid 5 to trial04342147-20 2026-02-20T11:16:17.921 INFO:tasks.rados.rados.0.trial043.stdout:150: oid 20 updating version 0 to 43 2026-02-20T11:16:17.921 INFO:tasks.rados.rados.0.trial043.stdout:150: oid 20 version 43 is already newer than 43 2026-02-20T11:16:17.921 INFO:tasks.rados.rados.0.trial043.stdout:150: oid 20 version 43 is already newer than 43 2026-02-20T11:16:17.921 INFO:tasks.rados.rados.0.trial043.stdout:150: oid 20 version 43 is already newer than 43 2026-02-20T11:16:17.921 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 20 v 43 (ObjNum 96 snap 11 seq_num 96) dirty exists 2026-02-20T11:16:17.921 INFO:tasks.rados.rados.0.trial043.stdout:150: left oid 20 (ObjNum 96 snap 11 seq_num 96) 2026-02-20T11:16:17.921 INFO:tasks.rados.rados.0.trial043.stdout:152: finishing write tid 1 to trial04342147-40 2026-02-20T11:16:17.923 INFO:tasks.rados.rados.0.trial043.stdout:149: done (4 left) 2026-02-20T11:16:17.923 INFO:tasks.rados.rados.0.trial043.stdout:150: done (3 left) 2026-02-20T11:16:17.923 INFO:tasks.rados.rados.0.trial043.stdout:154: done (2 left) 2026-02-20T11:16:17.923 INFO:tasks.rados.rados.0.trial043.stdout:155: write (excl) oid 3 current snap is 11 2026-02-20T11:16:17.923 INFO:tasks.rados.rados.0.trial043.stdout:155: seq_num 101 ranges {497507=712738,1710446=661696,3140781=556726} 2026-02-20T11:16:17.925 INFO:tasks.rados.rados.0.trial043.stdout:155: writing trial04342147-3 from 497507 to 1210245 tid 1 2026-02-20T11:16:17.930 INFO:tasks.rados.rados.0.trial043.stdout:155: writing trial04342147-3 from 1710446 to 2372142 tid 2 2026-02-20T11:16:17.935 INFO:tasks.rados.rados.0.trial043.stdout:155: writing trial04342147-3 from 3140781 to 3697507 tid 3 2026-02-20T11:16:17.935 INFO:tasks.rados.rados.0.trial043.stdout:156: write oid 26 current snap is 11 2026-02-20T11:16:17.936 INFO:tasks.rados.rados.0.trial043.stdout:153: finishing write tid 1 to trial04342147-49 2026-02-20T11:16:17.936 INFO:tasks.rados.rados.0.trial043.stdout:156: seq_num 102 ranges {514314=787987,1768928=405354,2768204=509510} 2026-02-20T11:16:17.940 INFO:tasks.rados.rados.0.trial043.stdout:156: writing trial04342147-26 from 514314 to 1302301 tid 1 2026-02-20T11:16:17.943 INFO:tasks.rados.rados.0.trial043.stdout:156: writing trial04342147-26 from 1768928 to 2174282 tid 2 2026-02-20T11:16:17.947 INFO:tasks.rados.rados.0.trial043.stdout:156: writing trial04342147-26 from 2768204 to 3277714 tid 3 2026-02-20T11:16:17.947 INFO:tasks.rados.rados.0.trial043.stdout:157: snap_create 2026-02-20T11:16:17.947 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 4 2026-02-20T11:16:17.947 INFO:tasks.rados.rados.0.trial043.stdout:152: finishing write tid 2 to trial04342147-40 2026-02-20T11:16:17.947 INFO:tasks.rados.rados.0.trial043.stdout:152: finishing write tid 3 to trial04342147-40 2026-02-20T11:16:17.947 INFO:tasks.rados.rados.0.trial043.stdout:152: finishing write tid 4 to trial04342147-40 2026-02-20T11:16:17.948 INFO:tasks.rados.rados.0.trial043.stdout:152: finishing write tid 5 to trial04342147-40 2026-02-20T11:16:17.948 INFO:tasks.rados.rados.0.trial043.stdout:152: oid 40 updating version 0 to 24 2026-02-20T11:16:17.948 INFO:tasks.rados.rados.0.trial043.stdout:152: oid 40 version 24 is already newer than 21 2026-02-20T11:16:17.948 INFO:tasks.rados.rados.0.trial043.stdout:152: oid 40 version 24 is already newer than 24 2026-02-20T11:16:17.948 INFO:tasks.rados.rados.0.trial043.stdout:152: oid 40 version 24 is already newer than 24 2026-02-20T11:16:17.948 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 40 v 24 (ObjNum 98 snap 11 seq_num 98) dirty exists 2026-02-20T11:16:17.948 INFO:tasks.rados.rados.0.trial043.stdout:152: left oid 40 (ObjNum 98 snap 11 seq_num 98) 2026-02-20T11:16:17.948 INFO:tasks.rados.rados.0.trial043.stdout:153: finishing write tid 2 to trial04342147-49 2026-02-20T11:16:17.948 INFO:tasks.rados.rados.0.trial043.stdout:152: done (3 left) 2026-02-20T11:16:17.948 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:16:17.948 INFO:tasks.rados.rados.0.trial043.stdout:155: finishing write tid 1 to trial04342147-3 2026-02-20T11:16:17.948 INFO:tasks.rados.rados.0.trial043.stdout:153: finishing write tid 3 to trial04342147-49 2026-02-20T11:16:17.953 INFO:tasks.rados.rados.0.trial043.stdout:153: finishing write tid 4 to trial04342147-49 2026-02-20T11:16:17.956 INFO:tasks.rados.rados.0.trial043.stdout:153: finishing write tid 5 to trial04342147-49 2026-02-20T11:16:17.956 INFO:tasks.rados.rados.0.trial043.stdout:153: oid 49 updating version 0 to 19 2026-02-20T11:16:17.956 INFO:tasks.rados.rados.0.trial043.stdout:153: oid 49 version 19 is already newer than 19 2026-02-20T11:16:17.956 INFO:tasks.rados.rados.0.trial043.stdout:153: oid 49 version 19 is already newer than 16 2026-02-20T11:16:17.956 INFO:tasks.rados.rados.0.trial043.stdout:153: oid 49 version 19 is already newer than 19 2026-02-20T11:16:17.956 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 49 v 19 (ObjNum 99 snap 11 seq_num 99) dirty exists 2026-02-20T11:16:17.956 INFO:tasks.rados.rados.0.trial043.stdout:153: left oid 49 (ObjNum 99 snap 11 seq_num 99) 2026-02-20T11:16:17.956 INFO:tasks.rados.rados.0.trial043.stdout:153: done (2 left) 2026-02-20T11:16:17.956 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:16:17.960 INFO:tasks.rados.rados.0.trial043.stdout:155: finishing write tid 2 to trial04342147-3 2026-02-20T11:16:17.972 INFO:tasks.rados.rados.0.trial043.stdout:155: finishing write tid 3 to trial04342147-3 2026-02-20T11:16:17.982 INFO:tasks.rados.rados.0.trial043.stdout:155: finishing write tid 4 to trial04342147-3 2026-02-20T11:16:17.984 INFO:tasks.rados.rados.0.trial043.stdout:155: finishing write tid 5 to trial04342147-3 2026-02-20T11:16:17.985 INFO:tasks.rados.rados.0.trial043.stdout:155: oid 3 updating version 0 to 47 2026-02-20T11:16:17.985 INFO:tasks.rados.rados.0.trial043.stdout:155: oid 3 version 47 is already newer than 47 2026-02-20T11:16:17.985 INFO:tasks.rados.rados.0.trial043.stdout:155: oid 3 version 47 is already newer than 47 2026-02-20T11:16:17.985 INFO:tasks.rados.rados.0.trial043.stdout:155: oid 3 version 47 is already newer than 44 2026-02-20T11:16:17.985 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 3 v 47 (ObjNum 101 snap 11 seq_num 101) dirty exists 2026-02-20T11:16:17.985 INFO:tasks.rados.rados.0.trial043.stdout:155: left oid 3 (ObjNum 101 snap 11 seq_num 101) 2026-02-20T11:16:17.985 INFO:tasks.rados.rados.0.trial043.stdout:155: done (1 left) 2026-02-20T11:16:17.985 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:16:17.996 INFO:tasks.rados.rados.0.trial043.stdout:156: finishing write tid 1 to trial04342147-26 2026-02-20T11:16:18.001 INFO:tasks.rados.rados.0.trial043.stdout:156: finishing write tid 2 to trial04342147-26 2026-02-20T11:16:18.007 INFO:tasks.rados.rados.0.trial043.stdout:156: finishing write tid 3 to trial04342147-26 2026-02-20T11:16:18.011 INFO:tasks.rados.rados.0.trial043.stdout:156: finishing write tid 4 to trial04342147-26 2026-02-20T11:16:18.013 INFO:tasks.rados.rados.0.trial043.stdout:156: finishing write tid 5 to trial04342147-26 2026-02-20T11:16:18.013 INFO:tasks.rados.rados.0.trial043.stdout:156: oid 26 updating version 0 to 39 2026-02-20T11:16:18.013 INFO:tasks.rados.rados.0.trial043.stdout:156: oid 26 version 39 is already newer than 39 2026-02-20T11:16:18.013 INFO:tasks.rados.rados.0.trial043.stdout:156: oid 26 version 39 is already newer than 39 2026-02-20T11:16:18.013 INFO:tasks.rados.rados.0.trial043.stdout:156: oid 26 version 39 is already newer than 39 2026-02-20T11:16:18.013 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 26 v 39 (ObjNum 102 snap 11 seq_num 102) dirty exists 2026-02-20T11:16:18.014 INFO:tasks.rados.rados.0.trial043.stdout:156: left oid 26 (ObjNum 102 snap 11 seq_num 102) 2026-02-20T11:16:18.014 INFO:tasks.rados.rados.0.trial043.stdout:156: done (0 left) 2026-02-20T11:16:18.351 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~33s 2026-02-20T11:16:18.805 INFO:tasks.rados.rados.0.trial043.stdout:157: done (0 left) 2026-02-20T11:16:18.806 INFO:tasks.rados.rados.0.trial043.stdout:158: snap_create 2026-02-20T11:16:19.765 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:16:19.806 INFO:tasks.rados.rados.0.trial043.stdout:158: done (0 left) 2026-02-20T11:16:19.807 INFO:tasks.rados.rados.0.trial043.stdout:159: read oid 2 snap -1 2026-02-20T11:16:19.807 INFO:tasks.rados.rados.0.trial043.stdout:159: expect (ObjNum 87 snap 9 seq_num 87) 2026-02-20T11:16:19.807 INFO:tasks.rados.rados.0.trial043.stdout:160: snap_remove snap 12 2026-02-20T11:16:19.845 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:19.843+0000 7fd0bc98c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:19.848 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:19.847+0000 7fd0bc98c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:19.873 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:16:19.888 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:16:19.889 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:16:20.812 INFO:tasks.rados.rados.0.trial043.stdout:160: done (1 left) 2026-02-20T11:16:20.812 INFO:tasks.rados.rados.0.trial043.stdout:161: write (excl) oid 40 current snap is 13 2026-02-20T11:16:20.812 INFO:tasks.rados.rados.0.trial043.stdout:161: seq_num 103 ranges {531121=463236,1711057=665365,3079274=651847} 2026-02-20T11:16:20.816 INFO:tasks.rados.rados.0.trial043.stdout:161: writing trial04342147-40 from 531121 to 994357 tid 1 2026-02-20T11:16:20.822 INFO:tasks.rados.rados.0.trial043.stdout:161: writing trial04342147-40 from 1711057 to 2376422 tid 2 2026-02-20T11:16:20.827 INFO:tasks.rados.rados.0.trial043.stdout:161: writing trial04342147-40 from 3079274 to 3731121 tid 3 2026-02-20T11:16:20.827 INFO:tasks.rados.rados.0.trial043.stdout:162: read oid 19 snap 11 2026-02-20T11:16:20.827 INFO:tasks.rados.rados.0.trial043.stdout:162: expect (ObjNum 18 snap 0 seq_num 18) 2026-02-20T11:16:20.827 INFO:tasks.rados.rados.0.trial043.stdout:163: write (excl) oid 34 current snap is 13 2026-02-20T11:16:20.827 INFO:tasks.rados.rados.0.trial043.stdout:163: seq_num 104 ranges {547928=538485,1769539=409023,2706697=532054} 2026-02-20T11:16:20.831 INFO:tasks.rados.rados.0.trial043.stdout:163: writing trial04342147-34 from 547928 to 1086413 tid 1 2026-02-20T11:16:20.836 INFO:tasks.rados.rados.0.trial043.stdout:163: writing trial04342147-34 from 1769539 to 2178562 tid 2 2026-02-20T11:16:20.840 INFO:tasks.rados.rados.0.trial043.stdout:163: writing trial04342147-34 from 2706697 to 3238751 tid 3 2026-02-20T11:16:20.840 INFO:tasks.rados.rados.0.trial043.stdout:164: read oid 5 snap 11 2026-02-20T11:16:20.840 INFO:tasks.rados.rados.0.trial043.stdout:164: expect (ObjNum 4 snap 0 seq_num 4) 2026-02-20T11:16:20.840 INFO:tasks.rados.rados.0.trial043.stdout:165: read oid 16 snap -1 2026-02-20T11:16:20.840 INFO:tasks.rados.rados.0.trial043.stdout:165: expect (ObjNum 90 snap 10 seq_num 90) 2026-02-20T11:16:20.841 INFO:tasks.rados.rados.0.trial043.stdout:166: write oid 4 current snap is 13 2026-02-20T11:16:20.841 INFO:tasks.rados.rados.0.trial043.stdout:159: expect (ObjNum 87 snap 9 seq_num 87) 2026-02-20T11:16:20.841 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:16:20.904 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 3281428 2026-02-20T11:16:20.914 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1211468 2026-02-20T11:16:20.941 INFO:tasks.rados.rados.0.trial043.stdout:162: expect (ObjNum 18 snap 0 seq_num 18) 2026-02-20T11:16:20.945 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1073764 2026-02-20T11:16:20.993 INFO:tasks.rados.rados.0.trial043.stdout:166: seq_num 105 ranges {564735=613734,1828021=669034,3134120=630615} 2026-02-20T11:16:20.998 INFO:tasks.rados.rados.0.trial043.stdout:166: writing trial04342147-4 from 564735 to 1178469 tid 1 2026-02-20T11:16:21.003 INFO:tasks.rados.rados.0.trial043.stdout:166: writing trial04342147-4 from 1828021 to 2497055 tid 2 2026-02-20T11:16:21.009 INFO:tasks.rados.rados.0.trial043.stdout:166: writing trial04342147-4 from 3134120 to 3764735 tid 3 2026-02-20T11:16:21.009 INFO:tasks.rados.rados.0.trial043.stdout:159: done (6 left) 2026-02-20T11:16:21.009 INFO:tasks.rados.rados.0.trial043.stdout:162: done (5 left) 2026-02-20T11:16:21.009 INFO:tasks.rados.rados.0.trial043.stdout:167: delete oid 20 current snap is 13 2026-02-20T11:16:21.009 INFO:tasks.rados.rados.0.trial043.stdout:164: expect (ObjNum 4 snap 0 seq_num 4) 2026-02-20T11:16:21.019 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1386226 2026-02-20T11:16:21.028 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:16:21.046 INFO:tasks.rados.rados.0.trial043.stdout:165: expect (ObjNum 90 snap 10 seq_num 90) 2026-02-20T11:16:21.158 INFO:tasks.rados.rados.0.trial043.stdout:161: finishing write tid 1 to trial04342147-40 2026-02-20T11:16:21.158 INFO:tasks.rados.rados.0.trial043.stdout:161: finishing write tid 2 to trial04342147-40 2026-02-20T11:16:21.158 INFO:tasks.rados.rados.0.trial043.stdout:161: finishing write tid 3 to trial04342147-40 2026-02-20T11:16:21.159 INFO:tasks.rados.rados.0.trial043.stdout:161: finishing write tid 4 to trial04342147-40 2026-02-20T11:16:21.159 INFO:tasks.rados.rados.0.trial043.stdout:161: finishing write tid 5 to trial04342147-40 2026-02-20T11:16:21.159 INFO:tasks.rados.rados.0.trial043.stdout:161: oid 40 updating version 0 to 26 2026-02-20T11:16:21.159 INFO:tasks.rados.rados.0.trial043.stdout:161: oid 40 updating version 26 to 29 2026-02-20T11:16:21.159 INFO:tasks.rados.rados.0.trial043.stdout:161: oid 40 version 29 is already newer than 29 2026-02-20T11:16:21.159 INFO:tasks.rados.rados.0.trial043.stdout:161: oid 40 version 29 is already newer than 29 2026-02-20T11:16:21.159 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 40 v 29 (ObjNum 103 snap 13 seq_num 103) dirty exists 2026-02-20T11:16:21.159 INFO:tasks.rados.rados.0.trial043.stdout:161: left oid 40 (ObjNum 103 snap 13 seq_num 103) 2026-02-20T11:16:21.159 INFO:tasks.rados.rados.0.trial043.stdout:163: finishing write tid 1 to trial04342147-34 2026-02-20T11:16:21.159 INFO:tasks.rados.rados.0.trial043.stdout:163: finishing write tid 2 to trial04342147-34 2026-02-20T11:16:21.159 INFO:tasks.rados.rados.0.trial043.stdout:163: finishing write tid 3 to trial04342147-34 2026-02-20T11:16:21.160 INFO:tasks.rados.rados.0.trial043.stdout:161: done (5 left) 2026-02-20T11:16:21.160 INFO:tasks.rados.rados.0.trial043.stdout:164: done (4 left) 2026-02-20T11:16:21.160 INFO:tasks.rados.rados.0.trial043.stdout:165: done (3 left) 2026-02-20T11:16:21.160 INFO:tasks.rados.rados.0.trial043.stdout:167: done (2 left) 2026-02-20T11:16:21.160 INFO:tasks.rados.rados.0.trial043.stdout:168: delete oid 35 current snap is 13 2026-02-20T11:16:21.160 INFO:tasks.rados.rados.0.trial043.stdout:163: finishing write tid 4 to trial04342147-34 2026-02-20T11:16:21.160 INFO:tasks.rados.rados.0.trial043.stdout:163: finishing write tid 5 to trial04342147-34 2026-02-20T11:16:21.160 INFO:tasks.rados.rados.0.trial043.stdout:163: oid 34 updating version 0 to 20 2026-02-20T11:16:21.160 INFO:tasks.rados.rados.0.trial043.stdout:163: oid 34 updating version 20 to 23 2026-02-20T11:16:21.160 INFO:tasks.rados.rados.0.trial043.stdout:163: oid 34 version 23 is already newer than 23 2026-02-20T11:16:21.160 INFO:tasks.rados.rados.0.trial043.stdout:163: oid 34 version 23 is already newer than 23 2026-02-20T11:16:21.160 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 34 v 23 (ObjNum 104 snap 13 seq_num 104) dirty exists 2026-02-20T11:16:21.161 INFO:tasks.rados.rados.0.trial043.stdout:163: left oid 34 (ObjNum 104 snap 13 seq_num 104) 2026-02-20T11:16:21.161 INFO:tasks.rados.rados.0.trial043.stdout:166: finishing write tid 1 to trial04342147-4 2026-02-20T11:16:21.161 INFO:tasks.rados.rados.0.trial043.stdout:166: finishing write tid 2 to trial04342147-4 2026-02-20T11:16:21.161 INFO:tasks.rados.rados.0.trial043.stdout:166: finishing write tid 3 to trial04342147-4 2026-02-20T11:16:21.161 INFO:tasks.rados.rados.0.trial043.stdout:166: finishing write tid 4 to trial04342147-4 2026-02-20T11:16:21.161 INFO:tasks.rados.rados.0.trial043.stdout:166: finishing write tid 5 to trial04342147-4 2026-02-20T11:16:21.161 INFO:tasks.rados.rados.0.trial043.stdout:166: oid 4 updating version 0 to 24 2026-02-20T11:16:21.161 INFO:tasks.rados.rados.0.trial043.stdout:166: oid 4 version 24 is already newer than 24 2026-02-20T11:16:21.161 INFO:tasks.rados.rados.0.trial043.stdout:166: oid 4 version 24 is already newer than 24 2026-02-20T11:16:21.161 INFO:tasks.rados.rados.0.trial043.stdout:166: oid 4 version 24 is already newer than 24 2026-02-20T11:16:21.161 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 4 v 24 (ObjNum 105 snap 13 seq_num 105) dirty exists 2026-02-20T11:16:21.161 INFO:tasks.rados.rados.0.trial043.stdout:166: left oid 4 (ObjNum 105 snap 13 seq_num 105) 2026-02-20T11:16:21.182 INFO:tasks.rados.rados.0.trial043.stdout:163: done (2 left) 2026-02-20T11:16:21.182 INFO:tasks.rados.rados.0.trial043.stdout:166: done (1 left) 2026-02-20T11:16:21.182 INFO:tasks.rados.rados.0.trial043.stdout:168: done (0 left) 2026-02-20T11:16:21.182 INFO:tasks.rados.rados.0.trial043.stdout:169: write (excl) oid 24 current snap is 13 2026-02-20T11:16:21.182 INFO:tasks.rados.rados.0.trial043.stdout:169: seq_num 108 ranges {615156=555834,1603467=416361,2816389=693495} 2026-02-20T11:16:21.186 INFO:tasks.rados.rados.0.trial043.stdout:169: writing trial04342147-24 from 615156 to 1170990 tid 1 2026-02-20T11:16:21.190 INFO:tasks.rados.rados.0.trial043.stdout:169: writing trial04342147-24 from 1603467 to 2019828 tid 2 2026-02-20T11:16:21.196 INFO:tasks.rados.rados.0.trial043.stdout:169: writing trial04342147-24 from 2816389 to 3509884 tid 3 2026-02-20T11:16:21.196 INFO:tasks.rados.rados.0.trial043.stdout:170: snap_remove snap 11 2026-02-20T11:16:21.520 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:16:21.591 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:16:21.594 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:16:21.595 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:16:21.815 INFO:tasks.rados.rados.0.trial043.stdout:170: done (1 left) 2026-02-20T11:16:21.816 INFO:tasks.rados.rados.0.trial043.stdout:171: write (excl) oid 49 current snap is 13 2026-02-20T11:16:21.816 INFO:tasks.rados.rados.0.trial043.stdout:169: finishing write tid 1 to trial04342147-24 2026-02-20T11:16:21.816 INFO:tasks.rados.rados.0.trial043.stdout:171: seq_num 109 ranges {631963=631083,2061949=560019,3243812=504767} 2026-02-20T11:16:21.820 INFO:tasks.rados.rados.0.trial043.stdout:171: writing trial04342147-49 from 631963 to 1263046 tid 1 2026-02-20T11:16:21.826 INFO:tasks.rados.rados.0.trial043.stdout:171: writing trial04342147-49 from 2061949 to 2621968 tid 2 2026-02-20T11:16:21.831 INFO:tasks.rados.rados.0.trial043.stdout:171: writing trial04342147-49 from 3243812 to 3748579 tid 3 2026-02-20T11:16:21.831 INFO:tasks.rados.rados.0.trial043.stdout:172: delete oid 13 current snap is 13 2026-02-20T11:16:21.831 INFO:tasks.rados.rados.0.trial043.stdout:169: finishing write tid 2 to trial04342147-24 2026-02-20T11:16:21.831 INFO:tasks.rados.rados.0.trial043.stdout:169: finishing write tid 3 to trial04342147-24 2026-02-20T11:16:21.831 INFO:tasks.rados.rados.0.trial043.stdout:169: finishing write tid 4 to trial04342147-24 2026-02-20T11:16:21.831 INFO:tasks.rados.rados.0.trial043.stdout:169: finishing write tid 5 to trial04342147-24 2026-02-20T11:16:21.831 INFO:tasks.rados.rados.0.trial043.stdout:169: oid 24 updating version 0 to 14 2026-02-20T11:16:21.831 INFO:tasks.rados.rados.0.trial043.stdout:169: oid 24 version 14 is already newer than 14 2026-02-20T11:16:21.831 INFO:tasks.rados.rados.0.trial043.stdout:169: oid 24 version 14 is already newer than 14 2026-02-20T11:16:21.831 INFO:tasks.rados.rados.0.trial043.stdout:169: oid 24 version 14 is already newer than 11 2026-02-20T11:16:21.832 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 24 v 14 (ObjNum 108 snap 13 seq_num 108) dirty exists 2026-02-20T11:16:21.832 INFO:tasks.rados.rados.0.trial043.stdout:169: left oid 24 (ObjNum 108 snap 13 seq_num 108) 2026-02-20T11:16:21.845 INFO:tasks.rados.rados.0.trial043.stdout:171: finishing write tid 1 to trial04342147-49 2026-02-20T11:16:21.846 INFO:tasks.rados.rados.0.trial043.stdout:169: done (2 left) 2026-02-20T11:16:21.846 INFO:tasks.rados.rados.0.trial043.stdout:172: done (1 left) 2026-02-20T11:16:21.846 INFO:tasks.rados.rados.0.trial043.stdout:173: delete oid 29 current snap is 13 2026-02-20T11:16:21.859 INFO:tasks.rados.rados.0.trial043.stdout:171: finishing write tid 2 to trial04342147-49 2026-02-20T11:16:21.863 INFO:tasks.rados.rados.0.trial043.stdout:173: done (1 left) 2026-02-20T11:16:21.864 INFO:tasks.rados.rados.0.trial043.stdout:174: read oid 26 snap -1 2026-02-20T11:16:21.864 INFO:tasks.rados.rados.0.trial043.stdout:174: expect (ObjNum 102 snap 11 seq_num 102) 2026-02-20T11:16:21.864 INFO:tasks.rados.rados.0.trial043.stdout:175: write oid 3 current snap is 13 2026-02-20T11:16:21.864 INFO:tasks.rados.rados.0.trial043.stdout:175: seq_num 112 ranges {682384=456830,1721042=423699,2809728=454936,3731357=151027} 2026-02-20T11:16:21.868 INFO:tasks.rados.rados.0.trial043.stdout:175: writing trial04342147-3 from 682384 to 1139214 tid 1 2026-02-20T11:16:21.871 INFO:tasks.rados.rados.0.trial043.stdout:175: writing trial04342147-3 from 1721042 to 2144741 tid 2 2026-02-20T11:16:21.876 INFO:tasks.rados.rados.0.trial043.stdout:175: writing trial04342147-3 from 2809728 to 3264664 tid 3 2026-02-20T11:16:21.879 INFO:tasks.rados.rados.0.trial043.stdout:175: writing trial04342147-3 from 3731357 to 3882384 tid 4 2026-02-20T11:16:21.880 INFO:tasks.rados.rados.0.trial043.stdout:176: write (excl) oid 37 current snap is 13 2026-02-20T11:16:21.880 INFO:tasks.rados.rados.0.trial043.stdout:171: finishing write tid 3 to trial04342147-49 2026-02-20T11:16:21.880 INFO:tasks.rados.rados.0.trial043.stdout:176: seq_num 113 ranges {699191=532079,1779524=567357,2837151=666208} 2026-02-20T11:16:21.883 INFO:tasks.rados.rados.0.trial043.stdout:176: writing trial04342147-37 from 699191 to 1231270 tid 1 2026-02-20T11:16:21.888 INFO:tasks.rados.rados.0.trial043.stdout:176: writing trial04342147-37 from 1779524 to 2346881 tid 2 2026-02-20T11:16:21.890 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:16:21.893 INFO:tasks.rados.rados.0.trial043.stdout:176: writing trial04342147-37 from 2837151 to 3503359 tid 3 2026-02-20T11:16:21.893 INFO:tasks.rados.rados.0.trial043.stdout:177: delete oid 45 current snap is 13 2026-02-20T11:16:21.894 INFO:tasks.rados.rados.0.trial043.stdout:171: finishing write tid 4 to trial04342147-49 2026-02-20T11:16:21.894 INFO:tasks.rados.rados.0.trial043.stdout:171: finishing write tid 5 to trial04342147-49 2026-02-20T11:16:21.894 INFO:tasks.rados.rados.0.trial043.stdout:171: oid 49 updating version 0 to 20 2026-02-20T11:16:21.894 INFO:tasks.rados.rados.0.trial043.stdout:171: oid 49 updating version 20 to 23 2026-02-20T11:16:21.894 INFO:tasks.rados.rados.0.trial043.stdout:171: oid 49 version 23 is already newer than 23 2026-02-20T11:16:21.895 INFO:tasks.rados.rados.0.trial043.stdout:171: oid 49 version 23 is already newer than 23 2026-02-20T11:16:21.895 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 49 v 23 (ObjNum 109 snap 13 seq_num 109) dirty exists 2026-02-20T11:16:21.895 INFO:tasks.rados.rados.0.trial043.stdout:171: left oid 49 (ObjNum 109 snap 13 seq_num 109) 2026-02-20T11:16:21.895 INFO:tasks.rados.rados.0.trial043.stdout:174: expect (ObjNum 102 snap 11 seq_num 102) 2026-02-20T11:16:21.895 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:16:21.948 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:16:21.967 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:21.965+0000 7fc00630b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:21.969 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:21.969+0000 7fc00630b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:21.997 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:16:22.012 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:16:22.012 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:16:22.018 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2082847 2026-02-20T11:16:22.052 INFO:tasks.rados.rados.0.trial043.stdout:175: finishing write tid 1 to trial04342147-3 2026-02-20T11:16:22.052 INFO:tasks.rados.rados.0.trial043.stdout:171: done (4 left) 2026-02-20T11:16:22.052 INFO:tasks.rados.rados.0.trial043.stdout:174: done (3 left) 2026-02-20T11:16:22.052 INFO:tasks.rados.rados.0.trial043.stdout:177: done (2 left) 2026-02-20T11:16:22.052 INFO:tasks.rados.rados.0.trial043.stdout:178: read oid 20 snap -1 2026-02-20T11:16:22.052 INFO:tasks.rados.rados.0.trial043.stdout:178: expect deleted 2026-02-20T11:16:22.053 INFO:tasks.rados.rados.0.trial043.stdout:179: snap_create 2026-02-20T11:16:22.053 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:16:22.053 INFO:tasks.rados.rados.0.trial043.stdout:175: finishing write tid 2 to trial04342147-3 2026-02-20T11:16:22.053 INFO:tasks.rados.rados.0.trial043.stdout:175: finishing write tid 3 to trial04342147-3 2026-02-20T11:16:22.053 INFO:tasks.rados.rados.0.trial043.stdout:175: finishing write tid 4 to trial04342147-3 2026-02-20T11:16:22.053 INFO:tasks.rados.rados.0.trial043.stdout:175: finishing write tid 5 to trial04342147-3 2026-02-20T11:16:22.053 INFO:tasks.rados.rados.0.trial043.stdout:175: finishing write tid 6 to trial04342147-3 2026-02-20T11:16:22.053 INFO:tasks.rados.rados.0.trial043.stdout:175: oid 3 updating version 0 to 57 2026-02-20T11:16:22.053 INFO:tasks.rados.rados.0.trial043.stdout:175: oid 3 version 57 is already newer than 57 2026-02-20T11:16:22.053 INFO:tasks.rados.rados.0.trial043.stdout:175: oid 3 version 57 is already newer than 57 2026-02-20T11:16:22.053 INFO:tasks.rados.rados.0.trial043.stdout:175: oid 3 version 57 is already newer than 57 2026-02-20T11:16:22.053 INFO:tasks.rados.rados.0.trial043.stdout:175: oid 3 version 57 is already newer than 57 2026-02-20T11:16:22.053 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 3 v 57 (ObjNum 112 snap 13 seq_num 112) dirty exists 2026-02-20T11:16:22.053 INFO:tasks.rados.rados.0.trial043.stdout:175: left oid 3 (ObjNum 112 snap 13 seq_num 112) 2026-02-20T11:16:22.053 INFO:tasks.rados.rados.0.trial043.stdout:176: finishing write tid 1 to trial04342147-37 2026-02-20T11:16:22.053 INFO:tasks.rados.rados.0.trial043.stdout:176: finishing write tid 2 to trial04342147-37 2026-02-20T11:16:22.053 INFO:tasks.rados.rados.0.trial043.stdout:175: done (2 left) 2026-02-20T11:16:22.053 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:16:22.053 INFO:tasks.rados.rados.0.trial043.stdout:176: finishing write tid 3 to trial04342147-37 2026-02-20T11:16:22.054 INFO:tasks.rados.rados.0.trial043.stdout:176: finishing write tid 4 to trial04342147-37 2026-02-20T11:16:22.054 INFO:tasks.rados.rados.0.trial043.stdout:176: finishing write tid 5 to trial04342147-37 2026-02-20T11:16:22.054 INFO:tasks.rados.rados.0.trial043.stdout:176: oid 37 updating version 0 to 58 2026-02-20T11:16:22.054 INFO:tasks.rados.rados.0.trial043.stdout:176: oid 37 updating version 58 to 61 2026-02-20T11:16:22.054 INFO:tasks.rados.rados.0.trial043.stdout:176: oid 37 version 61 is already newer than 61 2026-02-20T11:16:22.054 INFO:tasks.rados.rados.0.trial043.stdout:176: oid 37 version 61 is already newer than 61 2026-02-20T11:16:22.054 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 37 v 61 (ObjNum 113 snap 13 seq_num 113) dirty exists 2026-02-20T11:16:22.054 INFO:tasks.rados.rados.0.trial043.stdout:176: left oid 37 (ObjNum 113 snap 13 seq_num 113) 2026-02-20T11:16:22.054 INFO:tasks.rados.rados.0.trial043.stdout:176: done (1 left) 2026-02-20T11:16:22.054 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:16:22.056 INFO:tasks.rados.rados.0.trial043.stdout:178: done (0 left) 2026-02-20T11:16:22.820 INFO:tasks.rados.rados.0.trial043.stdout:179: done (0 left) 2026-02-20T11:16:22.821 INFO:tasks.rados.rados.0.trial043.stdout:180: write (excl) oid 45 current snap is 14 2026-02-20T11:16:22.821 INFO:tasks.rados.rados.0.trial043.stdout:180: seq_num 115 ranges {732805=798930,2296488=571026,3291997=405105} 2026-02-20T11:16:22.826 INFO:tasks.rados.rados.0.trial043.stdout:180: writing trial04342147-45 from 732805 to 1531735 tid 1 2026-02-20T11:16:22.832 INFO:tasks.rados.rados.0.trial043.stdout:180: writing trial04342147-45 from 2296488 to 2867514 tid 2 2026-02-20T11:16:22.835 INFO:tasks.rados.rados.0.trial043.stdout:180: writing trial04342147-45 from 3291997 to 3697102 tid 3 2026-02-20T11:16:22.835 INFO:tasks.rados.rados.0.trial043.stdout:181: write oid 25 current snap is 14 2026-02-20T11:16:22.835 INFO:tasks.rados.rados.0.trial043.stdout:181: seq_num 116 ranges {749612=474179,1954970=431037,2919420=616377} 2026-02-20T11:16:22.839 INFO:tasks.rados.rados.0.trial043.stdout:181: writing trial04342147-25 from 749612 to 1223791 tid 1 2026-02-20T11:16:22.844 INFO:tasks.rados.rados.0.trial043.stdout:181: writing trial04342147-25 from 1954970 to 2386007 tid 2 2026-02-20T11:16:22.848 INFO:tasks.rados.rados.0.trial043.stdout:181: writing trial04342147-25 from 2919420 to 3535797 tid 3 2026-02-20T11:16:22.848 INFO:tasks.rados.rados.0.trial043.stdout:182: write (excl) oid 19 current snap is 14 2026-02-20T11:16:22.848 INFO:tasks.rados.rados.0.trial043.stdout:182: seq_num 117 ranges {766419=549428,2013452=574695,3346843=427649} 2026-02-20T11:16:22.853 INFO:tasks.rados.rados.0.trial043.stdout:182: writing trial04342147-19 from 766419 to 1315847 tid 1 2026-02-20T11:16:22.858 INFO:tasks.rados.rados.0.trial043.stdout:182: writing trial04342147-19 from 2013452 to 2588147 tid 2 2026-02-20T11:16:22.865 INFO:tasks.rados.rados.0.trial043.stdout:182: writing trial04342147-19 from 3346843 to 3774492 tid 3 2026-02-20T11:16:22.865 INFO:tasks.rados.rados.0.trial043.stdout:183: read oid 44 snap -1 2026-02-20T11:16:22.865 INFO:tasks.rados.rados.0.trial043.stdout:183: expect (ObjNum 43 snap 0 seq_num 43) 2026-02-20T11:16:22.865 INFO:tasks.rados.rados.0.trial043.stdout:184: delete oid 27 current snap is 14 2026-02-20T11:16:22.865 INFO:tasks.rados.rados.0.trial043.stdout:180: finishing write tid 1 to trial04342147-45 2026-02-20T11:16:22.877 INFO:tasks.rados.rados.0.trial043.stdout:182: finishing write tid 1 to trial04342147-19 2026-02-20T11:16:22.882 INFO:tasks.rados.rados.0.trial043.stdout:180: finishing write tid 2 to trial04342147-45 2026-02-20T11:16:22.891 INFO:tasks.rados.rados.0.trial043.stdout:182: finishing write tid 2 to trial04342147-19 2026-02-20T11:16:22.892 INFO:tasks.rados.rados.0.trial043.stdout:180: finishing write tid 3 to trial04342147-45 2026-02-20T11:16:22.898 INFO:tasks.rados.rados.0.trial043.stdout:182: finishing write tid 3 to trial04342147-19 2026-02-20T11:16:22.899 INFO:tasks.rados.rados.0.trial043.stdout:183: expect (ObjNum 43 snap 0 seq_num 43) 2026-02-20T11:16:22.946 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:16:22.970 INFO:tasks.rados.rados.0.trial043.stdout:183: done (4 left) 2026-02-20T11:16:22.970 INFO:tasks.rados.rados.0.trial043.stdout:184: done (3 left) 2026-02-20T11:16:22.970 INFO:tasks.rados.rados.0.trial043.stdout:185: read oid 34 snap -1 2026-02-20T11:16:22.970 INFO:tasks.rados.rados.0.trial043.stdout:185: expect (ObjNum 104 snap 13 seq_num 104) 2026-02-20T11:16:22.971 INFO:tasks.rados.rados.0.trial043.stdout:186: write (excl) oid 47 current snap is 14 2026-02-20T11:16:22.971 INFO:tasks.rados.rados.0.trial043.stdout:180: finishing write tid 4 to trial04342147-45 2026-02-20T11:16:22.971 INFO:tasks.rados.rados.0.trial043.stdout:180: finishing write tid 5 to trial04342147-45 2026-02-20T11:16:22.971 INFO:tasks.rados.rados.0.trial043.stdout:180: oid 45 updating version 0 to 63 2026-02-20T11:16:22.971 INFO:tasks.rados.rados.0.trial043.stdout:180: oid 45 updating version 63 to 66 2026-02-20T11:16:22.971 INFO:tasks.rados.rados.0.trial043.stdout:180: oid 45 version 66 is already newer than 66 2026-02-20T11:16:22.971 INFO:tasks.rados.rados.0.trial043.stdout:180: oid 45 version 66 is already newer than 66 2026-02-20T11:16:22.971 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 45 v 66 (ObjNum 115 snap 14 seq_num 115) dirty exists 2026-02-20T11:16:22.971 INFO:tasks.rados.rados.0.trial043.stdout:180: left oid 45 (ObjNum 115 snap 14 seq_num 115) 2026-02-20T11:16:22.971 INFO:tasks.rados.rados.0.trial043.stdout:182: finishing write tid 4 to trial04342147-19 2026-02-20T11:16:22.971 INFO:tasks.rados.rados.0.trial043.stdout:182: finishing write tid 5 to trial04342147-19 2026-02-20T11:16:22.972 INFO:tasks.rados.rados.0.trial043.stdout:182: oid 19 updating version 0 to 12 2026-02-20T11:16:22.972 INFO:tasks.rados.rados.0.trial043.stdout:182: oid 19 version 12 is already newer than 12 2026-02-20T11:16:22.972 INFO:tasks.rados.rados.0.trial043.stdout:182: oid 19 version 12 is already newer than 9 2026-02-20T11:16:22.972 INFO:tasks.rados.rados.0.trial043.stdout:182: oid 19 version 12 is already newer than 12 2026-02-20T11:16:22.972 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 19 v 12 (ObjNum 117 snap 14 seq_num 117) dirty exists 2026-02-20T11:16:22.972 INFO:tasks.rados.rados.0.trial043.stdout:182: left oid 19 (ObjNum 117 snap 14 seq_num 117) 2026-02-20T11:16:22.972 INFO:tasks.rados.rados.0.trial043.stdout:181: finishing write tid 1 to trial04342147-25 2026-02-20T11:16:22.972 INFO:tasks.rados.rados.0.trial043.stdout:181: finishing write tid 2 to trial04342147-25 2026-02-20T11:16:22.972 INFO:tasks.rados.rados.0.trial043.stdout:181: finishing write tid 3 to trial04342147-25 2026-02-20T11:16:22.972 INFO:tasks.rados.rados.0.trial043.stdout:186: seq_num 119 ranges {400033=699926,1614063=385970} 2026-02-20T11:16:22.973 INFO:tasks.rados.rados.0.trial043.stdout:186: writing trial04342147-47 from 400033 to 1099959 tid 1 2026-02-20T11:16:22.977 INFO:tasks.rados.rados.0.trial043.stdout:186: writing trial04342147-47 from 1614063 to 2000033 tid 2 2026-02-20T11:16:22.977 INFO:tasks.rados.rados.0.trial043.stdout:180: done (4 left) 2026-02-20T11:16:22.977 INFO:tasks.rados.rados.0.trial043.stdout:182: done (3 left) 2026-02-20T11:16:22.977 INFO:tasks.rados.rados.0.trial043.stdout:187: snap_remove snap 13 2026-02-20T11:16:23.824 INFO:tasks.rados.rados.0.trial043.stdout:187: done (3 left) 2026-02-20T11:16:23.824 INFO:tasks.rados.rados.0.trial043.stdout:188: read oid 21 snap -1 2026-02-20T11:16:23.825 INFO:tasks.rados.rados.0.trial043.stdout:188: expect deleted 2026-02-20T11:16:23.825 INFO:tasks.rados.rados.0.trial043.stdout:189: snap_create 2026-02-20T11:16:23.825 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 4 2026-02-20T11:16:23.825 INFO:tasks.rados.rados.0.trial043.stdout:181: finishing write tid 4 to trial04342147-25 2026-02-20T11:16:23.825 INFO:tasks.rados.rados.0.trial043.stdout:181: finishing write tid 5 to trial04342147-25 2026-02-20T11:16:23.825 INFO:tasks.rados.rados.0.trial043.stdout:181: oid 25 updating version 0 to 28 2026-02-20T11:16:23.825 INFO:tasks.rados.rados.0.trial043.stdout:181: oid 25 version 28 is already newer than 28 2026-02-20T11:16:23.825 INFO:tasks.rados.rados.0.trial043.stdout:181: oid 25 version 28 is already newer than 28 2026-02-20T11:16:23.825 INFO:tasks.rados.rados.0.trial043.stdout:181: oid 25 version 28 is already newer than 28 2026-02-20T11:16:23.825 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 25 v 28 (ObjNum 116 snap 14 seq_num 116) dirty exists 2026-02-20T11:16:23.825 INFO:tasks.rados.rados.0.trial043.stdout:181: left oid 25 (ObjNum 116 snap 14 seq_num 116) 2026-02-20T11:16:23.826 INFO:tasks.rados.rados.0.trial043.stdout:181: done (3 left) 2026-02-20T11:16:23.826 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:16:23.826 INFO:tasks.rados.rados.0.trial043.stdout:186: finishing write tid 1 to trial04342147-47 2026-02-20T11:16:23.826 INFO:tasks.rados.rados.0.trial043.stdout:185: expect (ObjNum 104 snap 13 seq_num 104) 2026-02-20T11:16:23.826 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:16:23.856 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~39s 2026-02-20T11:16:23.999 INFO:tasks.rados.rados.0.trial043.stdout:186: finishing write tid 2 to trial04342147-47 2026-02-20T11:16:23.999 INFO:tasks.rados.rados.0.trial043.stdout:186: finishing write tid 3 to trial04342147-47 2026-02-20T11:16:23.999 INFO:tasks.rados.rados.0.trial043.stdout:186: finishing write tid 4 to trial04342147-47 2026-02-20T11:16:23.999 INFO:tasks.rados.rados.0.trial043.stdout:186: oid 47 updating version 0 to 44 2026-02-20T11:16:23.999 INFO:tasks.rados.rados.0.trial043.stdout:186: oid 47 updating version 44 to 46 2026-02-20T11:16:23.999 INFO:tasks.rados.rados.0.trial043.stdout:186: oid 47 version 46 is already newer than 46 2026-02-20T11:16:23.999 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 47 v 46 (ObjNum 119 snap 14 seq_num 119) dirty exists 2026-02-20T11:16:23.999 INFO:tasks.rados.rados.0.trial043.stdout:186: left oid 47 (ObjNum 119 snap 14 seq_num 119) 2026-02-20T11:16:24.000 INFO:tasks.rados.rados.0.trial043.stdout:185: done (2 left) 2026-02-20T11:16:24.000 INFO:tasks.rados.rados.0.trial043.stdout:186: done (1 left) 2026-02-20T11:16:24.000 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:16:24.000 INFO:tasks.rados.rados.0.trial043.stdout:188: done (0 left) 2026-02-20T11:16:24.012 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:16:24.086 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:24.084+0000 7ff9999b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:24.089 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:24.088+0000 7ff9999b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:24.117 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:16:24.133 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:16:24.133 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:16:24.828 INFO:tasks.rados.rados.0.trial043.stdout:189: done (0 left) 2026-02-20T11:16:24.828 INFO:tasks.rados.rados.0.trial043.stdout:190: write oid 20 current snap is 15 2026-02-20T11:16:24.828 INFO:tasks.rados.rados.0.trial043.stdout:190: seq_num 120 ranges {416840=775175,1672545=344295} 2026-02-20T11:16:24.832 INFO:tasks.rados.rados.0.trial043.stdout:190: writing trial04342147-20 from 416840 to 1192015 tid 1 2026-02-20T11:16:24.836 INFO:tasks.rados.rados.0.trial043.stdout:190: writing trial04342147-20 from 1672545 to 2016840 tid 2 2026-02-20T11:16:24.836 INFO:tasks.rados.rados.0.trial043.stdout:191: read oid 25 snap -1 2026-02-20T11:16:24.837 INFO:tasks.rados.rados.0.trial043.stdout:191: expect (ObjNum 116 snap 14 seq_num 116) 2026-02-20T11:16:24.837 INFO:tasks.rados.rados.0.trial043.stdout:192: read oid 36 snap -1 2026-02-20T11:16:24.837 INFO:tasks.rados.rados.0.trial043.stdout:192: expect (ObjNum 83 snap 8 seq_num 83) 2026-02-20T11:16:24.837 INFO:tasks.rados.rados.0.trial043.stdout:193: read oid 30 snap -1 2026-02-20T11:16:24.837 INFO:tasks.rados.rados.0.trial043.stdout:193: expect deleted 2026-02-20T11:16:24.837 INFO:tasks.rados.rados.0.trial043.stdout:194: read oid 12 snap -1 2026-02-20T11:16:24.837 INFO:tasks.rados.rados.0.trial043.stdout:194: expect (ObjNum 73 snap 3 seq_num 73) 2026-02-20T11:16:24.837 INFO:tasks.rados.rados.0.trial043.stdout:195: read oid 23 snap -1 2026-02-20T11:16:24.837 INFO:tasks.rados.rados.0.trial043.stdout:195: expect (ObjNum 74 snap 3 seq_num 74) 2026-02-20T11:16:24.837 INFO:tasks.rados.rados.0.trial043.stdout:196: snap_create 2026-02-20T11:16:24.838 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 6 2026-02-20T11:16:24.844 INFO:tasks.rados.rados.0.trial043.stdout:193: done (5 left) 2026-02-20T11:16:24.844 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 5 2026-02-20T11:16:24.857 INFO:tasks.rados.rados.0.trial043.stdout:194: expect (ObjNum 73 snap 3 seq_num 73) 2026-02-20T11:16:24.899 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:16:24.982 INFO:tasks.rados.rados.0.trial043.stdout:192: expect (ObjNum 83 snap 8 seq_num 83) 2026-02-20T11:16:25.048 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 476111 2026-02-20T11:16:25.084 INFO:tasks.rados.rados.0.trial043.stdout:190: finishing write tid 1 to trial04342147-20 2026-02-20T11:16:25.084 INFO:tasks.rados.rados.0.trial043.stdout:190: finishing write tid 2 to trial04342147-20 2026-02-20T11:16:25.084 INFO:tasks.rados.rados.0.trial043.stdout:192: done (4 left) 2026-02-20T11:16:25.085 INFO:tasks.rados.rados.0.trial043.stdout:194: done (3 left) 2026-02-20T11:16:25.085 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:16:25.085 INFO:tasks.rados.rados.0.trial043.stdout:190: finishing write tid 3 to trial04342147-20 2026-02-20T11:16:25.085 INFO:tasks.rados.rados.0.trial043.stdout:195: expect (ObjNum 74 snap 3 seq_num 74) 2026-02-20T11:16:25.085 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:16:25.164 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1913153 2026-02-20T11:16:25.179 INFO:tasks.rados.rados.0.trial043.stdout:190: finishing write tid 4 to trial04342147-20 2026-02-20T11:16:25.180 INFO:tasks.rados.rados.0.trial043.stdout:190: oid 20 updating version 0 to 69 2026-02-20T11:16:25.180 INFO:tasks.rados.rados.0.trial043.stdout:190: oid 20 version 69 is already newer than 69 2026-02-20T11:16:25.180 INFO:tasks.rados.rados.0.trial043.stdout:190: oid 20 version 69 is already newer than 69 2026-02-20T11:16:25.180 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 20 v 69 (ObjNum 120 snap 15 seq_num 120) dirty exists 2026-02-20T11:16:25.180 INFO:tasks.rados.rados.0.trial043.stdout:190: left oid 20 (ObjNum 120 snap 15 seq_num 120) 2026-02-20T11:16:25.180 INFO:tasks.rados.rados.0.trial043.stdout:191: expect (ObjNum 116 snap 14 seq_num 116) 2026-02-20T11:16:25.264 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2975496 2026-02-20T11:16:25.291 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1909123 2026-02-20T11:16:25.322 INFO:tasks.rados.rados.0.trial043.stdout:190: done (2 left) 2026-02-20T11:16:25.323 INFO:tasks.rados.rados.0.trial043.stdout:191: done (1 left) 2026-02-20T11:16:25.323 INFO:tasks.rados.rados.0.trial043.stdout:195: done (0 left) 2026-02-20T11:16:25.830 INFO:tasks.rados.rados.0.trial043.stdout:196: done (0 left) 2026-02-20T11:16:25.830 INFO:tasks.rados.rados.0.trial043.stdout:197: read oid 44 snap -1 2026-02-20T11:16:25.831 INFO:tasks.rados.rados.0.trial043.stdout:197: expect (ObjNum 43 snap 0 seq_num 43) 2026-02-20T11:16:25.831 INFO:tasks.rados.rados.0.trial043.stdout:198: snap_remove snap 15 2026-02-20T11:16:26.135 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:16:26.211 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:26.209+0000 7f7e2facb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:26.214 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:26.213+0000 7f7e2facb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:26.239 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:16:26.256 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:16:26.256 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:16:26.595 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:16:26.668 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:16:26.670 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:16:26.671 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:16:26.834 INFO:tasks.rados.rados.0.trial043.stdout:198: done (1 left) 2026-02-20T11:16:26.834 INFO:tasks.rados.rados.0.trial043.stdout:199: write (excl) oid 17 current snap is 16 2026-02-20T11:16:26.834 INFO:tasks.rados.rados.0.trial043.stdout:199: seq_num 121 ranges {433647=450424,1331027=582033} 2026-02-20T11:16:26.837 INFO:tasks.rados.rados.0.trial043.stdout:199: writing trial04342147-17 from 433647 to 884071 tid 1 2026-02-20T11:16:26.841 INFO:tasks.rados.rados.0.trial043.stdout:199: writing trial04342147-17 from 1331027 to 1913060 tid 2 2026-02-20T11:16:26.841 INFO:tasks.rados.rados.0.trial043.stdout:200: delete oid 46 current snap is 16 2026-02-20T11:16:26.841 INFO:tasks.rados.rados.0.trial043.stdout:197: expect (ObjNum 43 snap 0 seq_num 43) 2026-02-20T11:16:26.851 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 621680 2026-02-20T11:16:26.870 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 130767 2026-02-20T11:16:26.893 INFO:tasks.rados.rados.0.trial043.stdout:197: done (2 left) 2026-02-20T11:16:26.893 INFO:tasks.rados.rados.0.trial043.stdout:200: done (1 left) 2026-02-20T11:16:26.893 INFO:tasks.rados.rados.0.trial043.stdout:201: snap_remove snap 14 2026-02-20T11:16:27.840 INFO:tasks.rados.rados.0.trial043.stdout:201: done (1 left) 2026-02-20T11:16:27.840 INFO:tasks.rados.rados.0.trial043.stdout:202: delete oid 6 current snap is 16 2026-02-20T11:16:27.840 INFO:tasks.rados.rados.0.trial043.stdout:199: finishing write tid 1 to trial04342147-17 2026-02-20T11:16:27.840 INFO:tasks.rados.rados.0.trial043.stdout:199: finishing write tid 2 to trial04342147-17 2026-02-20T11:16:27.840 INFO:tasks.rados.rados.0.trial043.stdout:199: finishing write tid 3 to trial04342147-17 2026-02-20T11:16:27.841 INFO:tasks.rados.rados.0.trial043.stdout:199: finishing write tid 4 to trial04342147-17 2026-02-20T11:16:27.841 INFO:tasks.rados.rados.0.trial043.stdout:199: oid 17 updating version 0 to 45 2026-02-20T11:16:27.841 INFO:tasks.rados.rados.0.trial043.stdout:199: oid 17 updating version 45 to 47 2026-02-20T11:16:27.841 INFO:tasks.rados.rados.0.trial043.stdout:199: oid 17 version 47 is already newer than 47 2026-02-20T11:16:27.841 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 17 v 47 (ObjNum 121 snap 16 seq_num 121) dirty exists 2026-02-20T11:16:27.841 INFO:tasks.rados.rados.0.trial043.stdout:199: left oid 17 (ObjNum 121 snap 16 seq_num 121) 2026-02-20T11:16:27.862 INFO:tasks.rados.rados.0.trial043.stdout:199: done (1 left) 2026-02-20T11:16:27.862 INFO:tasks.rados.rados.0.trial043.stdout:202: done (0 left) 2026-02-20T11:16:27.863 INFO:tasks.rados.rados.0.trial043.stdout:203: write (excl) oid 13 current snap is 16 2026-02-20T11:16:27.863 INFO:tasks.rados.rados.0.trial043.stdout:203: seq_num 124 ranges {484068=792524,1906473=177595} 2026-02-20T11:16:27.868 INFO:tasks.rados.rados.0.trial043.stdout:203: writing trial04342147-13 from 484068 to 1276592 tid 1 2026-02-20T11:16:27.871 INFO:tasks.rados.rados.0.trial043.stdout:203: writing trial04342147-13 from 1906473 to 2084068 tid 2 2026-02-20T11:16:27.872 INFO:tasks.rados.rados.0.trial043.stdout:204: write oid 29 current snap is 16 2026-02-20T11:16:27.872 INFO:tasks.rados.rados.0.trial043.stdout:204: seq_num 125 ranges {500875=467773,1564955=535920} 2026-02-20T11:16:27.874 INFO:tasks.rados.rados.0.trial043.stdout:204: writing trial04342147-29 from 500875 to 968648 tid 1 2026-02-20T11:16:27.879 INFO:tasks.rados.rados.0.trial043.stdout:204: writing trial04342147-29 from 1564955 to 2100875 tid 2 2026-02-20T11:16:27.879 INFO:tasks.rados.rados.0.trial043.stdout:205: snap_create 2026-02-20T11:16:27.880 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:16:27.889 INFO:tasks.rados.rados.0.trial043.stdout:203: finishing write tid 1 to trial04342147-13 2026-02-20T11:16:27.891 INFO:tasks.rados.rados.0.trial043.stdout:204: finishing write tid 1 to trial04342147-29 2026-02-20T11:16:27.897 INFO:tasks.rados.rados.0.trial043.stdout:204: finishing write tid 2 to trial04342147-29 2026-02-20T11:16:27.900 INFO:tasks.rados.rados.0.trial043.stdout:204: finishing write tid 3 to trial04342147-29 2026-02-20T11:16:27.901 INFO:tasks.rados.rados.0.trial043.stdout:203: finishing write tid 2 to trial04342147-13 2026-02-20T11:16:27.902 INFO:tasks.rados.rados.0.trial043.stdout:204: finishing write tid 4 to trial04342147-29 2026-02-20T11:16:27.902 INFO:tasks.rados.rados.0.trial043.stdout:204: oid 29 updating version 0 to 34 2026-02-20T11:16:27.902 INFO:tasks.rados.rados.0.trial043.stdout:204: oid 29 version 34 is already newer than 34 2026-02-20T11:16:27.902 INFO:tasks.rados.rados.0.trial043.stdout:204: oid 29 version 34 is already newer than 34 2026-02-20T11:16:27.903 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 29 v 34 (ObjNum 125 snap 16 seq_num 125) dirty exists 2026-02-20T11:16:27.903 INFO:tasks.rados.rados.0.trial043.stdout:204: left oid 29 (ObjNum 125 snap 16 seq_num 125) 2026-02-20T11:16:27.903 INFO:tasks.rados.rados.0.trial043.stdout:204: done (1 left) 2026-02-20T11:16:27.903 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:16:27.905 INFO:tasks.rados.rados.0.trial043.stdout:203: finishing write tid 3 to trial04342147-13 2026-02-20T11:16:27.907 INFO:tasks.rados.rados.0.trial043.stdout:203: finishing write tid 4 to trial04342147-13 2026-02-20T11:16:27.907 INFO:tasks.rados.rados.0.trial043.stdout:203: oid 13 updating version 0 to 70 2026-02-20T11:16:27.907 INFO:tasks.rados.rados.0.trial043.stdout:203: oid 13 updating version 70 to 72 2026-02-20T11:16:27.907 INFO:tasks.rados.rados.0.trial043.stdout:203: oid 13 version 72 is already newer than 72 2026-02-20T11:16:27.907 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 13 v 72 (ObjNum 124 snap 16 seq_num 124) dirty exists 2026-02-20T11:16:27.907 INFO:tasks.rados.rados.0.trial043.stdout:203: left oid 13 (ObjNum 124 snap 16 seq_num 124) 2026-02-20T11:16:27.908 INFO:tasks.rados.rados.0.trial043.stdout:203: done (0 left) 2026-02-20T11:16:28.257 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:16:28.333 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:28.332+0000 7f9e47fec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:28.336 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:28.335+0000 7f9e47fec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:28.367 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:16:28.383 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:16:28.384 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:16:28.844 INFO:tasks.rados.rados.0.trial043.stdout:205: done (0 left) 2026-02-20T11:16:28.844 INFO:tasks.rados.rados.0.trial043.stdout:206: write (excl) oid 45 current snap is 17 2026-02-20T11:16:28.844 INFO:tasks.rados.rados.0.trial043.stdout:206: seq_num 126 ranges {517682=543022,1623437=494245} 2026-02-20T11:16:28.847 INFO:tasks.rados.rados.0.trial043.stdout:206: writing trial04342147-45 from 517682 to 1060704 tid 1 2026-02-20T11:16:28.852 INFO:tasks.rados.rados.0.trial043.stdout:206: writing trial04342147-45 from 1623437 to 2117682 tid 2 2026-02-20T11:16:28.852 INFO:tasks.rados.rados.0.trial043.stdout:207: snap_remove snap 16 2026-02-20T11:16:29.361 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~44s 2026-02-20T11:16:29.847 INFO:tasks.rados.rados.0.trial043.stdout:207: done (1 left) 2026-02-20T11:16:29.848 INFO:tasks.rados.rados.0.trial043.stdout:208: write (excl) oid 9 current snap is 17 2026-02-20T11:16:29.848 INFO:tasks.rados.rados.0.trial043.stdout:206: finishing write tid 1 to trial04342147-45 2026-02-20T11:16:29.848 INFO:tasks.rados.rados.0.trial043.stdout:208: seq_num 127 ranges {534489=618271,1565566=568923} 2026-02-20T11:16:29.851 INFO:tasks.rados.rados.0.trial043.stdout:208: writing trial04342147-9 from 534489 to 1152760 tid 1 2026-02-20T11:16:29.855 INFO:tasks.rados.rados.0.trial043.stdout:208: writing trial04342147-9 from 1565566 to 2134489 tid 2 2026-02-20T11:16:29.855 INFO:tasks.rados.rados.0.trial043.stdout:209: write (excl) oid 20 current snap is 17 2026-02-20T11:16:29.855 INFO:tasks.rados.rados.0.trial043.stdout:206: finishing write tid 2 to trial04342147-45 2026-02-20T11:16:29.855 INFO:tasks.rados.rados.0.trial043.stdout:209: seq_num 128 ranges {551296=693520,2024048=127248} 2026-02-20T11:16:29.860 INFO:tasks.rados.rados.0.trial043.stdout:209: writing trial04342147-20 from 551296 to 1244816 tid 1 2026-02-20T11:16:29.864 INFO:tasks.rados.rados.0.trial043.stdout:209: writing trial04342147-20 from 2024048 to 2151296 tid 2 2026-02-20T11:16:29.864 INFO:tasks.rados.rados.0.trial043.stdout:210: snap_create 2026-02-20T11:16:29.865 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:16:29.865 INFO:tasks.rados.rados.0.trial043.stdout:206: finishing write tid 3 to trial04342147-45 2026-02-20T11:16:29.865 INFO:tasks.rados.rados.0.trial043.stdout:206: finishing write tid 4 to trial04342147-45 2026-02-20T11:16:29.865 INFO:tasks.rados.rados.0.trial043.stdout:206: oid 45 updating version 0 to 76 2026-02-20T11:16:29.865 INFO:tasks.rados.rados.0.trial043.stdout:206: oid 45 version 76 is already newer than 74 2026-02-20T11:16:29.865 INFO:tasks.rados.rados.0.trial043.stdout:206: oid 45 version 76 is already newer than 76 2026-02-20T11:16:29.865 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 45 v 76 (ObjNum 126 snap 17 seq_num 126) dirty exists 2026-02-20T11:16:29.865 INFO:tasks.rados.rados.0.trial043.stdout:206: left oid 45 (ObjNum 126 snap 17 seq_num 126) 2026-02-20T11:16:29.865 INFO:tasks.rados.rados.0.trial043.stdout:206: done (2 left) 2026-02-20T11:16:29.865 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:16:29.876 INFO:tasks.rados.rados.0.trial043.stdout:208: finishing write tid 1 to trial04342147-9 2026-02-20T11:16:29.880 INFO:tasks.rados.rados.0.trial043.stdout:209: finishing write tid 1 to trial04342147-20 2026-02-20T11:16:29.890 INFO:tasks.rados.rados.0.trial043.stdout:208: finishing write tid 2 to trial04342147-9 2026-02-20T11:16:29.893 INFO:tasks.rados.rados.0.trial043.stdout:208: finishing write tid 3 to trial04342147-9 2026-02-20T11:16:29.894 INFO:tasks.rados.rados.0.trial043.stdout:209: finishing write tid 2 to trial04342147-20 2026-02-20T11:16:29.895 INFO:tasks.rados.rados.0.trial043.stdout:208: finishing write tid 4 to trial04342147-9 2026-02-20T11:16:29.895 INFO:tasks.rados.rados.0.trial043.stdout:208: oid 9 updating version 0 to 34 2026-02-20T11:16:29.895 INFO:tasks.rados.rados.0.trial043.stdout:208: oid 9 version 34 is already newer than 32 2026-02-20T11:16:29.895 INFO:tasks.rados.rados.0.trial043.stdout:208: oid 9 version 34 is already newer than 34 2026-02-20T11:16:29.895 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 9 v 34 (ObjNum 127 snap 17 seq_num 127) dirty exists 2026-02-20T11:16:29.895 INFO:tasks.rados.rados.0.trial043.stdout:208: left oid 9 (ObjNum 127 snap 17 seq_num 127) 2026-02-20T11:16:29.895 INFO:tasks.rados.rados.0.trial043.stdout:208: done (1 left) 2026-02-20T11:16:29.895 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:16:29.897 INFO:tasks.rados.rados.0.trial043.stdout:209: finishing write tid 3 to trial04342147-20 2026-02-20T11:16:29.900 INFO:tasks.rados.rados.0.trial043.stdout:209: finishing write tid 4 to trial04342147-20 2026-02-20T11:16:29.900 INFO:tasks.rados.rados.0.trial043.stdout:209: oid 20 updating version 0 to 79 2026-02-20T11:16:29.900 INFO:tasks.rados.rados.0.trial043.stdout:209: oid 20 version 79 is already newer than 79 2026-02-20T11:16:29.900 INFO:tasks.rados.rados.0.trial043.stdout:209: oid 20 version 79 is already newer than 77 2026-02-20T11:16:29.901 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 20 v 79 (ObjNum 128 snap 17 seq_num 128) dirty exists 2026-02-20T11:16:29.901 INFO:tasks.rados.rados.0.trial043.stdout:209: left oid 20 (ObjNum 128 snap 17 seq_num 128) 2026-02-20T11:16:29.901 INFO:tasks.rados.rados.0.trial043.stdout:209: done (0 left) 2026-02-20T11:16:30.385 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:16:30.461 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:30.459+0000 7f5dfc3fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:30.463 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:30.463+0000 7f5dfc3fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:30.489 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:16:30.505 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:16:30.505 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:16:30.851 INFO:tasks.rados.rados.0.trial043.stdout:210: done (0 left) 2026-02-20T11:16:30.851 INFO:tasks.rados.rados.0.trial043.stdout:211: snap_create 2026-02-20T11:16:31.672 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:16:31.745 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:16:31.748 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:16:31.748 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:16:31.856 INFO:tasks.rados.rados.0.trial043.stdout:211: done (0 left) 2026-02-20T11:16:31.856 INFO:tasks.rados.rados.0.trial043.stdout:212: snap_remove snap 18 2026-02-20T11:16:32.506 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:16:32.584 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:32.582+0000 7f8efcd0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:32.586 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:32.586+0000 7f8efcd0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:32.616 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:16:32.633 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:16:32.633 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:16:32.858 INFO:tasks.rados.rados.0.trial043.stdout:212: done (0 left) 2026-02-20T11:16:32.859 INFO:tasks.rados.rados.0.trial043.stdout:213: write (excl) oid 9 current snap is 19 2026-02-20T11:16:32.859 INFO:tasks.rados.rados.0.trial043.stdout:213: seq_num 129 ranges {568103=768769,2082530=85573} 2026-02-20T11:16:32.863 INFO:tasks.rados.rados.0.trial043.stdout:213: writing trial04342147-9 from 568103 to 1336872 tid 1 2026-02-20T11:16:32.866 INFO:tasks.rados.rados.0.trial043.stdout:213: writing trial04342147-9 from 2082530 to 2168103 tid 2 2026-02-20T11:16:32.867 INFO:tasks.rados.rados.0.trial043.stdout:214: snap_create 2026-02-20T11:16:32.867 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:16:32.895 INFO:tasks.rados.rados.0.trial043.stdout:213: finishing write tid 1 to trial04342147-9 2026-02-20T11:16:32.907 INFO:tasks.rados.rados.0.trial043.stdout:213: finishing write tid 2 to trial04342147-9 2026-02-20T11:16:32.910 INFO:tasks.rados.rados.0.trial043.stdout:213: finishing write tid 3 to trial04342147-9 2026-02-20T11:16:32.912 INFO:tasks.rados.rados.0.trial043.stdout:213: finishing write tid 4 to trial04342147-9 2026-02-20T11:16:32.912 INFO:tasks.rados.rados.0.trial043.stdout:213: oid 9 updating version 0 to 36 2026-02-20T11:16:32.912 INFO:tasks.rados.rados.0.trial043.stdout:213: oid 9 updating version 36 to 38 2026-02-20T11:16:32.913 INFO:tasks.rados.rados.0.trial043.stdout:213: oid 9 version 38 is already newer than 38 2026-02-20T11:16:32.913 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 9 v 38 (ObjNum 129 snap 19 seq_num 129) dirty exists 2026-02-20T11:16:32.913 INFO:tasks.rados.rados.0.trial043.stdout:213: left oid 9 (ObjNum 129 snap 19 seq_num 129) 2026-02-20T11:16:32.913 INFO:tasks.rados.rados.0.trial043.stdout:213: done (0 left) 2026-02-20T11:16:33.861 INFO:tasks.rados.rados.0.trial043.stdout:214: done (0 left) 2026-02-20T11:16:33.861 INFO:tasks.rados.rados.0.trial043.stdout:215: read oid 33 snap -1 2026-02-20T11:16:33.861 INFO:tasks.rados.rados.0.trial043.stdout:215: expect (ObjNum 32 snap 0 seq_num 32) 2026-02-20T11:16:33.861 INFO:tasks.rados.rados.0.trial043.stdout:216: snap_remove snap 17 2026-02-20T11:16:34.635 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:16:34.711 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:34.709+0000 7faaeda3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:34.714 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:34.714+0000 7faaeda3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:34.738 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:16:34.754 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:16:34.755 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:16:34.867 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~50s 2026-02-20T11:16:34.868 INFO:tasks.rados.rados.0.trial043.stdout:216: done (1 left) 2026-02-20T11:16:34.868 INFO:tasks.rados.rados.0.trial043.stdout:217: write (excl) oid 31 current snap is 20 2026-02-20T11:16:34.868 INFO:tasks.rados.rados.0.trial043.stdout:217: seq_num 130 ranges {584910=560371,1857365=327545} 2026-02-20T11:16:34.872 INFO:tasks.rados.rados.0.trial043.stdout:217: writing trial04342147-31 from 584910 to 1145281 tid 1 2026-02-20T11:16:34.876 INFO:tasks.rados.rados.0.trial043.stdout:217: writing trial04342147-31 from 1857365 to 2184910 tid 2 2026-02-20T11:16:34.876 INFO:tasks.rados.rados.0.trial043.stdout:218: delete oid 28 current snap is 20 2026-02-20T11:16:34.876 INFO:tasks.rados.rados.0.trial043.stdout:215: expect (ObjNum 32 snap 0 seq_num 32) 2026-02-20T11:16:34.898 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:16:34.941 INFO:tasks.rados.rados.0.trial043.stdout:215: done (2 left) 2026-02-20T11:16:34.941 INFO:tasks.rados.rados.0.trial043.stdout:218: done (1 left) 2026-02-20T11:16:34.942 INFO:tasks.rados.rados.0.trial043.stdout:219: delete oid 42 current snap is 20 2026-02-20T11:16:34.942 INFO:tasks.rados.rados.0.trial043.stdout:217: finishing write tid 1 to trial04342147-31 2026-02-20T11:16:34.942 INFO:tasks.rados.rados.0.trial043.stdout:217: finishing write tid 2 to trial04342147-31 2026-02-20T11:16:34.942 INFO:tasks.rados.rados.0.trial043.stdout:217: finishing write tid 3 to trial04342147-31 2026-02-20T11:16:34.942 INFO:tasks.rados.rados.0.trial043.stdout:217: finishing write tid 4 to trial04342147-31 2026-02-20T11:16:34.942 INFO:tasks.rados.rados.0.trial043.stdout:217: oid 31 updating version 0 to 27 2026-02-20T11:16:34.942 INFO:tasks.rados.rados.0.trial043.stdout:217: oid 31 version 27 is already newer than 25 2026-02-20T11:16:34.942 INFO:tasks.rados.rados.0.trial043.stdout:217: oid 31 version 27 is already newer than 27 2026-02-20T11:16:34.942 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 31 v 27 (ObjNum 130 snap 20 seq_num 130) dirty exists 2026-02-20T11:16:34.942 INFO:tasks.rados.rados.0.trial043.stdout:217: left oid 31 (ObjNum 130 snap 20 seq_num 130) 2026-02-20T11:16:34.967 INFO:tasks.rados.rados.0.trial043.stdout:217: done (1 left) 2026-02-20T11:16:34.967 INFO:tasks.rados.rados.0.trial043.stdout:219: done (0 left) 2026-02-20T11:16:34.967 INFO:tasks.rados.rados.0.trial043.stdout:220: read oid 3 snap -1 2026-02-20T11:16:34.968 INFO:tasks.rados.rados.0.trial043.stdout:220: expect (ObjNum 112 snap 13 seq_num 112) 2026-02-20T11:16:34.968 INFO:tasks.rados.rados.0.trial043.stdout:221: write (excl) oid 37 current snap is 20 2026-02-20T11:16:34.968 INFO:tasks.rados.rados.0.trial043.stdout:221: seq_num 133 ranges {635331=786118,1916458=318873} 2026-02-20T11:16:34.972 INFO:tasks.rados.rados.0.trial043.stdout:221: writing trial04342147-37 from 635331 to 1421449 tid 1 2026-02-20T11:16:34.975 INFO:tasks.rados.rados.0.trial043.stdout:221: writing trial04342147-37 from 1916458 to 2235331 tid 2 2026-02-20T11:16:34.976 INFO:tasks.rados.rados.0.trial043.stdout:222: write (excl) oid 11 current snap is 20 2026-02-20T11:16:34.976 INFO:tasks.rados.rados.0.trial043.stdout:222: seq_num 134 ranges {652138=461367,1574940=677198} 2026-02-20T11:16:34.979 INFO:tasks.rados.rados.0.trial043.stdout:222: writing trial04342147-11 from 652138 to 1113505 tid 1 2026-02-20T11:16:34.984 INFO:tasks.rados.rados.0.trial043.stdout:222: writing trial04342147-11 from 1574940 to 2252138 tid 2 2026-02-20T11:16:34.984 INFO:tasks.rados.rados.0.trial043.stdout:223: snap_remove snap 19 2026-02-20T11:16:35.873 INFO:tasks.rados.rados.0.trial043.stdout:223: done (3 left) 2026-02-20T11:16:35.874 INFO:tasks.rados.rados.0.trial043.stdout:224: write (excl) oid 19 current snap is 20 2026-02-20T11:16:35.874 INFO:tasks.rados.rados.0.trial043.stdout:224: seq_num 135 ranges {668945=536616,1633422=491363} 2026-02-20T11:16:35.878 INFO:tasks.rados.rados.0.trial043.stdout:224: writing trial04342147-19 from 668945 to 1205561 tid 1 2026-02-20T11:16:35.882 INFO:tasks.rados.rados.0.trial043.stdout:224: writing trial04342147-19 from 1633422 to 2124785 tid 2 2026-02-20T11:16:35.882 INFO:tasks.rados.rados.0.trial043.stdout:225: write oid 23 current snap is 20 2026-02-20T11:16:35.882 INFO:tasks.rados.rados.0.trial043.stdout:225: seq_num 136 ranges {685752=611865,1975551=310201} 2026-02-20T11:16:35.887 INFO:tasks.rados.rados.0.trial043.stdout:225: writing trial04342147-23 from 685752 to 1297617 tid 1 2026-02-20T11:16:35.891 INFO:tasks.rados.rados.0.trial043.stdout:225: writing trial04342147-23 from 1975551 to 2285752 tid 2 2026-02-20T11:16:35.891 INFO:tasks.rados.rados.0.trial043.stdout:226: snap_create 2026-02-20T11:16:35.891 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 5 2026-02-20T11:16:35.891 INFO:tasks.rados.rados.0.trial043.stdout:220: expect (ObjNum 112 snap 13 seq_num 112) 2026-02-20T11:16:35.905 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2021752 2026-02-20T11:16:35.992 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 3136295 2026-02-20T11:16:36.004 INFO:tasks.rados.rados.0.trial043.stdout:220: done (4 left) 2026-02-20T11:16:36.004 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 4 2026-02-20T11:16:36.004 INFO:tasks.rados.rados.0.trial043.stdout:222: finishing write tid 1 to trial04342147-11 2026-02-20T11:16:36.004 INFO:tasks.rados.rados.0.trial043.stdout:222: finishing write tid 2 to trial04342147-11 2026-02-20T11:16:36.004 INFO:tasks.rados.rados.0.trial043.stdout:221: finishing write tid 1 to trial04342147-37 2026-02-20T11:16:36.004 INFO:tasks.rados.rados.0.trial043.stdout:222: finishing write tid 3 to trial04342147-11 2026-02-20T11:16:36.004 INFO:tasks.rados.rados.0.trial043.stdout:222: finishing write tid 4 to trial04342147-11 2026-02-20T11:16:36.004 INFO:tasks.rados.rados.0.trial043.stdout:222: oid 11 updating version 0 to 38 2026-02-20T11:16:36.004 INFO:tasks.rados.rados.0.trial043.stdout:222: oid 11 version 38 is already newer than 38 2026-02-20T11:16:36.004 INFO:tasks.rados.rados.0.trial043.stdout:222: oid 11 version 38 is already newer than 36 2026-02-20T11:16:36.004 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 11 v 38 (ObjNum 134 snap 20 seq_num 134) dirty exists 2026-02-20T11:16:36.004 INFO:tasks.rados.rados.0.trial043.stdout:222: left oid 11 (ObjNum 134 snap 20 seq_num 134) 2026-02-20T11:16:36.004 INFO:tasks.rados.rados.0.trial043.stdout:221: finishing write tid 2 to trial04342147-37 2026-02-20T11:16:36.004 INFO:tasks.rados.rados.0.trial043.stdout:222: done (3 left) 2026-02-20T11:16:36.004 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:16:36.004 INFO:tasks.rados.rados.0.trial043.stdout:221: finishing write tid 3 to trial04342147-37 2026-02-20T11:16:36.004 INFO:tasks.rados.rados.0.trial043.stdout:221: finishing write tid 4 to trial04342147-37 2026-02-20T11:16:36.004 INFO:tasks.rados.rados.0.trial043.stdout:221: oid 37 updating version 0 to 83 2026-02-20T11:16:36.005 INFO:tasks.rados.rados.0.trial043.stdout:221: oid 37 version 83 is already newer than 81 2026-02-20T11:16:36.005 INFO:tasks.rados.rados.0.trial043.stdout:221: oid 37 version 83 is already newer than 83 2026-02-20T11:16:36.005 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 37 v 83 (ObjNum 133 snap 20 seq_num 133) dirty exists 2026-02-20T11:16:36.005 INFO:tasks.rados.rados.0.trial043.stdout:221: left oid 37 (ObjNum 133 snap 20 seq_num 133) 2026-02-20T11:16:36.005 INFO:tasks.rados.rados.0.trial043.stdout:221: done (2 left) 2026-02-20T11:16:36.005 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:16:36.005 INFO:tasks.rados.rados.0.trial043.stdout:225: finishing write tid 1 to trial04342147-23 2026-02-20T11:16:36.005 INFO:tasks.rados.rados.0.trial043.stdout:224: finishing write tid 1 to trial04342147-19 2026-02-20T11:16:36.005 INFO:tasks.rados.rados.0.trial043.stdout:225: finishing write tid 2 to trial04342147-23 2026-02-20T11:16:36.005 INFO:tasks.rados.rados.0.trial043.stdout:225: finishing write tid 3 to trial04342147-23 2026-02-20T11:16:36.005 INFO:tasks.rados.rados.0.trial043.stdout:225: finishing write tid 4 to trial04342147-23 2026-02-20T11:16:36.005 INFO:tasks.rados.rados.0.trial043.stdout:225: oid 23 updating version 0 to 36 2026-02-20T11:16:36.005 INFO:tasks.rados.rados.0.trial043.stdout:225: oid 23 version 36 is already newer than 36 2026-02-20T11:16:36.005 INFO:tasks.rados.rados.0.trial043.stdout:225: oid 23 version 36 is already newer than 36 2026-02-20T11:16:36.005 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 23 v 36 (ObjNum 136 snap 20 seq_num 136) dirty exists 2026-02-20T11:16:36.005 INFO:tasks.rados.rados.0.trial043.stdout:225: left oid 23 (ObjNum 136 snap 20 seq_num 136) 2026-02-20T11:16:36.005 INFO:tasks.rados.rados.0.trial043.stdout:224: finishing write tid 2 to trial04342147-19 2026-02-20T11:16:36.005 INFO:tasks.rados.rados.0.trial043.stdout:225: done (1 left) 2026-02-20T11:16:36.005 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:16:36.005 INFO:tasks.rados.rados.0.trial043.stdout:224: finishing write tid 3 to trial04342147-19 2026-02-20T11:16:36.005 INFO:tasks.rados.rados.0.trial043.stdout:224: finishing write tid 4 to trial04342147-19 2026-02-20T11:16:36.005 INFO:tasks.rados.rados.0.trial043.stdout:224: oid 19 updating version 0 to 15 2026-02-20T11:16:36.005 INFO:tasks.rados.rados.0.trial043.stdout:224: oid 19 version 15 is already newer than 15 2026-02-20T11:16:36.005 INFO:tasks.rados.rados.0.trial043.stdout:224: oid 19 version 15 is already newer than 13 2026-02-20T11:16:36.005 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 19 v 15 (ObjNum 135 snap 20 seq_num 135) dirty exists 2026-02-20T11:16:36.005 INFO:tasks.rados.rados.0.trial043.stdout:224: left oid 19 (ObjNum 135 snap 20 seq_num 135) 2026-02-20T11:16:36.006 INFO:tasks.rados.rados.0.trial043.stdout:224: done (0 left) 2026-02-20T11:16:36.749 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:16:36.755 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:16:36.820 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:16:36.823 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:16:36.824 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:16:36.831 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:36.830+0000 7f85409a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:36.834 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:36.834+0000 7f85409a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:36.860 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:16:36.876 INFO:tasks.rados.rados.0.trial043.stdout:226: done (0 left) 2026-02-20T11:16:36.876 INFO:tasks.rados.rados.0.trial043.stdout:227: read oid 13 snap -1 2026-02-20T11:16:36.876 INFO:tasks.rados.rados.0.trial043.stdout:227: expect (ObjNum 124 snap 16 seq_num 124) 2026-02-20T11:16:36.876 INFO:tasks.rados.rados.0.trial043.stdout:228: delete oid 37 current snap is 21 2026-02-20T11:16:36.877 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:16:36.877 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:16:36.907 INFO:tasks.rados.rados.0.trial043.stdout:227: expect (ObjNum 124 snap 16 seq_num 124) 2026-02-20T11:16:36.925 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 308650 2026-02-20T11:16:36.960 INFO:tasks.rados.rados.0.trial043.stdout:227: done (1 left) 2026-02-20T11:16:36.961 INFO:tasks.rados.rados.0.trial043.stdout:228: done (0 left) 2026-02-20T11:16:36.961 INFO:tasks.rados.rados.0.trial043.stdout:229: read oid 31 snap 20 2026-02-20T11:16:36.961 INFO:tasks.rados.rados.0.trial043.stdout:229: expect (ObjNum 130 snap 20 seq_num 130) 2026-02-20T11:16:36.961 INFO:tasks.rados.rados.0.trial043.stdout:230: read oid 10 snap -1 2026-02-20T11:16:36.961 INFO:tasks.rados.rados.0.trial043.stdout:230: expect (ObjNum 69 snap 3 seq_num 69) 2026-02-20T11:16:36.961 INFO:tasks.rados.rados.0.trial043.stdout:231: read oid 21 snap -1 2026-02-20T11:16:36.961 INFO:tasks.rados.rados.0.trial043.stdout:231: expect deleted 2026-02-20T11:16:36.961 INFO:tasks.rados.rados.0.trial043.stdout:232: read oid 2 snap -1 2026-02-20T11:16:36.961 INFO:tasks.rados.rados.0.trial043.stdout:232: expect (ObjNum 87 snap 9 seq_num 87) 2026-02-20T11:16:36.961 INFO:tasks.rados.rados.0.trial043.stdout:233: read oid 49 snap -1 2026-02-20T11:16:36.961 INFO:tasks.rados.rados.0.trial043.stdout:233: expect (ObjNum 109 snap 13 seq_num 109) 2026-02-20T11:16:36.962 INFO:tasks.rados.rados.0.trial043.stdout:234: read oid 28 snap -1 2026-02-20T11:16:36.962 INFO:tasks.rados.rados.0.trial043.stdout:234: expect deleted 2026-02-20T11:16:36.962 INFO:tasks.rados.rados.0.trial043.stdout:235: snap_create 2026-02-20T11:16:36.962 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 6 2026-02-20T11:16:36.964 INFO:tasks.rados.rados.0.trial043.stdout:231: done (5 left) 2026-02-20T11:16:36.964 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 5 2026-02-20T11:16:36.970 INFO:tasks.rados.rados.0.trial043.stdout:234: done (4 left) 2026-02-20T11:16:36.970 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 4 2026-02-20T11:16:36.979 INFO:tasks.rados.rados.0.trial043.stdout:229: expect (ObjNum 130 snap 20 seq_num 130) 2026-02-20T11:16:36.979 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:16:37.005 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 483607 2026-02-20T11:16:37.042 INFO:tasks.rados.rados.0.trial043.stdout:232: expect (ObjNum 87 snap 9 seq_num 87) 2026-02-20T11:16:37.097 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2542302 2026-02-20T11:16:37.113 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 783999 2026-02-20T11:16:37.142 INFO:tasks.rados.rados.0.trial043.stdout:230: expect (ObjNum 69 snap 3 seq_num 69) 2026-02-20T11:16:37.161 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2565775 2026-02-20T11:16:37.257 INFO:tasks.rados.rados.0.trial043.stdout:233: expect (ObjNum 109 snap 13 seq_num 109) 2026-02-20T11:16:37.299 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:16:37.356 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1713067 2026-02-20T11:16:37.384 INFO:tasks.rados.rados.0.trial043.stdout:229: done (3 left) 2026-02-20T11:16:37.384 INFO:tasks.rados.rados.0.trial043.stdout:230: done (2 left) 2026-02-20T11:16:37.384 INFO:tasks.rados.rados.0.trial043.stdout:232: done (1 left) 2026-02-20T11:16:37.384 INFO:tasks.rados.rados.0.trial043.stdout:233: done (0 left) 2026-02-20T11:16:37.880 INFO:tasks.rados.rados.0.trial043.stdout:235: done (0 left) 2026-02-20T11:16:37.880 INFO:tasks.rados.rados.0.trial043.stdout:236: read oid 16 snap -1 2026-02-20T11:16:37.880 INFO:tasks.rados.rados.0.trial043.stdout:236: expect (ObjNum 90 snap 10 seq_num 90) 2026-02-20T11:16:37.880 INFO:tasks.rados.rados.0.trial043.stdout:237: delete oid 15 current snap is 22 2026-02-20T11:16:37.923 INFO:tasks.rados.rados.0.trial043.stdout:236: expect (ObjNum 90 snap 10 seq_num 90) 2026-02-20T11:16:37.923 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:16:37.969 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1405030 2026-02-20T11:16:37.998 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:16:38.039 INFO:tasks.rados.rados.0.trial043.stdout:236: done (1 left) 2026-02-20T11:16:38.039 INFO:tasks.rados.rados.0.trial043.stdout:237: done (0 left) 2026-02-20T11:16:38.040 INFO:tasks.rados.rados.0.trial043.stdout:238: write oid 13 current snap is 22 2026-02-20T11:16:38.040 INFO:tasks.rados.rados.0.trial043.stdout:238: seq_num 139 ranges {736173=553965,1867350=468823} 2026-02-20T11:16:38.044 INFO:tasks.rados.rados.0.trial043.stdout:238: writing trial04342147-13 from 736173 to 1290138 tid 1 2026-02-20T11:16:38.048 INFO:tasks.rados.rados.0.trial043.stdout:238: writing trial04342147-13 from 1867350 to 2336173 tid 2 2026-02-20T11:16:38.048 INFO:tasks.rados.rados.0.trial043.stdout:239: snap_remove snap 20 2026-02-20T11:16:38.879 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:16:38.882 INFO:tasks.rados.rados.0.trial043.stdout:239: done (1 left) 2026-02-20T11:16:38.883 INFO:tasks.rados.rados.0.trial043.stdout:240: read oid 28 snap 21 2026-02-20T11:16:38.883 INFO:tasks.rados.rados.0.trial043.stdout:240: expect deleted 2026-02-20T11:16:38.883 INFO:tasks.rados.rados.0.trial043.stdout:241: snap_create 2026-02-20T11:16:38.883 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:16:38.883 INFO:tasks.rados.rados.0.trial043.stdout:238: finishing write tid 1 to trial04342147-13 2026-02-20T11:16:38.883 INFO:tasks.rados.rados.0.trial043.stdout:238: finishing write tid 2 to trial04342147-13 2026-02-20T11:16:38.883 INFO:tasks.rados.rados.0.trial043.stdout:238: finishing write tid 3 to trial04342147-13 2026-02-20T11:16:38.883 INFO:tasks.rados.rados.0.trial043.stdout:238: finishing write tid 4 to trial04342147-13 2026-02-20T11:16:38.883 INFO:tasks.rados.rados.0.trial043.stdout:238: oid 13 updating version 0 to 89 2026-02-20T11:16:38.883 INFO:tasks.rados.rados.0.trial043.stdout:238: oid 13 version 89 is already newer than 89 2026-02-20T11:16:38.883 INFO:tasks.rados.rados.0.trial043.stdout:238: oid 13 version 89 is already newer than 89 2026-02-20T11:16:38.883 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 13 v 89 (ObjNum 139 snap 22 seq_num 139) dirty exists 2026-02-20T11:16:38.883 INFO:tasks.rados.rados.0.trial043.stdout:238: left oid 13 (ObjNum 139 snap 22 seq_num 139) 2026-02-20T11:16:38.883 INFO:tasks.rados.rados.0.trial043.stdout:238: done (1 left) 2026-02-20T11:16:38.883 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:16:38.901 INFO:tasks.rados.rados.0.trial043.stdout:240: done (0 left) 2026-02-20T11:16:38.955 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:38.954+0000 7f7de5ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:38.958 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:38.958+0000 7f7de5ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:38.983 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:16:38.998 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:16:38.998 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:16:39.888 INFO:tasks.rados.rados.0.trial043.stdout:241: done (0 left) 2026-02-20T11:16:39.888 INFO:tasks.rados.rados.0.trial043.stdout:242: snap_remove snap 22 2026-02-20T11:16:40.371 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~55s 2026-02-20T11:16:40.894 INFO:tasks.rados.rados.0.trial043.stdout:242: done (0 left) 2026-02-20T11:16:40.894 INFO:tasks.rados.rados.0.trial043.stdout:243: snap_create 2026-02-20T11:16:40.999 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:16:41.075 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:41.074+0000 7f52ce0cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:41.077 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:41.078+0000 7f52ce0cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:41.103 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:16:41.120 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:16:41.121 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:16:41.824 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:16:41.893 INFO:tasks.rados.rados.0.trial043.stdout:243: done (0 left) 2026-02-20T11:16:41.893 INFO:tasks.rados.rados.0.trial043.stdout:244: read oid 34 snap -1 2026-02-20T11:16:41.894 INFO:tasks.rados.rados.0.trial043.stdout:244: expect (ObjNum 104 snap 13 seq_num 104) 2026-02-20T11:16:41.894 INFO:tasks.rados.rados.0.trial043.stdout:245: snap_create 2026-02-20T11:16:41.894 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:16:41.897 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:16:41.900 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:16:41.901 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:16:41.930 INFO:tasks.rados.rados.0.trial043.stdout:244: expect (ObjNum 104 snap 13 seq_num 104) 2026-02-20T11:16:41.995 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:16:42.057 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:16:42.119 INFO:tasks.rados.rados.0.trial043.stdout:244: done (0 left) 2026-02-20T11:16:42.900 INFO:tasks.rados.rados.0.trial043.stdout:245: done (0 left) 2026-02-20T11:16:42.900 INFO:tasks.rados.rados.0.trial043.stdout:246: read oid 2 snap -1 2026-02-20T11:16:42.900 INFO:tasks.rados.rados.0.trial043.stdout:246: expect (ObjNum 87 snap 9 seq_num 87) 2026-02-20T11:16:42.900 INFO:tasks.rados.rados.0.trial043.stdout:247: write (excl) oid 8 current snap is 25 2026-02-20T11:16:42.900 INFO:tasks.rados.rados.0.trial043.stdout:247: seq_num 140 ranges {752980=629214,1809479=543501} 2026-02-20T11:16:42.905 INFO:tasks.rados.rados.0.trial043.stdout:247: writing trial04342147-8 from 752980 to 1382194 tid 1 2026-02-20T11:16:42.909 INFO:tasks.rados.rados.0.trial043.stdout:247: writing trial04342147-8 from 1809479 to 2352980 tid 2 2026-02-20T11:16:42.909 INFO:tasks.rados.rados.0.trial043.stdout:248: delete oid 32 current snap is 25 2026-02-20T11:16:42.939 INFO:tasks.rados.rados.0.trial043.stdout:246: expect (ObjNum 87 snap 9 seq_num 87) 2026-02-20T11:16:42.939 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:16:43.000 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2951252 2026-02-20T11:16:43.033 INFO:tasks.rados.rados.0.trial043.stdout:247: finishing write tid 1 to trial04342147-8 2026-02-20T11:16:43.033 INFO:tasks.rados.rados.0.trial043.stdout:247: finishing write tid 2 to trial04342147-8 2026-02-20T11:16:43.033 INFO:tasks.rados.rados.0.trial043.stdout:246: done (2 left) 2026-02-20T11:16:43.033 INFO:tasks.rados.rados.0.trial043.stdout:248: done (1 left) 2026-02-20T11:16:43.033 INFO:tasks.rados.rados.0.trial043.stdout:249: delete oid 10 current snap is 25 2026-02-20T11:16:43.033 INFO:tasks.rados.rados.0.trial043.stdout:247: finishing write tid 3 to trial04342147-8 2026-02-20T11:16:43.033 INFO:tasks.rados.rados.0.trial043.stdout:247: finishing write tid 4 to trial04342147-8 2026-02-20T11:16:43.033 INFO:tasks.rados.rados.0.trial043.stdout:247: oid 8 updating version 0 to 28 2026-02-20T11:16:43.033 INFO:tasks.rados.rados.0.trial043.stdout:247: oid 8 version 28 is already newer than 26 2026-02-20T11:16:43.033 INFO:tasks.rados.rados.0.trial043.stdout:247: oid 8 version 28 is already newer than 28 2026-02-20T11:16:43.034 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 8 v 28 (ObjNum 140 snap 25 seq_num 140) dirty exists 2026-02-20T11:16:43.034 INFO:tasks.rados.rados.0.trial043.stdout:247: left oid 8 (ObjNum 140 snap 25 seq_num 140) 2026-02-20T11:16:43.062 INFO:tasks.rados.rados.0.trial043.stdout:247: done (1 left) 2026-02-20T11:16:43.062 INFO:tasks.rados.rados.0.trial043.stdout:249: done (0 left) 2026-02-20T11:16:43.062 INFO:tasks.rados.rados.0.trial043.stdout:250: write oid 26 current snap is 25 2026-02-20T11:16:43.062 INFO:tasks.rados.rados.0.trial043.stdout:250: seq_num 143 ranges {403401=454961,1584925=506039} 2026-02-20T11:16:43.065 INFO:tasks.rados.rados.0.trial043.stdout:250: writing trial04342147-26 from 403401 to 858362 tid 1 2026-02-20T11:16:43.070 INFO:tasks.rados.rados.0.trial043.stdout:250: writing trial04342147-26 from 1584925 to 2090964 tid 2 2026-02-20T11:16:43.070 INFO:tasks.rados.rados.0.trial043.stdout:251: snap_remove snap 24 2026-02-20T11:16:43.122 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:16:43.197 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:43.197+0000 7f519ee5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:43.200 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:43.200+0000 7f519ee5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:43.225 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:16:43.241 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:16:43.242 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:16:43.741 INFO:tasks.rados.rados.0.trial043.stdout:251: done (1 left) 2026-02-20T11:16:43.741 INFO:tasks.rados.rados.0.trial043.stdout:252: snap_create 2026-02-20T11:16:43.741 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:16:43.741 INFO:tasks.rados.rados.0.trial043.stdout:250: finishing write tid 1 to trial04342147-26 2026-02-20T11:16:43.741 INFO:tasks.rados.rados.0.trial043.stdout:250: finishing write tid 2 to trial04342147-26 2026-02-20T11:16:43.742 INFO:tasks.rados.rados.0.trial043.stdout:250: finishing write tid 3 to trial04342147-26 2026-02-20T11:16:43.742 INFO:tasks.rados.rados.0.trial043.stdout:250: finishing write tid 4 to trial04342147-26 2026-02-20T11:16:43.742 INFO:tasks.rados.rados.0.trial043.stdout:250: oid 26 updating version 0 to 53 2026-02-20T11:16:43.742 INFO:tasks.rados.rados.0.trial043.stdout:250: oid 26 version 53 is already newer than 53 2026-02-20T11:16:43.742 INFO:tasks.rados.rados.0.trial043.stdout:250: oid 26 version 53 is already newer than 53 2026-02-20T11:16:43.742 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 26 v 53 (ObjNum 143 snap 25 seq_num 143) dirty exists 2026-02-20T11:16:43.742 INFO:tasks.rados.rados.0.trial043.stdout:250: left oid 26 (ObjNum 143 snap 25 seq_num 143) 2026-02-20T11:16:43.742 INFO:tasks.rados.rados.0.trial043.stdout:250: done (0 left) 2026-02-20T11:16:44.742 INFO:tasks.rados.rados.0.trial043.stdout:252: done (0 left) 2026-02-20T11:16:44.742 INFO:tasks.rados.rados.0.trial043.stdout:253: write (excl) oid 29 current snap is 26 2026-02-20T11:16:44.742 INFO:tasks.rados.rados.0.trial043.stdout:253: seq_num 144 ranges {420208=530210,1527054=766050} 2026-02-20T11:16:44.746 INFO:tasks.rados.rados.0.trial043.stdout:253: writing trial04342147-29 from 420208 to 950418 tid 1 2026-02-20T11:16:44.751 INFO:tasks.rados.rados.0.trial043.stdout:253: writing trial04342147-29 from 1527054 to 2293104 tid 2 2026-02-20T11:16:44.751 INFO:tasks.rados.rados.0.trial043.stdout:254: snap_remove snap 23 2026-02-20T11:16:45.243 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:16:45.321 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:45.319+0000 7f91e0e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:45.323 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:45.323+0000 7f91e0e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:45.349 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:16:45.366 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:16:45.367 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:16:45.743 INFO:tasks.rados.rados.0.trial043.stdout:254: done (1 left) 2026-02-20T11:16:45.743 INFO:tasks.rados.rados.0.trial043.stdout:255: write (excl) oid 28 current snap is 26 2026-02-20T11:16:45.743 INFO:tasks.rados.rados.0.trial043.stdout:253: finishing write tid 1 to trial04342147-29 2026-02-20T11:16:45.744 INFO:tasks.rados.rados.0.trial043.stdout:255: seq_num 145 ranges {437015=721812,1701889=509708} 2026-02-20T11:16:45.747 INFO:tasks.rados.rados.0.trial043.stdout:255: writing trial04342147-28 from 437015 to 1158827 tid 1 2026-02-20T11:16:45.752 INFO:tasks.rados.rados.0.trial043.stdout:255: writing trial04342147-28 from 1701889 to 2211597 tid 2 2026-02-20T11:16:45.752 INFO:tasks.rados.rados.0.trial043.stdout:256: write oid 19 current snap is 26 2026-02-20T11:16:45.752 INFO:tasks.rados.rados.0.trial043.stdout:253: finishing write tid 2 to trial04342147-29 2026-02-20T11:16:45.752 INFO:tasks.rados.rados.0.trial043.stdout:256: seq_num 146 ranges {453822=797061,1760371=653366} 2026-02-20T11:16:45.757 INFO:tasks.rados.rados.0.trial043.stdout:256: writing trial04342147-19 from 453822 to 1250883 tid 1 2026-02-20T11:16:45.763 INFO:tasks.rados.rados.0.trial043.stdout:256: writing trial04342147-19 from 1760371 to 2413737 tid 2 2026-02-20T11:16:45.763 INFO:tasks.rados.rados.0.trial043.stdout:257: read oid 41 snap 21 2026-02-20T11:16:45.763 INFO:tasks.rados.rados.0.trial043.stdout:257: expect (ObjNum 95 snap 11 seq_num 95) 2026-02-20T11:16:45.763 INFO:tasks.rados.rados.0.trial043.stdout:253: finishing write tid 3 to trial04342147-29 2026-02-20T11:16:45.763 INFO:tasks.rados.rados.0.trial043.stdout:253: finishing write tid 4 to trial04342147-29 2026-02-20T11:16:45.763 INFO:tasks.rados.rados.0.trial043.stdout:253: oid 29 updating version 0 to 44 2026-02-20T11:16:45.763 INFO:tasks.rados.rados.0.trial043.stdout:253: oid 29 version 44 is already newer than 44 2026-02-20T11:16:45.763 INFO:tasks.rados.rados.0.trial043.stdout:253: oid 29 version 44 is already newer than 42 2026-02-20T11:16:45.764 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 29 v 44 (ObjNum 144 snap 26 seq_num 144) dirty exists 2026-02-20T11:16:45.764 INFO:tasks.rados.rados.0.trial043.stdout:253: left oid 29 (ObjNum 144 snap 26 seq_num 144) 2026-02-20T11:16:45.764 INFO:tasks.rados.rados.0.trial043.stdout:258: read oid 39 snap -1 2026-02-20T11:16:45.764 INFO:tasks.rados.rados.0.trial043.stdout:258: expect (ObjNum 71 snap 3 seq_num 71) 2026-02-20T11:16:45.764 INFO:tasks.rados.rados.0.trial043.stdout:253: done (4 left) 2026-02-20T11:16:45.764 INFO:tasks.rados.rados.0.trial043.stdout:259: read oid 46 snap -1 2026-02-20T11:16:45.764 INFO:tasks.rados.rados.0.trial043.stdout:259: expect deleted 2026-02-20T11:16:45.764 INFO:tasks.rados.rados.0.trial043.stdout:260: write (excl) oid 48 current snap is 26 2026-02-20T11:16:45.764 INFO:tasks.rados.rados.0.trial043.stdout:260: seq_num 147 ranges {470629=472310,1418853=513377} 2026-02-20T11:16:45.765 INFO:tasks.rados.rados.0.trial043.stdout:260: writing trial04342147-48 from 470629 to 942939 tid 1 2026-02-20T11:16:45.770 INFO:tasks.rados.rados.0.trial043.stdout:260: writing trial04342147-48 from 1418853 to 1932230 tid 2 2026-02-20T11:16:45.770 INFO:tasks.rados.rados.0.trial043.stdout:261: snap_create 2026-02-20T11:16:45.770 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 6 2026-02-20T11:16:45.771 INFO:tasks.rados.rados.0.trial043.stdout:259: done (5 left) 2026-02-20T11:16:45.771 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 5 2026-02-20T11:16:45.775 INFO:tasks.rados.rados.0.trial043.stdout:255: finishing write tid 1 to trial04342147-28 2026-02-20T11:16:45.785 INFO:tasks.rados.rados.0.trial043.stdout:257: expect (ObjNum 95 snap 11 seq_num 95) 2026-02-20T11:16:45.817 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:16:45.847 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 19491 2026-02-20T11:16:45.875 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~61s 2026-02-20T11:16:45.876 INFO:tasks.rados.rados.0.trial043.stdout:258: expect (ObjNum 71 snap 3 seq_num 71) 2026-02-20T11:16:45.887 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2084398 2026-02-20T11:16:45.951 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2106181 2026-02-20T11:16:45.968 INFO:tasks.rados.rados.0.trial043.stdout:260: finishing write tid 1 to trial04342147-48 2026-02-20T11:16:45.968 INFO:tasks.rados.rados.0.trial043.stdout:257: done (4 left) 2026-02-20T11:16:45.968 INFO:tasks.rados.rados.0.trial043.stdout:258: done (3 left) 2026-02-20T11:16:45.969 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:16:45.969 INFO:tasks.rados.rados.0.trial043.stdout:255: finishing write tid 2 to trial04342147-28 2026-02-20T11:16:45.969 INFO:tasks.rados.rados.0.trial043.stdout:255: finishing write tid 3 to trial04342147-28 2026-02-20T11:16:45.969 INFO:tasks.rados.rados.0.trial043.stdout:255: finishing write tid 4 to trial04342147-28 2026-02-20T11:16:45.969 INFO:tasks.rados.rados.0.trial043.stdout:255: oid 28 updating version 0 to 33 2026-02-20T11:16:45.969 INFO:tasks.rados.rados.0.trial043.stdout:255: oid 28 version 33 is already newer than 33 2026-02-20T11:16:45.969 INFO:tasks.rados.rados.0.trial043.stdout:255: oid 28 version 33 is already newer than 31 2026-02-20T11:16:45.969 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 28 v 33 (ObjNum 145 snap 26 seq_num 145) dirty exists 2026-02-20T11:16:45.969 INFO:tasks.rados.rados.0.trial043.stdout:255: left oid 28 (ObjNum 145 snap 26 seq_num 145) 2026-02-20T11:16:45.969 INFO:tasks.rados.rados.0.trial043.stdout:256: finishing write tid 1 to trial04342147-19 2026-02-20T11:16:45.969 INFO:tasks.rados.rados.0.trial043.stdout:260: finishing write tid 2 to trial04342147-48 2026-02-20T11:16:45.969 INFO:tasks.rados.rados.0.trial043.stdout:260: finishing write tid 3 to trial04342147-48 2026-02-20T11:16:45.969 INFO:tasks.rados.rados.0.trial043.stdout:260: finishing write tid 4 to trial04342147-48 2026-02-20T11:16:45.969 INFO:tasks.rados.rados.0.trial043.stdout:260: oid 48 updating version 0 to 28 2026-02-20T11:16:45.969 INFO:tasks.rados.rados.0.trial043.stdout:260: oid 48 updating version 28 to 30 2026-02-20T11:16:45.969 INFO:tasks.rados.rados.0.trial043.stdout:260: oid 48 version 30 is already newer than 30 2026-02-20T11:16:45.969 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 48 v 30 (ObjNum 147 snap 26 seq_num 147) dirty exists 2026-02-20T11:16:45.969 INFO:tasks.rados.rados.0.trial043.stdout:260: left oid 48 (ObjNum 147 snap 26 seq_num 147) 2026-02-20T11:16:45.969 INFO:tasks.rados.rados.0.trial043.stdout:256: finishing write tid 2 to trial04342147-19 2026-02-20T11:16:45.969 INFO:tasks.rados.rados.0.trial043.stdout:256: finishing write tid 3 to trial04342147-19 2026-02-20T11:16:45.969 INFO:tasks.rados.rados.0.trial043.stdout:256: finishing write tid 4 to trial04342147-19 2026-02-20T11:16:45.969 INFO:tasks.rados.rados.0.trial043.stdout:256: oid 19 updating version 0 to 19 2026-02-20T11:16:45.969 INFO:tasks.rados.rados.0.trial043.stdout:256: oid 19 version 19 is already newer than 19 2026-02-20T11:16:45.969 INFO:tasks.rados.rados.0.trial043.stdout:256: oid 19 version 19 is already newer than 19 2026-02-20T11:16:45.969 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 19 v 19 (ObjNum 146 snap 26 seq_num 146) dirty exists 2026-02-20T11:16:45.969 INFO:tasks.rados.rados.0.trial043.stdout:256: left oid 19 (ObjNum 146 snap 26 seq_num 146) 2026-02-20T11:16:45.969 INFO:tasks.rados.rados.0.trial043.stdout:255: done (2 left) 2026-02-20T11:16:45.969 INFO:tasks.rados.rados.0.trial043.stdout:256: done (1 left) 2026-02-20T11:16:45.969 INFO:tasks.rados.rados.0.trial043.stdout:260: done (0 left) 2026-02-20T11:16:46.747 INFO:tasks.rados.rados.0.trial043.stdout:261: done (0 left) 2026-02-20T11:16:46.748 INFO:tasks.rados.rados.0.trial043.stdout:262: write oid 41 current snap is 27 2026-02-20T11:16:46.748 INFO:tasks.rados.rados.0.trial043.stdout:262: seq_num 148 ranges {487436=547559,1760982=657035} 2026-02-20T11:16:46.752 INFO:tasks.rados.rados.0.trial043.stdout:262: writing trial04342147-41 from 487436 to 1034995 tid 1 2026-02-20T11:16:46.758 INFO:tasks.rados.rados.0.trial043.stdout:262: writing trial04342147-41 from 1760982 to 2418017 tid 2 2026-02-20T11:16:46.758 INFO:tasks.rados.rados.0.trial043.stdout:263: snap_remove snap 26 2026-02-20T11:16:46.902 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:16:46.972 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:16:46.975 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:16:46.976 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:16:47.367 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:16:47.445 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:47.444+0000 7fb59d85a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:47.447 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:47.448+0000 7fb59d85a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:47.474 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:16:47.491 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:16:47.492 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:16:47.753 INFO:tasks.rados.rados.0.trial043.stdout:263: done (1 left) 2026-02-20T11:16:47.753 INFO:tasks.rados.rados.0.trial043.stdout:264: snap_create 2026-02-20T11:16:47.753 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:16:47.753 INFO:tasks.rados.rados.0.trial043.stdout:262: finishing write tid 1 to trial04342147-41 2026-02-20T11:16:47.753 INFO:tasks.rados.rados.0.trial043.stdout:262: finishing write tid 2 to trial04342147-41 2026-02-20T11:16:47.753 INFO:tasks.rados.rados.0.trial043.stdout:262: finishing write tid 3 to trial04342147-41 2026-02-20T11:16:47.753 INFO:tasks.rados.rados.0.trial043.stdout:262: finishing write tid 4 to trial04342147-41 2026-02-20T11:16:47.753 INFO:tasks.rados.rados.0.trial043.stdout:262: oid 41 updating version 0 to 44 2026-02-20T11:16:47.753 INFO:tasks.rados.rados.0.trial043.stdout:262: oid 41 version 44 is already newer than 44 2026-02-20T11:16:47.754 INFO:tasks.rados.rados.0.trial043.stdout:262: oid 41 version 44 is already newer than 44 2026-02-20T11:16:47.754 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 41 v 44 (ObjNum 148 snap 27 seq_num 148) dirty exists 2026-02-20T11:16:47.754 INFO:tasks.rados.rados.0.trial043.stdout:262: left oid 41 (ObjNum 148 snap 27 seq_num 148) 2026-02-20T11:16:47.754 INFO:tasks.rados.rados.0.trial043.stdout:262: done (0 left) 2026-02-20T11:16:48.743 INFO:tasks.rados.rados.0.trial043.stdout:264: done (0 left) 2026-02-20T11:16:48.743 INFO:tasks.rados.rados.0.trial043.stdout:265: delete oid 33 current snap is 28 2026-02-20T11:16:48.779 INFO:tasks.rados.rados.0.trial043.stdout:265: done (0 left) 2026-02-20T11:16:48.779 INFO:tasks.rados.rados.0.trial043.stdout:266: snap_create 2026-02-20T11:16:49.493 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:16:49.570 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:49.569+0000 7f346e993640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:49.572 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:49.572+0000 7f346e993640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:49.598 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:16:49.611 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:16:49.612 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:16:49.760 INFO:tasks.rados.rados.0.trial043.stdout:266: done (0 left) 2026-02-20T11:16:49.760 INFO:tasks.rados.rados.0.trial043.stdout:267: write (excl) oid 13 current snap is 29 2026-02-20T11:16:49.760 INFO:tasks.rados.rados.0.trial043.stdout:267: seq_num 150 ranges {521050=698057,1877946=643104} 2026-02-20T11:16:49.765 INFO:tasks.rados.rados.0.trial043.stdout:267: writing trial04342147-13 from 521050 to 1219107 tid 1 2026-02-20T11:16:49.770 INFO:tasks.rados.rados.0.trial043.stdout:267: writing trial04342147-13 from 1877946 to 2521050 tid 2 2026-02-20T11:16:49.770 INFO:tasks.rados.rados.0.trial043.stdout:268: snap_create 2026-02-20T11:16:49.770 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:16:49.817 INFO:tasks.rados.rados.0.trial043.stdout:267: finishing write tid 1 to trial04342147-13 2026-02-20T11:16:49.833 INFO:tasks.rados.rados.0.trial043.stdout:267: finishing write tid 2 to trial04342147-13 2026-02-20T11:16:49.837 INFO:tasks.rados.rados.0.trial043.stdout:267: finishing write tid 3 to trial04342147-13 2026-02-20T11:16:49.838 INFO:tasks.rados.rados.0.trial043.stdout:267: finishing write tid 4 to trial04342147-13 2026-02-20T11:16:49.839 INFO:tasks.rados.rados.0.trial043.stdout:267: oid 13 updating version 0 to 93 2026-02-20T11:16:49.839 INFO:tasks.rados.rados.0.trial043.stdout:267: oid 13 version 93 is already newer than 91 2026-02-20T11:16:49.839 INFO:tasks.rados.rados.0.trial043.stdout:267: oid 13 version 93 is already newer than 93 2026-02-20T11:16:49.839 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 13 v 93 (ObjNum 150 snap 29 seq_num 150) dirty exists 2026-02-20T11:16:49.839 INFO:tasks.rados.rados.0.trial043.stdout:267: left oid 13 (ObjNum 150 snap 29 seq_num 150) 2026-02-20T11:16:49.839 INFO:tasks.rados.rados.0.trial043.stdout:267: done (0 left) 2026-02-20T11:16:50.764 INFO:tasks.rados.rados.0.trial043.stdout:268: done (0 left) 2026-02-20T11:16:50.764 INFO:tasks.rados.rados.0.trial043.stdout:269: read oid 49 snap -1 2026-02-20T11:16:50.765 INFO:tasks.rados.rados.0.trial043.stdout:269: expect (ObjNum 109 snap 13 seq_num 109) 2026-02-20T11:16:50.765 INFO:tasks.rados.rados.0.trial043.stdout:270: snap_create 2026-02-20T11:16:50.765 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:16:50.795 INFO:tasks.rados.rados.0.trial043.stdout:269: expect (ObjNum 109 snap 13 seq_num 109) 2026-02-20T11:16:50.795 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:16:50.888 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 449399 2026-02-20T11:16:50.929 INFO:tasks.rados.rados.0.trial043.stdout:269: done (0 left) 2026-02-20T11:16:51.378 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~66s 2026-02-20T11:16:51.612 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:16:51.687 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:51.686+0000 7f8bce2cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:51.689 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:51.689+0000 7f8bce2cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:51.715 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:16:51.731 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:16:51.732 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:16:51.768 INFO:tasks.rados.rados.0.trial043.stdout:270: done (0 left) 2026-02-20T11:16:51.768 INFO:tasks.rados.rados.0.trial043.stdout:271: read oid 20 snap -1 2026-02-20T11:16:51.768 INFO:tasks.rados.rados.0.trial043.stdout:271: expect (ObjNum 128 snap 17 seq_num 128) 2026-02-20T11:16:51.768 INFO:tasks.rados.rados.0.trial043.stdout:272: snap_remove snap 30 2026-02-20T11:16:51.977 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:16:52.048 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:16:52.051 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:16:52.051 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:16:52.771 INFO:tasks.rados.rados.0.trial043.stdout:272: done (1 left) 2026-02-20T11:16:52.771 INFO:tasks.rados.rados.0.trial043.stdout:273: snap_remove snap 29 2026-02-20T11:16:53.733 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:16:53.775 INFO:tasks.rados.rados.0.trial043.stdout:273: done (1 left) 2026-02-20T11:16:53.776 INFO:tasks.rados.rados.0.trial043.stdout:274: delete oid 21 current snap is 31 2026-02-20T11:16:53.776 INFO:tasks.rados.rados.0.trial043.stdout:271: expect (ObjNum 128 snap 17 seq_num 128) 2026-02-20T11:16:53.776 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:16:53.814 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:53.813+0000 7f067205a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:53.816 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:53.816+0000 7f067205a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:53.828 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 542470 2026-02-20T11:16:53.841 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:16:53.849 INFO:tasks.rados.rados.0.trial043.stdout:271: done (1 left) 2026-02-20T11:16:53.849 INFO:tasks.rados.rados.0.trial043.stdout:274: done (0 left) 2026-02-20T11:16:53.849 INFO:tasks.rados.rados.0.trial043.stdout:275: write (excl) oid 24 current snap is 31 2026-02-20T11:16:53.849 INFO:tasks.rados.rados.0.trial043.stdout:275: seq_num 152 ranges {554664=448555,1478557=664373} 2026-02-20T11:16:53.852 INFO:tasks.rados.rados.0.trial043.stdout:275: writing trial04342147-24 from 554664 to 1003219 tid 1 2026-02-20T11:16:53.857 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:16:53.858 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:16:53.858 INFO:tasks.rados.rados.0.trial043.stdout:275: writing trial04342147-24 from 1478557 to 2142930 tid 2 2026-02-20T11:16:53.858 INFO:tasks.rados.rados.0.trial043.stdout:276: read oid 7 snap -1 2026-02-20T11:16:53.859 INFO:tasks.rados.rados.0.trial043.stdout:276: expect (ObjNum 63 snap 3 seq_num 63) 2026-02-20T11:16:53.859 INFO:tasks.rados.rados.0.trial043.stdout:277: read oid 45 snap -1 2026-02-20T11:16:53.859 INFO:tasks.rados.rados.0.trial043.stdout:277: expect (ObjNum 126 snap 17 seq_num 126) 2026-02-20T11:16:53.859 INFO:tasks.rados.rados.0.trial043.stdout:278: snap_remove snap 21 2026-02-20T11:16:54.778 INFO:tasks.rados.rados.0.trial043.stdout:278: done (3 left) 2026-02-20T11:16:54.778 INFO:tasks.rados.rados.0.trial043.stdout:279: delete oid 36 current snap is 31 2026-02-20T11:16:54.778 INFO:tasks.rados.rados.0.trial043.stdout:276: expect (ObjNum 63 snap 3 seq_num 63) 2026-02-20T11:16:54.778 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:16:54.812 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 360257 2026-02-20T11:16:54.853 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1777147 2026-02-20T11:16:54.870 INFO:tasks.rados.rados.0.trial043.stdout:277: expect (ObjNum 126 snap 17 seq_num 126) 2026-02-20T11:16:54.895 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 662693 2026-02-20T11:16:54.925 INFO:tasks.rados.rados.0.trial043.stdout:275: finishing write tid 1 to trial04342147-24 2026-02-20T11:16:54.925 INFO:tasks.rados.rados.0.trial043.stdout:275: finishing write tid 2 to trial04342147-24 2026-02-20T11:16:54.925 INFO:tasks.rados.rados.0.trial043.stdout:275: finishing write tid 3 to trial04342147-24 2026-02-20T11:16:54.926 INFO:tasks.rados.rados.0.trial043.stdout:276: done (3 left) 2026-02-20T11:16:54.926 INFO:tasks.rados.rados.0.trial043.stdout:277: done (2 left) 2026-02-20T11:16:54.926 INFO:tasks.rados.rados.0.trial043.stdout:279: done (1 left) 2026-02-20T11:16:54.926 INFO:tasks.rados.rados.0.trial043.stdout:280: snap_create 2026-02-20T11:16:54.926 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:16:54.926 INFO:tasks.rados.rados.0.trial043.stdout:275: finishing write tid 4 to trial04342147-24 2026-02-20T11:16:54.926 INFO:tasks.rados.rados.0.trial043.stdout:275: oid 24 updating version 0 to 20 2026-02-20T11:16:54.926 INFO:tasks.rados.rados.0.trial043.stdout:275: oid 24 version 20 is already newer than 18 2026-02-20T11:16:54.926 INFO:tasks.rados.rados.0.trial043.stdout:275: oid 24 version 20 is already newer than 20 2026-02-20T11:16:54.926 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 24 v 20 (ObjNum 152 snap 31 seq_num 152) dirty exists 2026-02-20T11:16:54.926 INFO:tasks.rados.rados.0.trial043.stdout:275: left oid 24 (ObjNum 152 snap 31 seq_num 152) 2026-02-20T11:16:54.926 INFO:tasks.rados.rados.0.trial043.stdout:275: done (0 left) 2026-02-20T11:16:55.781 INFO:tasks.rados.rados.0.trial043.stdout:280: done (0 left) 2026-02-20T11:16:55.781 INFO:tasks.rados.rados.0.trial043.stdout:281: snap_create 2026-02-20T11:16:55.859 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:16:55.933 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:55.933+0000 7f30d3acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:55.936 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:55.936+0000 7f30d3acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:55.961 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:16:55.978 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:16:55.978 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:16:56.784 INFO:tasks.rados.rados.0.trial043.stdout:281: done (0 left) 2026-02-20T11:16:56.784 INFO:tasks.rados.rados.0.trial043.stdout:282: snap_remove snap 32 2026-02-20T11:16:56.882 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~72s 2026-02-20T11:16:57.052 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:16:57.123 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:16:57.126 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:16:57.126 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:16:57.787 INFO:tasks.rados.rados.0.trial043.stdout:282: done (0 left) 2026-02-20T11:16:57.787 INFO:tasks.rados.rados.0.trial043.stdout:283: snap_create 2026-02-20T11:16:57.979 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:16:58.054 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:58.054+0000 7f82a205a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:58.057 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:16:58.057+0000 7f82a205a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:16:58.082 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:16:58.099 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:16:58.099 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:16:58.744 INFO:tasks.rados.rados.0.trial043.stdout:283: done (0 left) 2026-02-20T11:16:58.744 INFO:tasks.rados.rados.0.trial043.stdout:284: write (excl) oid 4 current snap is 34 2026-02-20T11:16:58.744 INFO:tasks.rados.rados.0.trial043.stdout:284: seq_num 154 ranges {588278=715406,1711874=668042} 2026-02-20T11:16:58.749 INFO:tasks.rados.rados.0.trial043.stdout:284: writing trial04342147-4 from 588278 to 1303684 tid 1 2026-02-20T11:16:58.754 INFO:tasks.rados.rados.0.trial043.stdout:284: writing trial04342147-4 from 1711874 to 2379916 tid 2 2026-02-20T11:16:58.754 INFO:tasks.rados.rados.0.trial043.stdout:285: write (excl) oid 22 current snap is 34 2026-02-20T11:16:58.754 INFO:tasks.rados.rados.0.trial043.stdout:285: seq_num 155 ranges {605085=790655,2170356=434729} 2026-02-20T11:16:58.759 INFO:tasks.rados.rados.0.trial043.stdout:285: writing trial04342147-22 from 605085 to 1395740 tid 1 2026-02-20T11:16:58.764 INFO:tasks.rados.rados.0.trial043.stdout:285: writing trial04342147-22 from 2170356 to 2605085 tid 2 2026-02-20T11:16:58.764 INFO:tasks.rados.rados.0.trial043.stdout:286: write oid 34 current snap is 34 2026-02-20T11:16:58.764 INFO:tasks.rados.rados.0.trial043.stdout:286: seq_num 156 ranges {621892=465904,1712485=671711} 2026-02-20T11:16:58.768 INFO:tasks.rados.rados.0.trial043.stdout:286: writing trial04342147-34 from 621892 to 1087796 tid 1 2026-02-20T11:16:58.773 INFO:tasks.rados.rados.0.trial043.stdout:286: writing trial04342147-34 from 1712485 to 2384196 tid 2 2026-02-20T11:16:58.773 INFO:tasks.rados.rados.0.trial043.stdout:287: write oid 15 current snap is 34 2026-02-20T11:16:58.773 INFO:tasks.rados.rados.0.trial043.stdout:287: seq_num 157 ranges {638699=541153,1770967=531722} 2026-02-20T11:16:58.778 INFO:tasks.rados.rados.0.trial043.stdout:287: writing trial04342147-15 from 638699 to 1179852 tid 1 2026-02-20T11:16:58.782 INFO:tasks.rados.rados.0.trial043.stdout:287: writing trial04342147-15 from 1770967 to 2302689 tid 2 2026-02-20T11:16:58.782 INFO:tasks.rados.rados.0.trial043.stdout:288: read oid 10 snap 33 2026-02-20T11:16:58.783 INFO:tasks.rados.rados.0.trial043.stdout:288: expect deleted 2026-02-20T11:16:58.783 INFO:tasks.rados.rados.0.trial043.stdout:289: delete oid 11 current snap is 34 2026-02-20T11:16:58.791 INFO:tasks.rados.rados.0.trial043.stdout:284: finishing write tid 1 to trial04342147-4 2026-02-20T11:16:58.809 INFO:tasks.rados.rados.0.trial043.stdout:286: finishing write tid 1 to trial04342147-34 2026-02-20T11:16:58.819 INFO:tasks.rados.rados.0.trial043.stdout:287: finishing write tid 1 to trial04342147-15 2026-02-20T11:16:58.821 INFO:tasks.rados.rados.0.trial043.stdout:286: finishing write tid 2 to trial04342147-34 2026-02-20T11:16:58.833 INFO:tasks.rados.rados.0.trial043.stdout:286: finishing write tid 3 to trial04342147-34 2026-02-20T11:16:58.835 INFO:tasks.rados.rados.0.trial043.stdout:286: finishing write tid 4 to trial04342147-34 2026-02-20T11:16:58.835 INFO:tasks.rados.rados.0.trial043.stdout:286: oid 34 updating version 0 to 28 2026-02-20T11:16:58.835 INFO:tasks.rados.rados.0.trial043.stdout:286: oid 34 version 28 is already newer than 28 2026-02-20T11:16:58.835 INFO:tasks.rados.rados.0.trial043.stdout:286: oid 34 version 28 is already newer than 28 2026-02-20T11:16:58.835 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 34 v 28 (ObjNum 156 snap 34 seq_num 156) dirty exists 2026-02-20T11:16:58.836 INFO:tasks.rados.rados.0.trial043.stdout:286: left oid 34 (ObjNum 156 snap 34 seq_num 156) 2026-02-20T11:16:58.836 INFO:tasks.rados.rados.0.trial043.stdout:287: finishing write tid 2 to trial04342147-15 2026-02-20T11:16:58.847 INFO:tasks.rados.rados.0.trial043.stdout:287: finishing write tid 3 to trial04342147-15 2026-02-20T11:16:58.849 INFO:tasks.rados.rados.0.trial043.stdout:287: finishing write tid 4 to trial04342147-15 2026-02-20T11:16:58.849 INFO:tasks.rados.rados.0.trial043.stdout:287: oid 15 updating version 0 to 23 2026-02-20T11:16:58.849 INFO:tasks.rados.rados.0.trial043.stdout:287: oid 15 version 23 is already newer than 23 2026-02-20T11:16:58.849 INFO:tasks.rados.rados.0.trial043.stdout:287: oid 15 version 23 is already newer than 23 2026-02-20T11:16:58.850 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 15 v 23 (ObjNum 157 snap 34 seq_num 157) dirty exists 2026-02-20T11:16:58.850 INFO:tasks.rados.rados.0.trial043.stdout:287: left oid 15 (ObjNum 157 snap 34 seq_num 157) 2026-02-20T11:16:58.856 INFO:tasks.rados.rados.0.trial043.stdout:284: finishing write tid 2 to trial04342147-4 2026-02-20T11:16:58.867 INFO:tasks.rados.rados.0.trial043.stdout:284: finishing write tid 3 to trial04342147-4 2026-02-20T11:16:58.871 INFO:tasks.rados.rados.0.trial043.stdout:284: finishing write tid 4 to trial04342147-4 2026-02-20T11:16:58.871 INFO:tasks.rados.rados.0.trial043.stdout:284: oid 4 updating version 0 to 32 2026-02-20T11:16:58.871 INFO:tasks.rados.rados.0.trial043.stdout:284: oid 4 version 32 is already newer than 30 2026-02-20T11:16:58.871 INFO:tasks.rados.rados.0.trial043.stdout:284: oid 4 version 32 is already newer than 32 2026-02-20T11:16:58.871 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 4 v 32 (ObjNum 154 snap 34 seq_num 154) dirty exists 2026-02-20T11:16:58.871 INFO:tasks.rados.rados.0.trial043.stdout:284: left oid 4 (ObjNum 154 snap 34 seq_num 154) 2026-02-20T11:16:58.892 INFO:tasks.rados.rados.0.trial043.stdout:285: finishing write tid 1 to trial04342147-22 2026-02-20T11:16:58.907 INFO:tasks.rados.rados.0.trial043.stdout:285: finishing write tid 2 to trial04342147-22 2026-02-20T11:16:58.923 INFO:tasks.rados.rados.0.trial043.stdout:285: finishing write tid 3 to trial04342147-22 2026-02-20T11:16:58.926 INFO:tasks.rados.rados.0.trial043.stdout:285: finishing write tid 4 to trial04342147-22 2026-02-20T11:16:58.926 INFO:tasks.rados.rados.0.trial043.stdout:285: oid 22 updating version 0 to 36 2026-02-20T11:16:58.926 INFO:tasks.rados.rados.0.trial043.stdout:285: oid 22 version 36 is already newer than 36 2026-02-20T11:16:58.927 INFO:tasks.rados.rados.0.trial043.stdout:285: oid 22 version 36 is already newer than 34 2026-02-20T11:16:58.927 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 22 v 36 (ObjNum 155 snap 34 seq_num 155) dirty exists 2026-02-20T11:16:58.927 INFO:tasks.rados.rados.0.trial043.stdout:285: left oid 22 (ObjNum 155 snap 34 seq_num 155) 2026-02-20T11:16:58.939 INFO:tasks.rados.rados.0.trial043.stdout:284: done (5 left) 2026-02-20T11:16:58.940 INFO:tasks.rados.rados.0.trial043.stdout:285: done (4 left) 2026-02-20T11:16:58.940 INFO:tasks.rados.rados.0.trial043.stdout:286: done (3 left) 2026-02-20T11:16:58.940 INFO:tasks.rados.rados.0.trial043.stdout:287: done (2 left) 2026-02-20T11:16:58.940 INFO:tasks.rados.rados.0.trial043.stdout:288: done (1 left) 2026-02-20T11:16:58.940 INFO:tasks.rados.rados.0.trial043.stdout:289: done (0 left) 2026-02-20T11:16:58.941 INFO:tasks.rados.rados.0.trial043.stdout:290: delete oid 5 current snap is 34 2026-02-20T11:16:58.963 INFO:tasks.rados.rados.0.trial043.stdout:290: done (0 left) 2026-02-20T11:16:58.964 INFO:tasks.rados.rados.0.trial043.stdout:291: write (excl) oid 22 current snap is 34 2026-02-20T11:16:58.964 INFO:tasks.rados.rados.0.trial043.stdout:291: seq_num 160 ranges {689120=483253,1946413=679049} 2026-02-20T11:16:58.967 INFO:tasks.rados.rados.0.trial043.stdout:291: writing trial04342147-22 from 689120 to 1172373 tid 1 2026-02-20T11:16:58.973 INFO:tasks.rados.rados.0.trial043.stdout:291: writing trial04342147-22 from 1946413 to 2625462 tid 2 2026-02-20T11:16:58.973 INFO:tasks.rados.rados.0.trial043.stdout:292: snap_create 2026-02-20T11:16:58.973 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:16:58.982 INFO:tasks.rados.rados.0.trial043.stdout:291: finishing write tid 1 to trial04342147-22 2026-02-20T11:16:58.996 INFO:tasks.rados.rados.0.trial043.stdout:291: finishing write tid 2 to trial04342147-22 2026-02-20T11:16:59.001 INFO:tasks.rados.rados.0.trial043.stdout:291: finishing write tid 3 to trial04342147-22 2026-02-20T11:16:59.004 INFO:tasks.rados.rados.0.trial043.stdout:291: finishing write tid 4 to trial04342147-22 2026-02-20T11:16:59.004 INFO:tasks.rados.rados.0.trial043.stdout:291: oid 22 updating version 0 to 37 2026-02-20T11:16:59.004 INFO:tasks.rados.rados.0.trial043.stdout:291: oid 22 updating version 37 to 39 2026-02-20T11:16:59.004 INFO:tasks.rados.rados.0.trial043.stdout:291: oid 22 version 39 is already newer than 39 2026-02-20T11:16:59.004 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 22 v 39 (ObjNum 160 snap 34 seq_num 160) dirty exists 2026-02-20T11:16:59.004 INFO:tasks.rados.rados.0.trial043.stdout:291: left oid 22 (ObjNum 160 snap 34 seq_num 160) 2026-02-20T11:16:59.004 INFO:tasks.rados.rados.0.trial043.stdout:291: done (0 left) 2026-02-20T11:16:59.795 INFO:tasks.rados.rados.0.trial043.stdout:292: done (0 left) 2026-02-20T11:16:59.795 INFO:tasks.rados.rados.0.trial043.stdout:293: write oid 33 current snap is 35 2026-02-20T11:16:59.795 INFO:tasks.rados.rados.0.trial043.stdout:293: seq_num 161 ranges {705927=558502,2004895=422707} 2026-02-20T11:16:59.799 INFO:tasks.rados.rados.0.trial043.stdout:293: writing trial04342147-33 from 705927 to 1264429 tid 1 2026-02-20T11:16:59.804 INFO:tasks.rados.rados.0.trial043.stdout:293: writing trial04342147-33 from 2004895 to 2427602 tid 2 2026-02-20T11:16:59.804 INFO:tasks.rados.rados.0.trial043.stdout:294: read oid 5 snap -1 2026-02-20T11:16:59.804 INFO:tasks.rados.rados.0.trial043.stdout:294: expect deleted 2026-02-20T11:16:59.805 INFO:tasks.rados.rados.0.trial043.stdout:295: snap_create 2026-02-20T11:16:59.805 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:16:59.812 INFO:tasks.rados.rados.0.trial043.stdout:294: done (1 left) 2026-02-20T11:16:59.812 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:16:59.825 INFO:tasks.rados.rados.0.trial043.stdout:293: finishing write tid 1 to trial04342147-33 2026-02-20T11:16:59.831 INFO:tasks.rados.rados.0.trial043.stdout:293: finishing write tid 2 to trial04342147-33 2026-02-20T11:16:59.836 INFO:tasks.rados.rados.0.trial043.stdout:293: finishing write tid 3 to trial04342147-33 2026-02-20T11:16:59.837 INFO:tasks.rados.rados.0.trial043.stdout:293: finishing write tid 4 to trial04342147-33 2026-02-20T11:16:59.837 INFO:tasks.rados.rados.0.trial043.stdout:293: oid 33 updating version 0 to 57 2026-02-20T11:16:59.838 INFO:tasks.rados.rados.0.trial043.stdout:293: oid 33 version 57 is already newer than 57 2026-02-20T11:16:59.838 INFO:tasks.rados.rados.0.trial043.stdout:293: oid 33 version 57 is already newer than 57 2026-02-20T11:16:59.838 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 33 v 57 (ObjNum 161 snap 35 seq_num 161) dirty exists 2026-02-20T11:16:59.838 INFO:tasks.rados.rados.0.trial043.stdout:293: left oid 33 (ObjNum 161 snap 35 seq_num 161) 2026-02-20T11:16:59.838 INFO:tasks.rados.rados.0.trial043.stdout:293: done (0 left) 2026-02-20T11:17:00.100 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:17:00.174 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:00.173+0000 7f2971eb7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:00.176 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:00.176+0000 7f2971eb7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:00.203 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:17:00.219 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:17:00.220 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:17:00.797 INFO:tasks.rados.rados.0.trial043.stdout:295: done (0 left) 2026-02-20T11:17:00.797 INFO:tasks.rados.rados.0.trial043.stdout:296: write oid 38 current snap is 36 2026-02-20T11:17:00.797 INFO:tasks.rados.rados.0.trial043.stdout:296: seq_num 162 ranges {722734=633751,2063377=659357} 2026-02-20T11:17:00.802 INFO:tasks.rados.rados.0.trial043.stdout:296: writing trial04342147-38 from 722734 to 1356485 tid 1 2026-02-20T11:17:00.809 INFO:tasks.rados.rados.0.trial043.stdout:296: writing trial04342147-38 from 2063377 to 2722734 tid 2 2026-02-20T11:17:00.810 INFO:tasks.rados.rados.0.trial043.stdout:297: read oid 40 snap -1 2026-02-20T11:17:00.810 INFO:tasks.rados.rados.0.trial043.stdout:297: expect (ObjNum 103 snap 13 seq_num 103) 2026-02-20T11:17:00.810 INFO:tasks.rados.rados.0.trial043.stdout:298: read oid 13 snap 33 2026-02-20T11:17:00.810 INFO:tasks.rados.rados.0.trial043.stdout:298: expect (ObjNum 150 snap 29 seq_num 150) 2026-02-20T11:17:00.810 INFO:tasks.rados.rados.0.trial043.stdout:299: read oid 30 snap -1 2026-02-20T11:17:00.810 INFO:tasks.rados.rados.0.trial043.stdout:299: expect deleted 2026-02-20T11:17:00.810 INFO:tasks.rados.rados.0.trial043.stdout:300: delete oid 27 current snap is 36 2026-02-20T11:17:00.828 INFO:tasks.rados.rados.0.trial043.stdout:299: done (4 left) 2026-02-20T11:17:00.828 INFO:tasks.rados.rados.0.trial043.stdout:300: done (3 left) 2026-02-20T11:17:00.829 INFO:tasks.rados.rados.0.trial043.stdout:301: read oid 28 snap -1 2026-02-20T11:17:00.829 INFO:tasks.rados.rados.0.trial043.stdout:301: expect (ObjNum 145 snap 26 seq_num 145) 2026-02-20T11:17:00.829 INFO:tasks.rados.rados.0.trial043.stdout:302: snap_create 2026-02-20T11:17:00.829 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 4 2026-02-20T11:17:00.839 INFO:tasks.rados.rados.0.trial043.stdout:298: expect (ObjNum 150 snap 29 seq_num 150) 2026-02-20T11:17:00.918 INFO:tasks.rados.rados.0.trial043.stdout:296: finishing write tid 1 to trial04342147-38 2026-02-20T11:17:00.918 INFO:tasks.rados.rados.0.trial043.stdout:301: expect (ObjNum 145 snap 26 seq_num 145) 2026-02-20T11:17:00.949 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:17:00.970 INFO:tasks.rados.rados.0.trial043.stdout:297: expect (ObjNum 103 snap 13 seq_num 103) 2026-02-20T11:17:00.970 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 20754 2026-02-20T11:17:01.100 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2424407 2026-02-20T11:17:01.127 INFO:tasks.rados.rados.0.trial043.stdout:296: finishing write tid 2 to trial04342147-38 2026-02-20T11:17:01.127 INFO:tasks.rados.rados.0.trial043.stdout:296: finishing write tid 3 to trial04342147-38 2026-02-20T11:17:01.127 INFO:tasks.rados.rados.0.trial043.stdout:297: done (3 left) 2026-02-20T11:17:01.127 INFO:tasks.rados.rados.0.trial043.stdout:298: done (2 left) 2026-02-20T11:17:01.127 INFO:tasks.rados.rados.0.trial043.stdout:301: done (1 left) 2026-02-20T11:17:01.127 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:17:01.127 INFO:tasks.rados.rados.0.trial043.stdout:296: finishing write tid 4 to trial04342147-38 2026-02-20T11:17:01.127 INFO:tasks.rados.rados.0.trial043.stdout:296: oid 38 updating version 0 to 18 2026-02-20T11:17:01.127 INFO:tasks.rados.rados.0.trial043.stdout:296: oid 38 version 18 is already newer than 18 2026-02-20T11:17:01.127 INFO:tasks.rados.rados.0.trial043.stdout:296: oid 38 version 18 is already newer than 18 2026-02-20T11:17:01.127 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 38 v 18 (ObjNum 162 snap 36 seq_num 162) dirty exists 2026-02-20T11:17:01.127 INFO:tasks.rados.rados.0.trial043.stdout:296: left oid 38 (ObjNum 162 snap 36 seq_num 162) 2026-02-20T11:17:01.127 INFO:tasks.rados.rados.0.trial043.stdout:296: done (0 left) 2026-02-20T11:17:01.800 INFO:tasks.rados.rados.0.trial043.stdout:302: done (0 left) 2026-02-20T11:17:01.800 INFO:tasks.rados.rados.0.trial043.stdout:303: read oid 13 snap 34 2026-02-20T11:17:01.800 INFO:tasks.rados.rados.0.trial043.stdout:303: expect (ObjNum 150 snap 29 seq_num 150) 2026-02-20T11:17:01.801 INFO:tasks.rados.rados.0.trial043.stdout:304: snap_remove snap 33 2026-02-20T11:17:02.127 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:17:02.199 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:17:02.202 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:17:02.202 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:17:02.221 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:17:02.297 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:02.296+0000 7f6ef199c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:02.299 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:02.300+0000 7f6ef199c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:02.326 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:17:02.342 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:17:02.343 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:17:02.386 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~77s 2026-02-20T11:17:02.803 INFO:tasks.rados.rados.0.trial043.stdout:304: done (1 left) 2026-02-20T11:17:02.804 INFO:tasks.rados.rados.0.trial043.stdout:305: write (excl) oid 36 current snap is 37 2026-02-20T11:17:02.804 INFO:tasks.rados.rados.0.trial043.stdout:305: seq_num 164 ranges {756348=784249,2063988=686387} 2026-02-20T11:17:02.809 INFO:tasks.rados.rados.0.trial043.stdout:305: writing trial04342147-36 from 756348 to 1540597 tid 1 2026-02-20T11:17:02.814 INFO:tasks.rados.rados.0.trial043.stdout:305: writing trial04342147-36 from 2063988 to 2750375 tid 2 2026-02-20T11:17:02.814 INFO:tasks.rados.rados.0.trial043.stdout:306: snap_create 2026-02-20T11:17:02.814 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:17:02.814 INFO:tasks.rados.rados.0.trial043.stdout:303: expect (ObjNum 150 snap 29 seq_num 150) 2026-02-20T11:17:02.867 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:17:02.897 INFO:tasks.rados.rados.0.trial043.stdout:305: finishing write tid 1 to trial04342147-36 2026-02-20T11:17:02.897 INFO:tasks.rados.rados.0.trial043.stdout:305: finishing write tid 2 to trial04342147-36 2026-02-20T11:17:02.897 INFO:tasks.rados.rados.0.trial043.stdout:303: done (1 left) 2026-02-20T11:17:02.897 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:17:02.897 INFO:tasks.rados.rados.0.trial043.stdout:305: finishing write tid 3 to trial04342147-36 2026-02-20T11:17:02.898 INFO:tasks.rados.rados.0.trial043.stdout:305: finishing write tid 4 to trial04342147-36 2026-02-20T11:17:02.898 INFO:tasks.rados.rados.0.trial043.stdout:305: oid 36 updating version 0 to 19 2026-02-20T11:17:02.898 INFO:tasks.rados.rados.0.trial043.stdout:305: oid 36 version 19 is already newer than 17 2026-02-20T11:17:02.898 INFO:tasks.rados.rados.0.trial043.stdout:305: oid 36 version 19 is already newer than 19 2026-02-20T11:17:02.898 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 36 v 19 (ObjNum 164 snap 37 seq_num 164) dirty exists 2026-02-20T11:17:02.898 INFO:tasks.rados.rados.0.trial043.stdout:305: left oid 36 (ObjNum 164 snap 37 seq_num 164) 2026-02-20T11:17:02.898 INFO:tasks.rados.rados.0.trial043.stdout:305: done (0 left) 2026-02-20T11:17:03.809 INFO:tasks.rados.rados.0.trial043.stdout:306: done (0 left) 2026-02-20T11:17:03.809 INFO:tasks.rados.rados.0.trial043.stdout:307: read oid 26 snap 37 2026-02-20T11:17:03.809 INFO:tasks.rados.rados.0.trial043.stdout:307: expect (ObjNum 143 snap 25 seq_num 143) 2026-02-20T11:17:03.809 INFO:tasks.rados.rados.0.trial043.stdout:308: write oid 23 current snap is 38 2026-02-20T11:17:03.809 INFO:tasks.rados.rados.0.trial043.stdout:308: seq_num 165 ranges {773155=459498,1722470=430045,2648676=124479} 2026-02-20T11:17:03.813 INFO:tasks.rados.rados.0.trial043.stdout:308: writing trial04342147-23 from 773155 to 1232653 tid 1 2026-02-20T11:17:03.817 INFO:tasks.rados.rados.0.trial043.stdout:308: writing trial04342147-23 from 1722470 to 2152515 tid 2 2026-02-20T11:17:03.820 INFO:tasks.rados.rados.0.trial043.stdout:308: writing trial04342147-23 from 2648676 to 2773155 tid 3 2026-02-20T11:17:03.820 INFO:tasks.rados.rados.0.trial043.stdout:309: read oid 1 snap -1 2026-02-20T11:17:03.820 INFO:tasks.rados.rados.0.trial043.stdout:309: expect (ObjNum 0 snap 0 seq_num 0) 2026-02-20T11:17:03.820 INFO:tasks.rados.rados.0.trial043.stdout:310: snap_create 2026-02-20T11:17:03.820 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:17:03.843 INFO:tasks.rados.rados.0.trial043.stdout:307: expect (ObjNum 143 snap 25 seq_num 143) 2026-02-20T11:17:03.843 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:17:03.879 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:17:03.951 INFO:tasks.rados.rados.0.trial043.stdout:309: expect (ObjNum 0 snap 0 seq_num 0) 2026-02-20T11:17:03.977 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:17:03.994 INFO:tasks.rados.rados.0.trial043.stdout:308: finishing write tid 1 to trial04342147-23 2026-02-20T11:17:03.994 INFO:tasks.rados.rados.0.trial043.stdout:308: finishing write tid 2 to trial04342147-23 2026-02-20T11:17:03.995 INFO:tasks.rados.rados.0.trial043.stdout:308: finishing write tid 3 to trial04342147-23 2026-02-20T11:17:03.995 INFO:tasks.rados.rados.0.trial043.stdout:307: done (2 left) 2026-02-20T11:17:03.995 INFO:tasks.rados.rados.0.trial043.stdout:309: done (1 left) 2026-02-20T11:17:03.995 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:17:03.995 INFO:tasks.rados.rados.0.trial043.stdout:308: finishing write tid 4 to trial04342147-23 2026-02-20T11:17:03.995 INFO:tasks.rados.rados.0.trial043.stdout:308: finishing write tid 5 to trial04342147-23 2026-02-20T11:17:03.995 INFO:tasks.rados.rados.0.trial043.stdout:308: oid 23 updating version 0 to 45 2026-02-20T11:17:03.995 INFO:tasks.rados.rados.0.trial043.stdout:308: oid 23 version 45 is already newer than 45 2026-02-20T11:17:03.995 INFO:tasks.rados.rados.0.trial043.stdout:308: oid 23 version 45 is already newer than 45 2026-02-20T11:17:03.995 INFO:tasks.rados.rados.0.trial043.stdout:308: oid 23 version 45 is already newer than 45 2026-02-20T11:17:03.995 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 23 v 45 (ObjNum 165 snap 38 seq_num 165) dirty exists 2026-02-20T11:17:03.996 INFO:tasks.rados.rados.0.trial043.stdout:308: left oid 23 (ObjNum 165 snap 38 seq_num 165) 2026-02-20T11:17:03.996 INFO:tasks.rados.rados.0.trial043.stdout:308: done (0 left) 2026-02-20T11:17:04.344 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:17:04.420 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:04.419+0000 7fe1ec1f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:04.422 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:04.423+0000 7fe1ec1f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:04.448 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:17:04.463 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:17:04.463 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:17:04.814 INFO:tasks.rados.rados.0.trial043.stdout:310: done (0 left) 2026-02-20T11:17:04.814 INFO:tasks.rados.rados.0.trial043.stdout:311: write oid 37 current snap is 39 2026-02-20T11:17:04.814 INFO:tasks.rados.rados.0.trial043.stdout:311: seq_num 166 ranges {789962=534747,1780952=690056} 2026-02-20T11:17:04.819 INFO:tasks.rados.rados.0.trial043.stdout:311: writing trial04342147-37 from 789962 to 1324709 tid 1 2026-02-20T11:17:04.824 INFO:tasks.rados.rados.0.trial043.stdout:311: writing trial04342147-37 from 1780952 to 2471008 tid 2 2026-02-20T11:17:04.824 INFO:tasks.rados.rados.0.trial043.stdout:312: write oid 9 current snap is 39 2026-02-20T11:17:04.824 INFO:tasks.rados.rados.0.trial043.stdout:312: seq_num 167 ranges {406769=609996,1439434=433714,2303522=503247} 2026-02-20T11:17:04.828 INFO:tasks.rados.rados.0.trial043.stdout:312: writing trial04342147-9 from 406769 to 1016765 tid 1 2026-02-20T11:17:04.831 INFO:tasks.rados.rados.0.trial043.stdout:312: writing trial04342147-9 from 1439434 to 1873148 tid 2 2026-02-20T11:17:04.835 INFO:tasks.rados.rados.0.trial043.stdout:312: writing trial04342147-9 from 2303522 to 2806769 tid 3 2026-02-20T11:17:04.835 INFO:tasks.rados.rados.0.trial043.stdout:313: delete oid 24 current snap is 39 2026-02-20T11:17:04.847 INFO:tasks.rados.rados.0.trial043.stdout:311: finishing write tid 1 to trial04342147-37 2026-02-20T11:17:04.854 INFO:tasks.rados.rados.0.trial043.stdout:311: finishing write tid 2 to trial04342147-37 2026-02-20T11:17:04.861 INFO:tasks.rados.rados.0.trial043.stdout:311: finishing write tid 3 to trial04342147-37 2026-02-20T11:17:04.862 INFO:tasks.rados.rados.0.trial043.stdout:311: finishing write tid 4 to trial04342147-37 2026-02-20T11:17:04.862 INFO:tasks.rados.rados.0.trial043.stdout:311: oid 37 updating version 0 to 96 2026-02-20T11:17:04.862 INFO:tasks.rados.rados.0.trial043.stdout:311: oid 37 version 96 is already newer than 96 2026-02-20T11:17:04.862 INFO:tasks.rados.rados.0.trial043.stdout:311: oid 37 version 96 is already newer than 96 2026-02-20T11:17:04.862 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 37 v 96 (ObjNum 166 snap 39 seq_num 166) dirty exists 2026-02-20T11:17:04.862 INFO:tasks.rados.rados.0.trial043.stdout:311: left oid 37 (ObjNum 166 snap 39 seq_num 166) 2026-02-20T11:17:04.865 INFO:tasks.rados.rados.0.trial043.stdout:311: done (2 left) 2026-02-20T11:17:04.865 INFO:tasks.rados.rados.0.trial043.stdout:313: done (1 left) 2026-02-20T11:17:04.865 INFO:tasks.rados.rados.0.trial043.stdout:314: snap_remove snap 37 2026-02-20T11:17:05.817 INFO:tasks.rados.rados.0.trial043.stdout:314: done (1 left) 2026-02-20T11:17:05.817 INFO:tasks.rados.rados.0.trial043.stdout:315: read oid 22 snap -1 2026-02-20T11:17:05.817 INFO:tasks.rados.rados.0.trial043.stdout:315: expect (ObjNum 160 snap 34 seq_num 160) 2026-02-20T11:17:05.817 INFO:tasks.rados.rados.0.trial043.stdout:316: read oid 21 snap -1 2026-02-20T11:17:05.818 INFO:tasks.rados.rados.0.trial043.stdout:316: expect deleted 2026-02-20T11:17:05.818 INFO:tasks.rados.rados.0.trial043.stdout:317: read oid 4 snap -1 2026-02-20T11:17:05.818 INFO:tasks.rados.rados.0.trial043.stdout:317: expect (ObjNum 154 snap 34 seq_num 154) 2026-02-20T11:17:05.818 INFO:tasks.rados.rados.0.trial043.stdout:318: delete oid 39 current snap is 39 2026-02-20T11:17:05.818 INFO:tasks.rados.rados.0.trial043.stdout:312: finishing write tid 1 to trial04342147-9 2026-02-20T11:17:05.818 INFO:tasks.rados.rados.0.trial043.stdout:312: finishing write tid 2 to trial04342147-9 2026-02-20T11:17:05.818 INFO:tasks.rados.rados.0.trial043.stdout:312: finishing write tid 3 to trial04342147-9 2026-02-20T11:17:05.818 INFO:tasks.rados.rados.0.trial043.stdout:312: finishing write tid 4 to trial04342147-9 2026-02-20T11:17:05.818 INFO:tasks.rados.rados.0.trial043.stdout:312: finishing write tid 5 to trial04342147-9 2026-02-20T11:17:05.818 INFO:tasks.rados.rados.0.trial043.stdout:312: oid 9 updating version 0 to 53 2026-02-20T11:17:05.818 INFO:tasks.rados.rados.0.trial043.stdout:312: oid 9 version 53 is already newer than 53 2026-02-20T11:17:05.818 INFO:tasks.rados.rados.0.trial043.stdout:312: oid 9 version 53 is already newer than 53 2026-02-20T11:17:05.819 INFO:tasks.rados.rados.0.trial043.stdout:312: oid 9 version 53 is already newer than 53 2026-02-20T11:17:05.819 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 9 v 53 (ObjNum 167 snap 39 seq_num 167) dirty exists 2026-02-20T11:17:05.819 INFO:tasks.rados.rados.0.trial043.stdout:312: left oid 9 (ObjNum 167 snap 39 seq_num 167) 2026-02-20T11:17:05.856 INFO:tasks.rados.rados.0.trial043.stdout:315: expect (ObjNum 160 snap 34 seq_num 160) 2026-02-20T11:17:05.887 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:17:05.925 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1774588 2026-02-20T11:17:05.942 INFO:tasks.rados.rados.0.trial043.stdout:312: done (4 left) 2026-02-20T11:17:05.942 INFO:tasks.rados.rados.0.trial043.stdout:315: done (3 left) 2026-02-20T11:17:05.942 INFO:tasks.rados.rados.0.trial043.stdout:316: done (2 left) 2026-02-20T11:17:05.942 INFO:tasks.rados.rados.0.trial043.stdout:318: done (1 left) 2026-02-20T11:17:05.942 INFO:tasks.rados.rados.0.trial043.stdout:319: write (excl) oid 13 current snap is 39 2026-02-20T11:17:05.942 INFO:tasks.rados.rados.0.trial043.stdout:317: expect (ObjNum 154 snap 34 seq_num 154) 2026-02-20T11:17:05.950 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 604541 2026-02-20T11:17:06.038 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 598311 2026-02-20T11:17:06.073 INFO:tasks.rados.rados.0.trial043.stdout:319: seq_num 170 ranges {457190=552096,1614880=581041,2785791=71399} 2026-02-20T11:17:06.076 INFO:tasks.rados.rados.0.trial043.stdout:319: writing trial04342147-13 from 457190 to 1009286 tid 1 2026-02-20T11:17:06.081 INFO:tasks.rados.rados.0.trial043.stdout:319: writing trial04342147-13 from 1614880 to 2195921 tid 2 2026-02-20T11:17:06.084 INFO:tasks.rados.rados.0.trial043.stdout:319: writing trial04342147-13 from 2785791 to 2857190 tid 3 2026-02-20T11:17:06.084 INFO:tasks.rados.rados.0.trial043.stdout:317: done (1 left) 2026-02-20T11:17:06.084 INFO:tasks.rados.rados.0.trial043.stdout:320: read oid 25 snap -1 2026-02-20T11:17:06.084 INFO:tasks.rados.rados.0.trial043.stdout:320: expect (ObjNum 116 snap 14 seq_num 116) 2026-02-20T11:17:06.084 INFO:tasks.rados.rados.0.trial043.stdout:321: snap_create 2026-02-20T11:17:06.084 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:17:06.122 INFO:tasks.rados.rados.0.trial043.stdout:319: finishing write tid 1 to trial04342147-13 2026-02-20T11:17:06.142 INFO:tasks.rados.rados.0.trial043.stdout:319: finishing write tid 2 to trial04342147-13 2026-02-20T11:17:06.152 INFO:tasks.rados.rados.0.trial043.stdout:319: finishing write tid 3 to trial04342147-13 2026-02-20T11:17:06.155 INFO:tasks.rados.rados.0.trial043.stdout:320: expect (ObjNum 116 snap 14 seq_num 116) 2026-02-20T11:17:06.220 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1299679 2026-02-20T11:17:06.317 INFO:tasks.rados.rados.0.trial043.stdout:319: finishing write tid 4 to trial04342147-13 2026-02-20T11:17:06.317 INFO:tasks.rados.rados.0.trial043.stdout:319: finishing write tid 5 to trial04342147-13 2026-02-20T11:17:06.317 INFO:tasks.rados.rados.0.trial043.stdout:319: oid 13 updating version 0 to 101 2026-02-20T11:17:06.317 INFO:tasks.rados.rados.0.trial043.stdout:319: oid 13 version 101 is already newer than 101 2026-02-20T11:17:06.318 INFO:tasks.rados.rados.0.trial043.stdout:319: oid 13 version 101 is already newer than 98 2026-02-20T11:17:06.318 INFO:tasks.rados.rados.0.trial043.stdout:319: oid 13 version 101 is already newer than 101 2026-02-20T11:17:06.318 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 13 v 101 (ObjNum 170 snap 39 seq_num 170) dirty exists 2026-02-20T11:17:06.318 INFO:tasks.rados.rados.0.trial043.stdout:319: left oid 13 (ObjNum 170 snap 39 seq_num 170) 2026-02-20T11:17:06.318 INFO:tasks.rados.rados.0.trial043.stdout:319: done (1 left) 2026-02-20T11:17:06.318 INFO:tasks.rados.rados.0.trial043.stdout:320: done (0 left) 2026-02-20T11:17:06.465 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:17:06.542 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:06.541+0000 7f010e0cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:06.545 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:06.545+0000 7f010e0cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:06.570 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:17:06.587 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:17:06.587 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:17:06.820 INFO:tasks.rados.rados.0.trial043.stdout:321: done (0 left) 2026-02-20T11:17:06.820 INFO:tasks.rados.rados.0.trial043.stdout:322: snap_remove snap 28 2026-02-20T11:17:07.203 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:17:07.275 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:17:07.277 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:17:07.278 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:17:07.823 INFO:tasks.rados.rados.0.trial043.stdout:322: done (0 left) 2026-02-20T11:17:07.823 INFO:tasks.rados.rados.0.trial043.stdout:323: read oid 22 snap -1 2026-02-20T11:17:07.823 INFO:tasks.rados.rados.0.trial043.stdout:323: expect (ObjNum 160 snap 34 seq_num 160) 2026-02-20T11:17:07.823 INFO:tasks.rados.rados.0.trial043.stdout:324: delete oid 28 current snap is 40 2026-02-20T11:17:07.864 INFO:tasks.rados.rados.0.trial043.stdout:324: done (1 left) 2026-02-20T11:17:07.865 INFO:tasks.rados.rados.0.trial043.stdout:325: read oid 9 snap -1 2026-02-20T11:17:07.865 INFO:tasks.rados.rados.0.trial043.stdout:325: expect (ObjNum 167 snap 39 seq_num 167) 2026-02-20T11:17:07.865 INFO:tasks.rados.rados.0.trial043.stdout:326: snap_remove snap 31 2026-02-20T11:17:07.890 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~83s 2026-02-20T11:17:08.588 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:17:08.662 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:08.661+0000 7f8e24d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:08.664 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:08.665+0000 7f8e24d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:08.691 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:17:08.707 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:17:08.707 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:17:08.744 INFO:tasks.rados.rados.0.trial043.stdout:326: done (2 left) 2026-02-20T11:17:08.744 INFO:tasks.rados.rados.0.trial043.stdout:327: snap_create 2026-02-20T11:17:08.744 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:17:08.745 INFO:tasks.rados.rados.0.trial043.stdout:323: expect (ObjNum 160 snap 34 seq_num 160) 2026-02-20T11:17:08.749 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 710966 2026-02-20T11:17:08.803 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:17:08.835 INFO:tasks.rados.rados.0.trial043.stdout:325: expect (ObjNum 167 snap 39 seq_num 167) 2026-02-20T11:17:08.925 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2773022 2026-02-20T11:17:08.925 INFO:tasks.rados.rados.0.trial043.stdout:323: done (1 left) 2026-02-20T11:17:08.925 INFO:tasks.rados.rados.0.trial043.stdout:325: done (0 left) 2026-02-20T11:17:09.829 INFO:tasks.rados.rados.0.trial043.stdout:327: done (0 left) 2026-02-20T11:17:09.829 INFO:tasks.rados.rados.0.trial043.stdout:328: snap_create 2026-02-20T11:17:10.708 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:17:10.785 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:10.783+0000 7f25d9c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:10.787 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:10.786+0000 7f25d9c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:10.813 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:17:10.829 INFO:tasks.rados.rados.0.trial043.stdout:328: done (0 left) 2026-02-20T11:17:10.829 INFO:tasks.rados.rados.0.trial043.stdout:329: write (excl) oid 14 current snap is 42 2026-02-20T11:17:10.829 INFO:tasks.rados.rados.0.trial043.stdout:329: seq_num 172 ranges {490804=702594,1615491=584710,2724284=166520} 2026-02-20T11:17:10.830 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:17:10.831 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:17:10.833 INFO:tasks.rados.rados.0.trial043.stdout:329: writing trial04342147-14 from 490804 to 1193398 tid 1 2026-02-20T11:17:10.837 INFO:tasks.rados.rados.0.trial043.stdout:329: writing trial04342147-14 from 1615491 to 2200201 tid 2 2026-02-20T11:17:10.840 INFO:tasks.rados.rados.0.trial043.stdout:329: writing trial04342147-14 from 2724284 to 2890804 tid 3 2026-02-20T11:17:10.840 INFO:tasks.rados.rados.0.trial043.stdout:330: write (excl) oid 17 current snap is 42 2026-02-20T11:17:10.840 INFO:tasks.rados.rados.0.trial043.stdout:330: seq_num 173 ranges {507611=777843,2073973=444721} 2026-02-20T11:17:10.844 INFO:tasks.rados.rados.0.trial043.stdout:330: writing trial04342147-17 from 507611 to 1285454 tid 1 2026-02-20T11:17:10.849 INFO:tasks.rados.rados.0.trial043.stdout:330: writing trial04342147-17 from 2073973 to 2518694 tid 2 2026-02-20T11:17:10.849 INFO:tasks.rados.rados.0.trial043.stdout:331: write oid 24 current snap is 42 2026-02-20T11:17:10.849 INFO:tasks.rados.rados.0.trial043.stdout:331: seq_num 174 ranges {524418=453092,1732455=588379,2779130=145288} 2026-02-20T11:17:10.853 INFO:tasks.rados.rados.0.trial043.stdout:331: writing trial04342147-24 from 524418 to 977510 tid 1 2026-02-20T11:17:10.858 INFO:tasks.rados.rados.0.trial043.stdout:331: writing trial04342147-24 from 1732455 to 2320834 tid 2 2026-02-20T11:17:10.860 INFO:tasks.rados.rados.0.trial043.stdout:331: writing trial04342147-24 from 2779130 to 2924418 tid 3 2026-02-20T11:17:10.860 INFO:tasks.rados.rados.0.trial043.stdout:332: snap_remove snap 41 2026-02-20T11:17:11.835 INFO:tasks.rados.rados.0.trial043.stdout:332: done (3 left) 2026-02-20T11:17:11.835 INFO:tasks.rados.rados.0.trial043.stdout:333: snap_create 2026-02-20T11:17:11.835 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:17:11.835 INFO:tasks.rados.rados.0.trial043.stdout:329: finishing write tid 1 to trial04342147-14 2026-02-20T11:17:11.836 INFO:tasks.rados.rados.0.trial043.stdout:329: finishing write tid 2 to trial04342147-14 2026-02-20T11:17:11.836 INFO:tasks.rados.rados.0.trial043.stdout:329: finishing write tid 3 to trial04342147-14 2026-02-20T11:17:11.836 INFO:tasks.rados.rados.0.trial043.stdout:329: finishing write tid 4 to trial04342147-14 2026-02-20T11:17:11.836 INFO:tasks.rados.rados.0.trial043.stdout:329: finishing write tid 5 to trial04342147-14 2026-02-20T11:17:11.836 INFO:tasks.rados.rados.0.trial043.stdout:329: oid 14 updating version 0 to 64 2026-02-20T11:17:11.836 INFO:tasks.rados.rados.0.trial043.stdout:329: oid 14 version 64 is already newer than 64 2026-02-20T11:17:11.836 INFO:tasks.rados.rados.0.trial043.stdout:329: oid 14 version 64 is already newer than 61 2026-02-20T11:17:11.836 INFO:tasks.rados.rados.0.trial043.stdout:329: oid 14 version 64 is already newer than 64 2026-02-20T11:17:11.836 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 14 v 64 (ObjNum 172 snap 42 seq_num 172) dirty exists 2026-02-20T11:17:11.836 INFO:tasks.rados.rados.0.trial043.stdout:329: left oid 14 (ObjNum 172 snap 42 seq_num 172) 2026-02-20T11:17:11.836 INFO:tasks.rados.rados.0.trial043.stdout:331: finishing write tid 1 to trial04342147-24 2026-02-20T11:17:11.837 INFO:tasks.rados.rados.0.trial043.stdout:329: done (2 left) 2026-02-20T11:17:11.837 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:17:11.837 INFO:tasks.rados.rados.0.trial043.stdout:331: finishing write tid 2 to trial04342147-24 2026-02-20T11:17:11.837 INFO:tasks.rados.rados.0.trial043.stdout:331: finishing write tid 3 to trial04342147-24 2026-02-20T11:17:11.837 INFO:tasks.rados.rados.0.trial043.stdout:331: finishing write tid 4 to trial04342147-24 2026-02-20T11:17:11.837 INFO:tasks.rados.rados.0.trial043.stdout:331: finishing write tid 5 to trial04342147-24 2026-02-20T11:17:11.837 INFO:tasks.rados.rados.0.trial043.stdout:331: oid 24 updating version 0 to 29 2026-02-20T11:17:11.837 INFO:tasks.rados.rados.0.trial043.stdout:331: oid 24 version 29 is already newer than 29 2026-02-20T11:17:11.837 INFO:tasks.rados.rados.0.trial043.stdout:331: oid 24 version 29 is already newer than 29 2026-02-20T11:17:11.837 INFO:tasks.rados.rados.0.trial043.stdout:331: oid 24 version 29 is already newer than 29 2026-02-20T11:17:11.837 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 24 v 29 (ObjNum 174 snap 42 seq_num 174) dirty exists 2026-02-20T11:17:11.837 INFO:tasks.rados.rados.0.trial043.stdout:331: left oid 24 (ObjNum 174 snap 42 seq_num 174) 2026-02-20T11:17:11.838 INFO:tasks.rados.rados.0.trial043.stdout:330: finishing write tid 1 to trial04342147-17 2026-02-20T11:17:11.838 INFO:tasks.rados.rados.0.trial043.stdout:331: done (1 left) 2026-02-20T11:17:11.838 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:17:11.838 INFO:tasks.rados.rados.0.trial043.stdout:330: finishing write tid 2 to trial04342147-17 2026-02-20T11:17:11.838 INFO:tasks.rados.rados.0.trial043.stdout:330: finishing write tid 3 to trial04342147-17 2026-02-20T11:17:11.838 INFO:tasks.rados.rados.0.trial043.stdout:330: finishing write tid 4 to trial04342147-17 2026-02-20T11:17:11.838 INFO:tasks.rados.rados.0.trial043.stdout:330: oid 17 updating version 0 to 66 2026-02-20T11:17:11.838 INFO:tasks.rados.rados.0.trial043.stdout:330: oid 17 updating version 66 to 68 2026-02-20T11:17:11.838 INFO:tasks.rados.rados.0.trial043.stdout:330: oid 17 version 68 is already newer than 68 2026-02-20T11:17:11.838 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 17 v 68 (ObjNum 173 snap 42 seq_num 173) dirty exists 2026-02-20T11:17:11.838 INFO:tasks.rados.rados.0.trial043.stdout:330: left oid 17 (ObjNum 173 snap 42 seq_num 173) 2026-02-20T11:17:11.838 INFO:tasks.rados.rados.0.trial043.stdout:330: done (0 left) 2026-02-20T11:17:12.279 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:17:12.351 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:17:12.354 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:17:12.354 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:17:12.832 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:17:12.838 INFO:tasks.rados.rados.0.trial043.stdout:333: done (0 left) 2026-02-20T11:17:12.838 INFO:tasks.rados.rados.0.trial043.stdout:334: delete oid 12 current snap is 43 2026-02-20T11:17:12.879 INFO:tasks.rados.rados.0.trial043.stdout:334: done (0 left) 2026-02-20T11:17:12.880 INFO:tasks.rados.rados.0.trial043.stdout:335: write (excl) oid 43 current snap is 43 2026-02-20T11:17:12.880 INFO:tasks.rados.rados.0.trial043.stdout:335: seq_num 176 ranges {558032=719943,1849419=592048} 2026-02-20T11:17:12.884 INFO:tasks.rados.rados.0.trial043.stdout:335: writing trial04342147-43 from 558032 to 1277975 tid 1 2026-02-20T11:17:12.889 INFO:tasks.rados.rados.0.trial043.stdout:335: writing trial04342147-43 from 1849419 to 2441467 tid 2 2026-02-20T11:17:12.889 INFO:tasks.rados.rados.0.trial043.stdout:336: snap_create 2026-02-20T11:17:12.890 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:17:12.912 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:12.910+0000 7f748b964640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:12.915 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:12.914+0000 7f748b964640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:12.915 INFO:tasks.rados.rados.0.trial043.stdout:335: finishing write tid 1 to trial04342147-43 2026-02-20T11:17:12.935 INFO:tasks.rados.rados.0.trial043.stdout:335: finishing write tid 2 to trial04342147-43 2026-02-20T11:17:12.940 INFO:tasks.rados.rados.0.trial043.stdout:335: finishing write tid 3 to trial04342147-43 2026-02-20T11:17:12.942 INFO:tasks.rados.rados.0.trial043.stdout:335: finishing write tid 4 to trial04342147-43 2026-02-20T11:17:12.942 INFO:tasks.rados.rados.0.trial043.stdout:335: oid 43 updating version 0 to 67 2026-02-20T11:17:12.942 INFO:tasks.rados.rados.0.trial043.stdout:335: oid 43 version 67 is already newer than 67 2026-02-20T11:17:12.942 INFO:tasks.rados.rados.0.trial043.stdout:335: oid 43 version 67 is already newer than 65 2026-02-20T11:17:12.942 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 43 v 67 (ObjNum 176 snap 43 seq_num 176) dirty exists 2026-02-20T11:17:12.942 INFO:tasks.rados.rados.0.trial043.stdout:335: left oid 43 (ObjNum 176 snap 43 seq_num 176) 2026-02-20T11:17:12.942 INFO:tasks.rados.rados.0.trial043.stdout:335: done (0 left) 2026-02-20T11:17:12.950 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:17:12.966 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:17:12.966 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:17:13.394 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~88s 2026-02-20T11:17:13.842 INFO:tasks.rados.rados.0.trial043.stdout:336: done (0 left) 2026-02-20T11:17:13.842 INFO:tasks.rados.rados.0.trial043.stdout:337: snap_create 2026-02-20T11:17:14.847 INFO:tasks.rados.rados.0.trial043.stdout:337: done (0 left) 2026-02-20T11:17:14.847 INFO:tasks.rados.rados.0.trial043.stdout:338: write oid 45 current snap is 45 2026-02-20T11:17:14.847 INFO:tasks.rados.rados.0.trial043.stdout:338: seq_num 177 ranges {574839=795192,1907901=452059} 2026-02-20T11:17:14.851 INFO:tasks.rados.rados.0.trial043.stdout:338: writing trial04342147-45 from 574839 to 1370031 tid 1 2026-02-20T11:17:14.855 INFO:tasks.rados.rados.0.trial043.stdout:338: writing trial04342147-45 from 1907901 to 2359960 tid 2 2026-02-20T11:17:14.855 INFO:tasks.rados.rados.0.trial043.stdout:339: snap_create 2026-02-20T11:17:14.856 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:17:14.891 INFO:tasks.rados.rados.0.trial043.stdout:338: finishing write tid 1 to trial04342147-45 2026-02-20T11:17:14.896 INFO:tasks.rados.rados.0.trial043.stdout:338: finishing write tid 2 to trial04342147-45 2026-02-20T11:17:14.902 INFO:tasks.rados.rados.0.trial043.stdout:338: finishing write tid 3 to trial04342147-45 2026-02-20T11:17:14.903 INFO:tasks.rados.rados.0.trial043.stdout:338: finishing write tid 4 to trial04342147-45 2026-02-20T11:17:14.903 INFO:tasks.rados.rados.0.trial043.stdout:338: oid 45 updating version 0 to 105 2026-02-20T11:17:14.903 INFO:tasks.rados.rados.0.trial043.stdout:338: oid 45 version 105 is already newer than 105 2026-02-20T11:17:14.903 INFO:tasks.rados.rados.0.trial043.stdout:338: oid 45 version 105 is already newer than 105 2026-02-20T11:17:14.903 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 45 v 105 (ObjNum 177 snap 45 seq_num 177) dirty exists 2026-02-20T11:17:14.903 INFO:tasks.rados.rados.0.trial043.stdout:338: left oid 45 (ObjNum 177 snap 45 seq_num 177) 2026-02-20T11:17:14.903 INFO:tasks.rados.rados.0.trial043.stdout:338: done (0 left) 2026-02-20T11:17:14.967 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:17:15.043 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:15.042+0000 7f0ddfccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:15.046 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:15.045+0000 7f0ddfccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:15.072 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:17:15.087 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:17:15.088 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:17:15.851 INFO:tasks.rados.rados.0.trial043.stdout:339: done (0 left) 2026-02-20T11:17:15.851 INFO:tasks.rados.rados.0.trial043.stdout:340: snap_create 2026-02-20T11:17:16.856 INFO:tasks.rados.rados.0.trial043.stdout:340: done (0 left) 2026-02-20T11:17:16.856 INFO:tasks.rados.rados.0.trial043.stdout:341: delete oid 13 current snap is 47 2026-02-20T11:17:16.896 INFO:tasks.rados.rados.0.trial043.stdout:341: done (0 left) 2026-02-20T11:17:16.896 INFO:tasks.rados.rados.0.trial043.stdout:342: write (excl) oid 36 current snap is 47 2026-02-20T11:17:16.896 INFO:tasks.rados.rados.0.trial043.stdout:342: seq_num 179 ranges {608453=545690,1624865=455728,2632598=375855} 2026-02-20T11:17:16.900 INFO:tasks.rados.rados.0.trial043.stdout:342: writing trial04342147-36 from 608453 to 1154143 tid 1 2026-02-20T11:17:16.903 INFO:tasks.rados.rados.0.trial043.stdout:342: writing trial04342147-36 from 1624865 to 2080593 tid 2 2026-02-20T11:17:16.907 INFO:tasks.rados.rados.0.trial043.stdout:342: writing trial04342147-36 from 2632598 to 3008453 tid 3 2026-02-20T11:17:16.907 INFO:tasks.rados.rados.0.trial043.stdout:343: snap_create 2026-02-20T11:17:16.907 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:17:16.926 INFO:tasks.rados.rados.0.trial043.stdout:342: finishing write tid 1 to trial04342147-36 2026-02-20T11:17:16.940 INFO:tasks.rados.rados.0.trial043.stdout:342: finishing write tid 2 to trial04342147-36 2026-02-20T11:17:16.947 INFO:tasks.rados.rados.0.trial043.stdout:342: finishing write tid 3 to trial04342147-36 2026-02-20T11:17:16.953 INFO:tasks.rados.rados.0.trial043.stdout:342: finishing write tid 4 to trial04342147-36 2026-02-20T11:17:16.955 INFO:tasks.rados.rados.0.trial043.stdout:342: finishing write tid 5 to trial04342147-36 2026-02-20T11:17:16.956 INFO:tasks.rados.rados.0.trial043.stdout:342: oid 36 updating version 0 to 26 2026-02-20T11:17:16.956 INFO:tasks.rados.rados.0.trial043.stdout:342: oid 36 version 26 is already newer than 23 2026-02-20T11:17:16.956 INFO:tasks.rados.rados.0.trial043.stdout:342: oid 36 version 26 is already newer than 26 2026-02-20T11:17:16.956 INFO:tasks.rados.rados.0.trial043.stdout:342: oid 36 version 26 is already newer than 26 2026-02-20T11:17:16.956 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 36 v 26 (ObjNum 179 snap 47 seq_num 179) dirty exists 2026-02-20T11:17:16.956 INFO:tasks.rados.rados.0.trial043.stdout:342: left oid 36 (ObjNum 179 snap 47 seq_num 179) 2026-02-20T11:17:16.956 INFO:tasks.rados.rados.0.trial043.stdout:342: done (0 left) 2026-02-20T11:17:17.089 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:17:17.167 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:17.165+0000 7f92ea411640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:17.169 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:17.168+0000 7f92ea411640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:17.194 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:17:17.207 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:17:17.208 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:17:17.355 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:17:17.427 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:17:17.429 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:17:17.430 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:17:17.858 INFO:tasks.rados.rados.0.trial043.stdout:343: done (0 left) 2026-02-20T11:17:17.858 INFO:tasks.rados.rados.0.trial043.stdout:344: read oid 48 snap -1 2026-02-20T11:17:17.858 INFO:tasks.rados.rados.0.trial043.stdout:344: expect (ObjNum 147 snap 26 seq_num 147) 2026-02-20T11:17:17.858 INFO:tasks.rados.rados.0.trial043.stdout:345: snap_remove snap 35 2026-02-20T11:17:18.748 INFO:tasks.rados.rados.0.trial043.stdout:345: done (1 left) 2026-02-20T11:17:18.748 INFO:tasks.rados.rados.0.trial043.stdout:346: write oid 27 current snap is 48 2026-02-20T11:17:18.748 INFO:tasks.rados.rados.0.trial043.stdout:346: seq_num 180 ranges {625260=620939,1683347=599386,2943668=81592} 2026-02-20T11:17:18.753 INFO:tasks.rados.rados.0.trial043.stdout:346: writing trial04342147-27 from 625260 to 1246199 tid 1 2026-02-20T11:17:18.758 INFO:tasks.rados.rados.0.trial043.stdout:346: writing trial04342147-27 from 1683347 to 2282733 tid 2 2026-02-20T11:17:18.761 INFO:tasks.rados.rados.0.trial043.stdout:346: writing trial04342147-27 from 2943668 to 3025260 tid 3 2026-02-20T11:17:18.761 INFO:tasks.rados.rados.0.trial043.stdout:347: read oid 23 snap -1 2026-02-20T11:17:18.761 INFO:tasks.rados.rados.0.trial043.stdout:347: expect (ObjNum 165 snap 38 seq_num 165) 2026-02-20T11:17:18.761 INFO:tasks.rados.rados.0.trial043.stdout:348: read oid 28 snap 44 2026-02-20T11:17:18.762 INFO:tasks.rados.rados.0.trial043.stdout:348: expect deleted 2026-02-20T11:17:18.762 INFO:tasks.rados.rados.0.trial043.stdout:349: snap_remove snap 47 2026-02-20T11:17:18.900 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~94s 2026-02-20T11:17:19.209 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:17:19.285 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:19.283+0000 7eff851a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:19.288 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:19.287+0000 7eff851a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:19.313 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:17:19.330 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:17:19.330 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:17:19.749 INFO:tasks.rados.rados.0.trial043.stdout:349: done (4 left) 2026-02-20T11:17:19.749 INFO:tasks.rados.rados.0.trial043.stdout:350: snap_remove snap 43 2026-02-20T11:17:20.750 INFO:tasks.rados.rados.0.trial043.stdout:350: done (4 left) 2026-02-20T11:17:20.750 INFO:tasks.rados.rados.0.trial043.stdout:351: write (excl) oid 26 current snap is 48 2026-02-20T11:17:20.750 INFO:tasks.rados.rados.0.trial043.stdout:351: seq_num 181 ranges {642067=696188,2025476=459397,2971091=70976} 2026-02-20T11:17:20.754 INFO:tasks.rados.rados.0.trial043.stdout:351: writing trial04342147-26 from 642067 to 1338255 tid 1 2026-02-20T11:17:20.759 INFO:tasks.rados.rados.0.trial043.stdout:351: writing trial04342147-26 from 2025476 to 2484873 tid 2 2026-02-20T11:17:20.761 INFO:tasks.rados.rados.0.trial043.stdout:351: writing trial04342147-26 from 2971091 to 3042067 tid 3 2026-02-20T11:17:20.761 INFO:tasks.rados.rados.0.trial043.stdout:344: expect (ObjNum 147 snap 26 seq_num 147) 2026-02-20T11:17:20.766 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2445181 2026-02-20T11:17:20.770 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:17:20.794 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1193846 2026-02-20T11:17:20.808 INFO:tasks.rados.rados.0.trial043.stdout:347: expect (ObjNum 165 snap 38 seq_num 165) 2026-02-20T11:17:20.808 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:17:20.923 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2738151 2026-02-20T11:17:20.924 INFO:tasks.rados.rados.0.trial043.stdout:346: finishing write tid 1 to trial04342147-27 2026-02-20T11:17:20.924 INFO:tasks.rados.rados.0.trial043.stdout:346: finishing write tid 2 to trial04342147-27 2026-02-20T11:17:20.924 INFO:tasks.rados.rados.0.trial043.stdout:346: finishing write tid 3 to trial04342147-27 2026-02-20T11:17:20.924 INFO:tasks.rados.rados.0.trial043.stdout:346: finishing write tid 4 to trial04342147-27 2026-02-20T11:17:20.924 INFO:tasks.rados.rados.0.trial043.stdout:346: finishing write tid 5 to trial04342147-27 2026-02-20T11:17:20.924 INFO:tasks.rados.rados.0.trial043.stdout:346: oid 27 updating version 0 to 76 2026-02-20T11:17:20.924 INFO:tasks.rados.rados.0.trial043.stdout:346: oid 27 version 76 is already newer than 76 2026-02-20T11:17:20.924 INFO:tasks.rados.rados.0.trial043.stdout:346: oid 27 version 76 is already newer than 76 2026-02-20T11:17:20.924 INFO:tasks.rados.rados.0.trial043.stdout:346: oid 27 version 76 is already newer than 76 2026-02-20T11:17:20.925 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 27 v 76 (ObjNum 180 snap 48 seq_num 180) dirty exists 2026-02-20T11:17:20.925 INFO:tasks.rados.rados.0.trial043.stdout:346: left oid 27 (ObjNum 180 snap 48 seq_num 180) 2026-02-20T11:17:20.925 INFO:tasks.rados.rados.0.trial043.stdout:351: finishing write tid 1 to trial04342147-26 2026-02-20T11:17:20.925 INFO:tasks.rados.rados.0.trial043.stdout:352: read oid 14 snap 38 2026-02-20T11:17:20.925 INFO:tasks.rados.rados.0.trial043.stdout:352: expect deleted 2026-02-20T11:17:20.925 INFO:tasks.rados.rados.0.trial043.stdout:344: done (5 left) 2026-02-20T11:17:20.925 INFO:tasks.rados.rados.0.trial043.stdout:346: done (4 left) 2026-02-20T11:17:20.925 INFO:tasks.rados.rados.0.trial043.stdout:347: done (3 left) 2026-02-20T11:17:20.925 INFO:tasks.rados.rados.0.trial043.stdout:348: done (2 left) 2026-02-20T11:17:20.925 INFO:tasks.rados.rados.0.trial043.stdout:353: write (excl) oid 9 current snap is 48 2026-02-20T11:17:20.925 INFO:tasks.rados.rados.0.trial043.stdout:351: finishing write tid 2 to trial04342147-26 2026-02-20T11:17:20.925 INFO:tasks.rados.rados.0.trial043.stdout:351: finishing write tid 3 to trial04342147-26 2026-02-20T11:17:20.926 INFO:tasks.rados.rados.0.trial043.stdout:351: finishing write tid 4 to trial04342147-26 2026-02-20T11:17:20.926 INFO:tasks.rados.rados.0.trial043.stdout:351: finishing write tid 5 to trial04342147-26 2026-02-20T11:17:20.926 INFO:tasks.rados.rados.0.trial043.stdout:351: oid 26 updating version 0 to 81 2026-02-20T11:17:20.926 INFO:tasks.rados.rados.0.trial043.stdout:351: oid 26 version 81 is already newer than 81 2026-02-20T11:17:20.926 INFO:tasks.rados.rados.0.trial043.stdout:351: oid 26 version 81 is already newer than 78 2026-02-20T11:17:20.926 INFO:tasks.rados.rados.0.trial043.stdout:351: oid 26 version 81 is already newer than 81 2026-02-20T11:17:20.926 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 26 v 81 (ObjNum 181 snap 48 seq_num 181) dirty exists 2026-02-20T11:17:20.926 INFO:tasks.rados.rados.0.trial043.stdout:351: left oid 26 (ObjNum 181 snap 48 seq_num 181) 2026-02-20T11:17:20.926 INFO:tasks.rados.rados.0.trial043.stdout:353: seq_num 182 ranges {658874=771437,2083958=603055} 2026-02-20T11:17:20.928 INFO:tasks.rados.rados.0.trial043.stdout:353: writing trial04342147-9 from 658874 to 1430311 tid 1 2026-02-20T11:17:20.933 INFO:tasks.rados.rados.0.trial043.stdout:353: writing trial04342147-9 from 2083958 to 2687013 tid 2 2026-02-20T11:17:20.934 INFO:tasks.rados.rados.0.trial043.stdout:351: done (2 left) 2026-02-20T11:17:20.934 INFO:tasks.rados.rados.0.trial043.stdout:354: read oid 50 snap 34 2026-02-20T11:17:20.934 INFO:tasks.rados.rados.0.trial043.stdout:354: expect (ObjNum 49 snap 0 seq_num 49) 2026-02-20T11:17:20.934 INFO:tasks.rados.rados.0.trial043.stdout:355: read oid 17 snap 39 2026-02-20T11:17:20.934 INFO:tasks.rados.rados.0.trial043.stdout:355: expect (ObjNum 121 snap 16 seq_num 121) 2026-02-20T11:17:20.934 INFO:tasks.rados.rados.0.trial043.stdout:356: snap_remove snap 45 2026-02-20T11:17:21.332 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:17:21.408 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:21.406+0000 7f8e7d90b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:21.411 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:21.410+0000 7f8e7d90b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:21.436 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:17:21.452 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:17:21.453 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:17:21.866 INFO:tasks.rados.rados.0.trial043.stdout:356: done (4 left) 2026-02-20T11:17:21.866 INFO:tasks.rados.rados.0.trial043.stdout:357: snap_remove snap 27 2026-02-20T11:17:22.431 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:17:22.503 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:17:22.505 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:17:22.506 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:17:22.868 INFO:tasks.rados.rados.0.trial043.stdout:357: done (4 left) 2026-02-20T11:17:22.868 INFO:tasks.rados.rados.0.trial043.stdout:358: read oid 27 snap -1 2026-02-20T11:17:22.868 INFO:tasks.rados.rados.0.trial043.stdout:358: expect (ObjNum 180 snap 48 seq_num 180) 2026-02-20T11:17:22.868 INFO:tasks.rados.rados.0.trial043.stdout:359: write oid 11 current snap is 48 2026-02-20T11:17:22.868 INFO:tasks.rados.rados.0.trial043.stdout:359: seq_num 183 ranges {675681=563039,1858793=746713,3025937=49744} 2026-02-20T11:17:22.873 INFO:tasks.rados.rados.0.trial043.stdout:359: writing trial04342147-11 from 675681 to 1238720 tid 1 2026-02-20T11:17:22.879 INFO:tasks.rados.rados.0.trial043.stdout:359: writing trial04342147-11 from 1858793 to 2605506 tid 2 2026-02-20T11:17:22.881 INFO:tasks.rados.rados.0.trial043.stdout:359: writing trial04342147-11 from 3025937 to 3075681 tid 3 2026-02-20T11:17:22.881 INFO:tasks.rados.rados.0.trial043.stdout:360: write (excl) oid 49 current snap is 48 2026-02-20T11:17:22.881 INFO:tasks.rados.rados.0.trial043.stdout:360: seq_num 184 ranges {692488=638288,1917275=606724} 2026-02-20T11:17:22.886 INFO:tasks.rados.rados.0.trial043.stdout:360: writing trial04342147-49 from 692488 to 1330776 tid 1 2026-02-20T11:17:22.891 INFO:tasks.rados.rados.0.trial043.stdout:360: writing trial04342147-49 from 1917275 to 2523999 tid 2 2026-02-20T11:17:22.891 INFO:tasks.rados.rados.0.trial043.stdout:361: read oid 12 snap -1 2026-02-20T11:17:22.891 INFO:tasks.rados.rados.0.trial043.stdout:361: expect deleted 2026-02-20T11:17:22.891 INFO:tasks.rados.rados.0.trial043.stdout:362: snap_remove snap 42 2026-02-20T11:17:23.454 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:17:23.531 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:23.528+0000 7f7f6b3c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:23.533 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:23.532+0000 7f7f6b3c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:23.559 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:17:23.575 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:17:23.576 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:17:23.873 INFO:tasks.rados.rados.0.trial043.stdout:362: done (8 left) 2026-02-20T11:17:23.873 INFO:tasks.rados.rados.0.trial043.stdout:363: read oid 4 snap 40 2026-02-20T11:17:23.873 INFO:tasks.rados.rados.0.trial043.stdout:363: expect (ObjNum 154 snap 34 seq_num 154) 2026-02-20T11:17:23.873 INFO:tasks.rados.rados.0.trial043.stdout:364: write oid 3 current snap is 48 2026-02-20T11:17:23.873 INFO:tasks.rados.rados.0.trial043.stdout:364: seq_num 185 ranges {709295=713537,1859404=750382} 2026-02-20T11:17:23.878 INFO:tasks.rados.rados.0.trial043.stdout:364: writing trial04342147-3 from 709295 to 1422832 tid 1 2026-02-20T11:17:23.883 INFO:tasks.rados.rados.0.trial043.stdout:364: writing trial04342147-3 from 1859404 to 2609786 tid 2 2026-02-20T11:17:23.883 INFO:tasks.rados.rados.0.trial043.stdout:352: done (9 left) 2026-02-20T11:17:23.884 INFO:tasks.rados.rados.0.trial043.stdout:365: snap_remove snap 44 2026-02-20T11:17:24.404 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~99s 2026-02-20T11:17:24.878 INFO:tasks.rados.rados.0.trial043.stdout:365: done (9 left) 2026-02-20T11:17:24.878 INFO:tasks.rados.rados.0.trial043.stdout:366: read oid 21 snap -1 2026-02-20T11:17:24.878 INFO:tasks.rados.rados.0.trial043.stdout:366: expect deleted 2026-02-20T11:17:24.878 INFO:tasks.rados.rados.0.trial043.stdout:367: write (excl) oid 24 current snap is 48 2026-02-20T11:17:24.878 INFO:tasks.rados.rados.0.trial043.stdout:367: seq_num 186 ranges {726102=788786,1917886=610393,3108206=17896} 2026-02-20T11:17:24.884 INFO:tasks.rados.rados.0.trial043.stdout:367: writing trial04342147-24 from 726102 to 1514888 tid 1 2026-02-20T11:17:24.888 INFO:tasks.rados.rados.0.trial043.stdout:367: writing trial04342147-24 from 1917886 to 2528279 tid 2 2026-02-20T11:17:24.891 INFO:tasks.rados.rados.0.trial043.stdout:367: writing trial04342147-24 from 3108206 to 3126102 tid 3 2026-02-20T11:17:24.891 INFO:tasks.rados.rados.0.trial043.stdout:368: write oid 13 current snap is 48 2026-02-20T11:17:24.891 INFO:tasks.rados.rados.0.trial043.stdout:368: seq_num 187 ranges {742909=464035,1976368=754051} 2026-02-20T11:17:24.895 INFO:tasks.rados.rados.0.trial043.stdout:368: writing trial04342147-13 from 742909 to 1206944 tid 1 2026-02-20T11:17:24.902 INFO:tasks.rados.rados.0.trial043.stdout:368: writing trial04342147-13 from 1976368 to 2730419 tid 2 2026-02-20T11:17:24.902 INFO:tasks.rados.rados.0.trial043.stdout:369: write oid 1 current snap is 48 2026-02-20T11:17:24.902 INFO:tasks.rados.rados.0.trial043.stdout:369: seq_num 188 ranges {759716=539284,2034850=614062} 2026-02-20T11:17:24.906 INFO:tasks.rados.rados.0.trial043.stdout:369: writing trial04342147-1 from 759716 to 1299000 tid 1 2026-02-20T11:17:24.911 INFO:tasks.rados.rados.0.trial043.stdout:369: writing trial04342147-1 from 2034850 to 2648912 tid 2 2026-02-20T11:17:24.912 INFO:tasks.rados.rados.0.trial043.stdout:370: delete oid 5 current snap is 48 2026-02-20T11:17:24.912 INFO:tasks.rados.rados.0.trial043.stdout:354: expect (ObjNum 49 snap 0 seq_num 49) 2026-02-20T11:17:24.921 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2477848 2026-02-20T11:17:24.924 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:17:24.947 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:17:24.971 INFO:tasks.rados.rados.0.trial043.stdout:353: finishing write tid 1 to trial04342147-9 2026-02-20T11:17:24.971 INFO:tasks.rados.rados.0.trial043.stdout:354: done (13 left) 2026-02-20T11:17:24.971 INFO:tasks.rados.rados.0.trial043.stdout:370: done (12 left) 2026-02-20T11:17:24.971 INFO:tasks.rados.rados.0.trial043.stdout:371: read oid 16 snap -1 2026-02-20T11:17:24.971 INFO:tasks.rados.rados.0.trial043.stdout:371: expect (ObjNum 90 snap 10 seq_num 90) 2026-02-20T11:17:24.971 INFO:tasks.rados.rados.0.trial043.stdout:372: read oid 26 snap 39 2026-02-20T11:17:24.972 INFO:tasks.rados.rados.0.trial043.stdout:372: expect (ObjNum 143 snap 25 seq_num 143) 2026-02-20T11:17:24.972 INFO:tasks.rados.rados.0.trial043.stdout:373: write (excl) oid 2 current snap is 48 2026-02-20T11:17:24.972 INFO:tasks.rados.rados.0.trial043.stdout:373: seq_num 190 ranges {793330=689782,2035461=617731,3101545=91785} 2026-02-20T11:17:24.976 INFO:tasks.rados.rados.0.trial043.stdout:373: writing trial04342147-2 from 793330 to 1483112 tid 1 2026-02-20T11:17:24.980 INFO:tasks.rados.rados.0.trial043.stdout:373: writing trial04342147-2 from 2035461 to 2653192 tid 2 2026-02-20T11:17:24.983 INFO:tasks.rados.rados.0.trial043.stdout:373: writing trial04342147-2 from 3101545 to 3193330 tid 3 2026-02-20T11:17:24.983 INFO:tasks.rados.rados.0.trial043.stdout:374: write oid 10 current snap is 48 2026-02-20T11:17:24.983 INFO:tasks.rados.rados.0.trial043.stdout:374: seq_num 191 ranges {410137=481384,1410296=761389,2728968=481169} 2026-02-20T11:17:24.986 INFO:tasks.rados.rados.0.trial043.stdout:374: writing trial04342147-10 from 410137 to 891521 tid 1 2026-02-20T11:17:24.991 INFO:tasks.rados.rados.0.trial043.stdout:374: writing trial04342147-10 from 1410296 to 2171685 tid 2 2026-02-20T11:17:24.995 INFO:tasks.rados.rados.0.trial043.stdout:374: writing trial04342147-10 from 2728968 to 3210137 tid 3 2026-02-20T11:17:24.995 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:17:24.995 INFO:tasks.rados.rados.0.trial043.stdout:355: expect (ObjNum 121 snap 16 seq_num 121) 2026-02-20T11:17:25.059 INFO:tasks.rados.rados.0.trial043.stdout:353: finishing write tid 2 to trial04342147-9 2026-02-20T11:17:25.060 INFO:tasks.rados.rados.0.trial043.stdout:353: finishing write tid 3 to trial04342147-9 2026-02-20T11:17:25.060 INFO:tasks.rados.rados.0.trial043.stdout:353: finishing write tid 4 to trial04342147-9 2026-02-20T11:17:25.060 INFO:tasks.rados.rados.0.trial043.stdout:353: oid 9 updating version 0 to 65 2026-02-20T11:17:25.060 INFO:tasks.rados.rados.0.trial043.stdout:353: oid 9 version 65 is already newer than 65 2026-02-20T11:17:25.060 INFO:tasks.rados.rados.0.trial043.stdout:353: oid 9 version 65 is already newer than 63 2026-02-20T11:17:25.060 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 9 v 65 (ObjNum 182 snap 48 seq_num 182) dirty exists 2026-02-20T11:17:25.060 INFO:tasks.rados.rados.0.trial043.stdout:353: left oid 9 (ObjNum 182 snap 48 seq_num 182) 2026-02-20T11:17:25.060 INFO:tasks.rados.rados.0.trial043.stdout:353: done (15 left) 2026-02-20T11:17:25.060 INFO:tasks.rados.rados.0.trial043.stdout:355: done (14 left) 2026-02-20T11:17:25.060 INFO:tasks.rados.rados.0.trial043.stdout:375: snap_create 2026-02-20T11:17:25.060 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 14 2026-02-20T11:17:25.060 INFO:tasks.rados.rados.0.trial043.stdout:360: finishing write tid 1 to trial04342147-49 2026-02-20T11:17:25.060 INFO:tasks.rados.rados.0.trial043.stdout:358: expect (ObjNum 180 snap 48 seq_num 180) 2026-02-20T11:17:25.085 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:17:25.117 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2736173 2026-02-20T11:17:25.124 INFO:tasks.rados.rados.0.trial043.stdout:359: finishing write tid 1 to trial04342147-11 2026-02-20T11:17:25.124 INFO:tasks.rados.rados.0.trial043.stdout:360: finishing write tid 2 to trial04342147-49 2026-02-20T11:17:25.124 INFO:tasks.rados.rados.0.trial043.stdout:358: done (13 left) 2026-02-20T11:17:25.125 INFO:tasks.rados.rados.0.trial043.stdout:361: done (12 left) 2026-02-20T11:17:25.125 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 12 2026-02-20T11:17:25.125 INFO:tasks.rados.rados.0.trial043.stdout:359: finishing write tid 2 to trial04342147-11 2026-02-20T11:17:25.125 INFO:tasks.rados.rados.0.trial043.stdout:360: finishing write tid 3 to trial04342147-49 2026-02-20T11:17:25.125 INFO:tasks.rados.rados.0.trial043.stdout:360: finishing write tid 4 to trial04342147-49 2026-02-20T11:17:25.125 INFO:tasks.rados.rados.0.trial043.stdout:360: oid 49 updating version 0 to 34 2026-02-20T11:17:25.125 INFO:tasks.rados.rados.0.trial043.stdout:360: oid 49 version 34 is already newer than 32 2026-02-20T11:17:25.125 INFO:tasks.rados.rados.0.trial043.stdout:360: oid 49 version 34 is already newer than 34 2026-02-20T11:17:25.125 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 49 v 34 (ObjNum 184 snap 48 seq_num 184) dirty exists 2026-02-20T11:17:25.125 INFO:tasks.rados.rados.0.trial043.stdout:360: left oid 49 (ObjNum 184 snap 48 seq_num 184) 2026-02-20T11:17:25.125 INFO:tasks.rados.rados.0.trial043.stdout:359: finishing write tid 3 to trial04342147-11 2026-02-20T11:17:25.126 INFO:tasks.rados.rados.0.trial043.stdout:360: done (11 left) 2026-02-20T11:17:25.126 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 11 2026-02-20T11:17:25.126 INFO:tasks.rados.rados.0.trial043.stdout:359: finishing write tid 4 to trial04342147-11 2026-02-20T11:17:25.126 INFO:tasks.rados.rados.0.trial043.stdout:359: finishing write tid 5 to trial04342147-11 2026-02-20T11:17:25.126 INFO:tasks.rados.rados.0.trial043.stdout:359: oid 11 updating version 0 to 77 2026-02-20T11:17:25.126 INFO:tasks.rados.rados.0.trial043.stdout:359: oid 11 version 77 is already newer than 77 2026-02-20T11:17:25.126 INFO:tasks.rados.rados.0.trial043.stdout:359: oid 11 version 77 is already newer than 77 2026-02-20T11:17:25.126 INFO:tasks.rados.rados.0.trial043.stdout:359: oid 11 version 77 is already newer than 77 2026-02-20T11:17:25.126 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 11 v 77 (ObjNum 183 snap 48 seq_num 183) dirty exists 2026-02-20T11:17:25.126 INFO:tasks.rados.rados.0.trial043.stdout:359: left oid 11 (ObjNum 183 snap 48 seq_num 183) 2026-02-20T11:17:25.126 INFO:tasks.rados.rados.0.trial043.stdout:363: expect (ObjNum 154 snap 34 seq_num 154) 2026-02-20T11:17:25.186 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1893635 2026-02-20T11:17:25.222 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1787395 2026-02-20T11:17:25.244 INFO:tasks.rados.rados.0.trial043.stdout:364: finishing write tid 1 to trial04342147-3 2026-02-20T11:17:25.244 INFO:tasks.rados.rados.0.trial043.stdout:364: finishing write tid 2 to trial04342147-3 2026-02-20T11:17:25.244 INFO:tasks.rados.rados.0.trial043.stdout:359: done (10 left) 2026-02-20T11:17:25.244 INFO:tasks.rados.rados.0.trial043.stdout:363: done (9 left) 2026-02-20T11:17:25.244 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 9 2026-02-20T11:17:25.245 INFO:tasks.rados.rados.0.trial043.stdout:364: finishing write tid 3 to trial04342147-3 2026-02-20T11:17:25.245 INFO:tasks.rados.rados.0.trial043.stdout:364: finishing write tid 4 to trial04342147-3 2026-02-20T11:17:25.245 INFO:tasks.rados.rados.0.trial043.stdout:364: oid 3 updating version 0 to 111 2026-02-20T11:17:25.245 INFO:tasks.rados.rados.0.trial043.stdout:364: oid 3 version 111 is already newer than 111 2026-02-20T11:17:25.245 INFO:tasks.rados.rados.0.trial043.stdout:364: oid 3 version 111 is already newer than 111 2026-02-20T11:17:25.245 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 3 v 111 (ObjNum 185 snap 48 seq_num 185) dirty exists 2026-02-20T11:17:25.245 INFO:tasks.rados.rados.0.trial043.stdout:364: left oid 3 (ObjNum 185 snap 48 seq_num 185) 2026-02-20T11:17:25.245 INFO:tasks.rados.rados.0.trial043.stdout:364: done (8 left) 2026-02-20T11:17:25.245 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 8 2026-02-20T11:17:25.245 INFO:tasks.rados.rados.0.trial043.stdout:368: finishing write tid 1 to trial04342147-13 2026-02-20T11:17:25.245 INFO:tasks.rados.rados.0.trial043.stdout:367: finishing write tid 1 to trial04342147-24 2026-02-20T11:17:25.245 INFO:tasks.rados.rados.0.trial043.stdout:366: done (7 left) 2026-02-20T11:17:25.246 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 7 2026-02-20T11:17:25.246 INFO:tasks.rados.rados.0.trial043.stdout:368: finishing write tid 2 to trial04342147-13 2026-02-20T11:17:25.246 INFO:tasks.rados.rados.0.trial043.stdout:367: finishing write tid 2 to trial04342147-24 2026-02-20T11:17:25.246 INFO:tasks.rados.rados.0.trial043.stdout:367: finishing write tid 3 to trial04342147-24 2026-02-20T11:17:25.246 INFO:tasks.rados.rados.0.trial043.stdout:368: finishing write tid 3 to trial04342147-13 2026-02-20T11:17:25.246 INFO:tasks.rados.rados.0.trial043.stdout:368: finishing write tid 4 to trial04342147-13 2026-02-20T11:17:25.246 INFO:tasks.rados.rados.0.trial043.stdout:368: oid 13 updating version 0 to 114 2026-02-20T11:17:25.246 INFO:tasks.rados.rados.0.trial043.stdout:368: oid 13 version 114 is already newer than 114 2026-02-20T11:17:25.246 INFO:tasks.rados.rados.0.trial043.stdout:368: oid 13 version 114 is already newer than 114 2026-02-20T11:17:25.246 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 13 v 114 (ObjNum 187 snap 48 seq_num 187) dirty exists 2026-02-20T11:17:25.246 INFO:tasks.rados.rados.0.trial043.stdout:368: left oid 13 (ObjNum 187 snap 48 seq_num 187) 2026-02-20T11:17:25.246 INFO:tasks.rados.rados.0.trial043.stdout:367: finishing write tid 4 to trial04342147-24 2026-02-20T11:17:25.246 INFO:tasks.rados.rados.0.trial043.stdout:368: done (6 left) 2026-02-20T11:17:25.247 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 6 2026-02-20T11:17:25.247 INFO:tasks.rados.rados.0.trial043.stdout:367: finishing write tid 5 to trial04342147-24 2026-02-20T11:17:25.247 INFO:tasks.rados.rados.0.trial043.stdout:367: oid 24 updating version 0 to 34 2026-02-20T11:17:25.247 INFO:tasks.rados.rados.0.trial043.stdout:367: oid 24 version 34 is already newer than 31 2026-02-20T11:17:25.247 INFO:tasks.rados.rados.0.trial043.stdout:367: oid 24 version 34 is already newer than 34 2026-02-20T11:17:25.247 INFO:tasks.rados.rados.0.trial043.stdout:367: oid 24 version 34 is already newer than 34 2026-02-20T11:17:25.247 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 24 v 34 (ObjNum 186 snap 48 seq_num 186) dirty exists 2026-02-20T11:17:25.247 INFO:tasks.rados.rados.0.trial043.stdout:367: left oid 24 (ObjNum 186 snap 48 seq_num 186) 2026-02-20T11:17:25.247 INFO:tasks.rados.rados.0.trial043.stdout:369: finishing write tid 1 to trial04342147-1 2026-02-20T11:17:25.247 INFO:tasks.rados.rados.0.trial043.stdout:369: finishing write tid 2 to trial04342147-1 2026-02-20T11:17:25.247 INFO:tasks.rados.rados.0.trial043.stdout:369: finishing write tid 3 to trial04342147-1 2026-02-20T11:17:25.247 INFO:tasks.rados.rados.0.trial043.stdout:369: finishing write tid 4 to trial04342147-1 2026-02-20T11:17:25.247 INFO:tasks.rados.rados.0.trial043.stdout:369: oid 1 updating version 0 to 39 2026-02-20T11:17:25.248 INFO:tasks.rados.rados.0.trial043.stdout:369: oid 1 version 39 is already newer than 39 2026-02-20T11:17:25.248 INFO:tasks.rados.rados.0.trial043.stdout:369: oid 1 version 39 is already newer than 39 2026-02-20T11:17:25.248 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 1 v 39 (ObjNum 188 snap 48 seq_num 188) dirty exists 2026-02-20T11:17:25.248 INFO:tasks.rados.rados.0.trial043.stdout:369: left oid 1 (ObjNum 188 snap 48 seq_num 188) 2026-02-20T11:17:25.248 INFO:tasks.rados.rados.0.trial043.stdout:367: done (5 left) 2026-02-20T11:17:25.248 INFO:tasks.rados.rados.0.trial043.stdout:369: done (4 left) 2026-02-20T11:17:25.248 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 4 2026-02-20T11:17:25.248 INFO:tasks.rados.rados.0.trial043.stdout:371: expect (ObjNum 90 snap 10 seq_num 90) 2026-02-20T11:17:25.299 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 3105804 2026-02-20T11:17:25.345 INFO:tasks.rados.rados.0.trial043.stdout:372: expect (ObjNum 143 snap 25 seq_num 143) 2026-02-20T11:17:25.367 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2397394 2026-02-20T11:17:25.403 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:17:25.441 INFO:tasks.rados.rados.0.trial043.stdout:374: finishing write tid 1 to trial04342147-10 2026-02-20T11:17:25.441 INFO:tasks.rados.rados.0.trial043.stdout:373: finishing write tid 1 to trial04342147-2 2026-02-20T11:17:25.441 INFO:tasks.rados.rados.0.trial043.stdout:374: finishing write tid 2 to trial04342147-10 2026-02-20T11:17:25.441 INFO:tasks.rados.rados.0.trial043.stdout:374: finishing write tid 3 to trial04342147-10 2026-02-20T11:17:25.441 INFO:tasks.rados.rados.0.trial043.stdout:374: finishing write tid 4 to trial04342147-10 2026-02-20T11:17:25.441 INFO:tasks.rados.rados.0.trial043.stdout:371: done (3 left) 2026-02-20T11:17:25.441 INFO:tasks.rados.rados.0.trial043.stdout:372: done (2 left) 2026-02-20T11:17:25.441 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:17:25.441 INFO:tasks.rados.rados.0.trial043.stdout:373: finishing write tid 2 to trial04342147-2 2026-02-20T11:17:25.442 INFO:tasks.rados.rados.0.trial043.stdout:374: finishing write tid 5 to trial04342147-10 2026-02-20T11:17:25.442 INFO:tasks.rados.rados.0.trial043.stdout:374: oid 10 updating version 0 to 49 2026-02-20T11:17:25.442 INFO:tasks.rados.rados.0.trial043.stdout:374: oid 10 version 49 is already newer than 49 2026-02-20T11:17:25.442 INFO:tasks.rados.rados.0.trial043.stdout:374: oid 10 version 49 is already newer than 49 2026-02-20T11:17:25.442 INFO:tasks.rados.rados.0.trial043.stdout:374: oid 10 version 49 is already newer than 49 2026-02-20T11:17:25.442 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 10 v 49 (ObjNum 191 snap 48 seq_num 191) dirty exists 2026-02-20T11:17:25.442 INFO:tasks.rados.rados.0.trial043.stdout:374: left oid 10 (ObjNum 191 snap 48 seq_num 191) 2026-02-20T11:17:25.442 INFO:tasks.rados.rados.0.trial043.stdout:373: finishing write tid 3 to trial04342147-2 2026-02-20T11:17:25.442 INFO:tasks.rados.rados.0.trial043.stdout:374: done (1 left) 2026-02-20T11:17:25.442 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:17:25.442 INFO:tasks.rados.rados.0.trial043.stdout:373: finishing write tid 4 to trial04342147-2 2026-02-20T11:17:25.443 INFO:tasks.rados.rados.0.trial043.stdout:373: finishing write tid 5 to trial04342147-2 2026-02-20T11:17:25.443 INFO:tasks.rados.rados.0.trial043.stdout:373: oid 2 updating version 0 to 55 2026-02-20T11:17:25.443 INFO:tasks.rados.rados.0.trial043.stdout:373: oid 2 version 55 is already newer than 55 2026-02-20T11:17:25.443 INFO:tasks.rados.rados.0.trial043.stdout:373: oid 2 version 55 is already newer than 55 2026-02-20T11:17:25.443 INFO:tasks.rados.rados.0.trial043.stdout:373: oid 2 version 55 is already newer than 52 2026-02-20T11:17:25.443 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 2 v 55 (ObjNum 190 snap 48 seq_num 190) dirty exists 2026-02-20T11:17:25.443 INFO:tasks.rados.rados.0.trial043.stdout:373: left oid 2 (ObjNum 190 snap 48 seq_num 190) 2026-02-20T11:17:25.443 INFO:tasks.rados.rados.0.trial043.stdout:373: done (0 left) 2026-02-20T11:17:25.577 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:17:25.654 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:25.652+0000 7f6544b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:25.656 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:25.655+0000 7f6544b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:25.682 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:17:25.697 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:17:25.698 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:17:25.880 INFO:tasks.rados.rados.0.trial043.stdout:375: done (0 left) 2026-02-20T11:17:25.881 INFO:tasks.rados.rados.0.trial043.stdout:376: snap_remove snap 40 2026-02-20T11:17:26.886 INFO:tasks.rados.rados.0.trial043.stdout:376: done (0 left) 2026-02-20T11:17:26.887 INFO:tasks.rados.rados.0.trial043.stdout:377: snap_create 2026-02-20T11:17:27.507 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:17:27.579 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:17:27.582 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:17:27.583 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:17:27.699 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:17:27.776 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:27.774+0000 7f5591011640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:27.778 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:27.778+0000 7f5591011640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:27.804 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:17:27.822 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:17:27.822 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:17:27.887 INFO:tasks.rados.rados.0.trial043.stdout:377: done (0 left) 2026-02-20T11:17:27.887 INFO:tasks.rados.rados.0.trial043.stdout:378: read oid 32 snap -1 2026-02-20T11:17:27.888 INFO:tasks.rados.rados.0.trial043.stdout:378: expect deleted 2026-02-20T11:17:27.888 INFO:tasks.rados.rados.0.trial043.stdout:379: read oid 30 snap -1 2026-02-20T11:17:27.888 INFO:tasks.rados.rados.0.trial043.stdout:379: expect deleted 2026-02-20T11:17:27.888 INFO:tasks.rados.rados.0.trial043.stdout:380: delete oid 17 current snap is 50 2026-02-20T11:17:27.931 INFO:tasks.rados.rados.0.trial043.stdout:378: done (2 left) 2026-02-20T11:17:27.931 INFO:tasks.rados.rados.0.trial043.stdout:379: done (1 left) 2026-02-20T11:17:27.931 INFO:tasks.rados.rados.0.trial043.stdout:380: done (0 left) 2026-02-20T11:17:27.931 INFO:tasks.rados.rados.0.trial043.stdout:381: snap_create 2026-02-20T11:17:28.750 INFO:tasks.rados.rados.0.trial043.stdout:381: done (0 left) 2026-02-20T11:17:28.750 INFO:tasks.rados.rados.0.trial043.stdout:382: write (excl) oid 44 current snap is 51 2026-02-20T11:17:28.750 INFO:tasks.rados.rados.0.trial043.stdout:382: seq_num 193 ranges {443751=631882,1810907=765058,3067461=176290} 2026-02-20T11:17:28.754 INFO:tasks.rados.rados.0.trial043.stdout:382: writing trial04342147-44 from 443751 to 1075633 tid 1 2026-02-20T11:17:28.762 INFO:tasks.rados.rados.0.trial043.stdout:382: writing trial04342147-44 from 1810907 to 2575965 tid 2 2026-02-20T11:17:28.764 INFO:tasks.rados.rados.0.trial043.stdout:382: writing trial04342147-44 from 3067461 to 3243751 tid 3 2026-02-20T11:17:28.765 INFO:tasks.rados.rados.0.trial043.stdout:383: write oid 49 current snap is 51 2026-02-20T11:17:28.765 INFO:tasks.rados.rados.0.trial043.stdout:383: seq_num 194 ranges {460558=707131,1869389=508716,3094884=165674} 2026-02-20T11:17:28.768 INFO:tasks.rados.rados.0.trial043.stdout:383: writing trial04342147-49 from 460558 to 1167689 tid 1 2026-02-20T11:17:28.773 INFO:tasks.rados.rados.0.trial043.stdout:383: writing trial04342147-49 from 1869389 to 2378105 tid 2 2026-02-20T11:17:28.776 INFO:tasks.rados.rados.0.trial043.stdout:383: writing trial04342147-49 from 3094884 to 3260558 tid 3 2026-02-20T11:17:28.777 INFO:tasks.rados.rados.0.trial043.stdout:384: read oid 17 snap -1 2026-02-20T11:17:28.777 INFO:tasks.rados.rados.0.trial043.stdout:384: expect deleted 2026-02-20T11:17:28.777 INFO:tasks.rados.rados.0.trial043.stdout:385: snap_create 2026-02-20T11:17:28.777 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:17:28.782 INFO:tasks.rados.rados.0.trial043.stdout:384: done (2 left) 2026-02-20T11:17:28.783 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:17:28.788 INFO:tasks.rados.rados.0.trial043.stdout:382: finishing write tid 1 to trial04342147-44 2026-02-20T11:17:28.802 INFO:tasks.rados.rados.0.trial043.stdout:383: finishing write tid 1 to trial04342147-49 2026-02-20T11:17:28.803 INFO:tasks.rados.rados.0.trial043.stdout:382: finishing write tid 2 to trial04342147-44 2026-02-20T11:17:28.809 INFO:tasks.rados.rados.0.trial043.stdout:383: finishing write tid 2 to trial04342147-49 2026-02-20T11:17:28.810 INFO:tasks.rados.rados.0.trial043.stdout:382: finishing write tid 3 to trial04342147-44 2026-02-20T11:17:28.814 INFO:tasks.rados.rados.0.trial043.stdout:383: finishing write tid 3 to trial04342147-49 2026-02-20T11:17:28.814 INFO:tasks.rados.rados.0.trial043.stdout:382: finishing write tid 4 to trial04342147-44 2026-02-20T11:17:28.817 INFO:tasks.rados.rados.0.trial043.stdout:382: finishing write tid 5 to trial04342147-44 2026-02-20T11:17:28.817 INFO:tasks.rados.rados.0.trial043.stdout:382: oid 44 updating version 0 to 60 2026-02-20T11:17:28.817 INFO:tasks.rados.rados.0.trial043.stdout:382: oid 44 version 60 is already newer than 60 2026-02-20T11:17:28.817 INFO:tasks.rados.rados.0.trial043.stdout:382: oid 44 version 60 is already newer than 60 2026-02-20T11:17:28.817 INFO:tasks.rados.rados.0.trial043.stdout:382: oid 44 version 60 is already newer than 57 2026-02-20T11:17:28.817 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 44 v 60 (ObjNum 193 snap 51 seq_num 193) dirty exists 2026-02-20T11:17:28.817 INFO:tasks.rados.rados.0.trial043.stdout:382: left oid 44 (ObjNum 193 snap 51 seq_num 193) 2026-02-20T11:17:28.817 INFO:tasks.rados.rados.0.trial043.stdout:382: done (1 left) 2026-02-20T11:17:28.817 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:17:28.818 INFO:tasks.rados.rados.0.trial043.stdout:383: finishing write tid 4 to trial04342147-49 2026-02-20T11:17:28.819 INFO:tasks.rados.rados.0.trial043.stdout:383: finishing write tid 5 to trial04342147-49 2026-02-20T11:17:28.819 INFO:tasks.rados.rados.0.trial043.stdout:383: oid 49 updating version 0 to 39 2026-02-20T11:17:28.819 INFO:tasks.rados.rados.0.trial043.stdout:383: oid 49 version 39 is already newer than 39 2026-02-20T11:17:28.819 INFO:tasks.rados.rados.0.trial043.stdout:383: oid 49 version 39 is already newer than 39 2026-02-20T11:17:28.819 INFO:tasks.rados.rados.0.trial043.stdout:383: oid 49 version 39 is already newer than 39 2026-02-20T11:17:28.819 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 49 v 39 (ObjNum 194 snap 51 seq_num 194) dirty exists 2026-02-20T11:17:28.819 INFO:tasks.rados.rados.0.trial043.stdout:383: left oid 49 (ObjNum 194 snap 51 seq_num 194) 2026-02-20T11:17:28.819 INFO:tasks.rados.rados.0.trial043.stdout:383: done (0 left) 2026-02-20T11:17:29.750 INFO:tasks.rados.rados.0.trial043.stdout:385: done (0 left) 2026-02-20T11:17:29.750 INFO:tasks.rados.rados.0.trial043.stdout:386: write (excl) oid 36 current snap is 52 2026-02-20T11:17:29.750 INFO:tasks.rados.rados.0.trial043.stdout:386: seq_num 195 ranges {477365=782380,1927871=768727,3122307=155058} 2026-02-20T11:17:29.755 INFO:tasks.rados.rados.0.trial043.stdout:386: writing trial04342147-36 from 477365 to 1259745 tid 1 2026-02-20T11:17:29.761 INFO:tasks.rados.rados.0.trial043.stdout:386: writing trial04342147-36 from 1927871 to 2696598 tid 2 2026-02-20T11:17:29.763 INFO:tasks.rados.rados.0.trial043.stdout:386: writing trial04342147-36 from 3122307 to 3277365 tid 3 2026-02-20T11:17:29.764 INFO:tasks.rados.rados.0.trial043.stdout:387: read oid 12 snap -1 2026-02-20T11:17:29.764 INFO:tasks.rados.rados.0.trial043.stdout:387: expect deleted 2026-02-20T11:17:29.764 INFO:tasks.rados.rados.0.trial043.stdout:388: write (excl) oid 28 current snap is 52 2026-02-20T11:17:29.764 INFO:tasks.rados.rados.0.trial043.stdout:388: seq_num 196 ranges {494172=457629,1586353=512385,2749730=544442} 2026-02-20T11:17:29.767 INFO:tasks.rados.rados.0.trial043.stdout:388: writing trial04342147-28 from 494172 to 951801 tid 1 2026-02-20T11:17:29.771 INFO:tasks.rados.rados.0.trial043.stdout:388: writing trial04342147-28 from 1586353 to 2098738 tid 2 2026-02-20T11:17:29.776 INFO:tasks.rados.rados.0.trial043.stdout:388: writing trial04342147-28 from 2749730 to 3294172 tid 3 2026-02-20T11:17:29.776 INFO:tasks.rados.rados.0.trial043.stdout:389: write (excl) oid 50 current snap is 52 2026-02-20T11:17:29.777 INFO:tasks.rados.rados.0.trial043.stdout:389: seq_num 197 ranges {510979=532878,1528482=772396,3060800=250179} 2026-02-20T11:17:29.780 INFO:tasks.rados.rados.0.trial043.stdout:389: writing trial04342147-50 from 510979 to 1043857 tid 1 2026-02-20T11:17:29.785 INFO:tasks.rados.rados.0.trial043.stdout:389: writing trial04342147-50 from 1528482 to 2300878 tid 2 2026-02-20T11:17:29.789 INFO:tasks.rados.rados.0.trial043.stdout:389: writing trial04342147-50 from 3060800 to 3310979 tid 3 2026-02-20T11:17:29.789 INFO:tasks.rados.rados.0.trial043.stdout:390: snap_create 2026-02-20T11:17:29.789 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 4 2026-02-20T11:17:29.789 INFO:tasks.rados.rados.0.trial043.stdout:388: finishing write tid 1 to trial04342147-28 2026-02-20T11:17:29.789 INFO:tasks.rados.rados.0.trial043.stdout:387: done (3 left) 2026-02-20T11:17:29.789 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:17:29.807 INFO:tasks.rados.rados.0.trial043.stdout:386: finishing write tid 1 to trial04342147-36 2026-02-20T11:17:29.809 INFO:tasks.rados.rados.0.trial043.stdout:388: finishing write tid 2 to trial04342147-28 2026-02-20T11:17:29.818 INFO:tasks.rados.rados.0.trial043.stdout:389: finishing write tid 1 to trial04342147-50 2026-02-20T11:17:29.823 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:17:29.823 INFO:tasks.rados.rados.0.trial043.stdout:388: finishing write tid 3 to trial04342147-28 2026-02-20T11:17:29.829 INFO:tasks.rados.rados.0.trial043.stdout:386: finishing write tid 2 to trial04342147-36 2026-02-20T11:17:29.830 INFO:tasks.rados.rados.0.trial043.stdout:388: finishing write tid 4 to trial04342147-28 2026-02-20T11:17:29.833 INFO:tasks.rados.rados.0.trial043.stdout:388: finishing write tid 5 to trial04342147-28 2026-02-20T11:17:29.833 INFO:tasks.rados.rados.0.trial043.stdout:388: oid 28 updating version 0 to 40 2026-02-20T11:17:29.833 INFO:tasks.rados.rados.0.trial043.stdout:388: oid 28 updating version 40 to 43 2026-02-20T11:17:29.833 INFO:tasks.rados.rados.0.trial043.stdout:388: oid 28 version 43 is already newer than 43 2026-02-20T11:17:29.833 INFO:tasks.rados.rados.0.trial043.stdout:388: oid 28 version 43 is already newer than 43 2026-02-20T11:17:29.833 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 28 v 43 (ObjNum 196 snap 52 seq_num 196) dirty exists 2026-02-20T11:17:29.833 INFO:tasks.rados.rados.0.trial043.stdout:388: left oid 28 (ObjNum 196 snap 52 seq_num 196) 2026-02-20T11:17:29.833 INFO:tasks.rados.rados.0.trial043.stdout:388: done (2 left) 2026-02-20T11:17:29.833 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:17:29.835 INFO:tasks.rados.rados.0.trial043.stdout:389: finishing write tid 2 to trial04342147-50 2026-02-20T11:17:29.839 INFO:tasks.rados.rados.0.trial043.stdout:386: finishing write tid 3 to trial04342147-36 2026-02-20T11:17:29.841 INFO:tasks.rados.rados.0.trial043.stdout:389: finishing write tid 3 to trial04342147-50 2026-02-20T11:17:29.845 INFO:tasks.rados.rados.0.trial043.stdout:389: finishing write tid 4 to trial04342147-50 2026-02-20T11:17:29.845 INFO:tasks.rados.rados.0.trial043.stdout:386: finishing write tid 4 to trial04342147-36 2026-02-20T11:17:29.846 INFO:tasks.rados.rados.0.trial043.stdout:389: finishing write tid 5 to trial04342147-50 2026-02-20T11:17:29.846 INFO:tasks.rados.rados.0.trial043.stdout:389: oid 50 updating version 0 to 41 2026-02-20T11:17:29.846 INFO:tasks.rados.rados.0.trial043.stdout:389: oid 50 updating version 41 to 44 2026-02-20T11:17:29.846 INFO:tasks.rados.rados.0.trial043.stdout:389: oid 50 version 44 is already newer than 44 2026-02-20T11:17:29.846 INFO:tasks.rados.rados.0.trial043.stdout:389: oid 50 version 44 is already newer than 44 2026-02-20T11:17:29.846 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 50 v 44 (ObjNum 197 snap 52 seq_num 197) dirty exists 2026-02-20T11:17:29.846 INFO:tasks.rados.rados.0.trial043.stdout:389: left oid 50 (ObjNum 197 snap 52 seq_num 197) 2026-02-20T11:17:29.846 INFO:tasks.rados.rados.0.trial043.stdout:386: finishing write tid 5 to trial04342147-36 2026-02-20T11:17:29.846 INFO:tasks.rados.rados.0.trial043.stdout:386: oid 36 updating version 0 to 28 2026-02-20T11:17:29.847 INFO:tasks.rados.rados.0.trial043.stdout:386: oid 36 updating version 28 to 31 2026-02-20T11:17:29.847 INFO:tasks.rados.rados.0.trial043.stdout:386: oid 36 version 31 is already newer than 31 2026-02-20T11:17:29.847 INFO:tasks.rados.rados.0.trial043.stdout:386: oid 36 version 31 is already newer than 31 2026-02-20T11:17:29.847 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 36 v 31 (ObjNum 195 snap 52 seq_num 195) dirty exists 2026-02-20T11:17:29.847 INFO:tasks.rados.rados.0.trial043.stdout:386: left oid 36 (ObjNum 195 snap 52 seq_num 195) 2026-02-20T11:17:29.847 INFO:tasks.rados.rados.0.trial043.stdout:386: done (1 left) 2026-02-20T11:17:29.847 INFO:tasks.rados.rados.0.trial043.stdout:389: done (0 left) 2026-02-20T11:17:29.897 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:29.894+0000 7fdda3d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:29.899 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:29.898+0000 7fdda3d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:29.908 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~105s 2026-02-20T11:17:29.924 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:17:29.940 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:17:29.941 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:17:30.755 INFO:tasks.rados.rados.0.trial043.stdout:390: done (0 left) 2026-02-20T11:17:30.756 INFO:tasks.rados.rados.0.trial043.stdout:391: write oid 18 current snap is 53 2026-02-20T11:17:30.756 INFO:tasks.rados.rados.0.trial043.stdout:391: seq_num 198 ranges {527786=724480,1703317=516054,2804576=435035} 2026-02-20T11:17:30.760 INFO:tasks.rados.rados.0.trial043.stdout:391: writing trial04342147-18 from 527786 to 1252266 tid 1 2026-02-20T11:17:30.764 INFO:tasks.rados.rados.0.trial043.stdout:391: writing trial04342147-18 from 1703317 to 2219371 tid 2 2026-02-20T11:17:30.768 INFO:tasks.rados.rados.0.trial043.stdout:391: writing trial04342147-18 from 2804576 to 3239611 tid 3 2026-02-20T11:17:30.768 INFO:tasks.rados.rados.0.trial043.stdout:392: read oid 47 snap -1 2026-02-20T11:17:30.768 INFO:tasks.rados.rados.0.trial043.stdout:392: expect (ObjNum 119 snap 14 seq_num 119) 2026-02-20T11:17:30.768 INFO:tasks.rados.rados.0.trial043.stdout:393: write (excl) oid 14 current snap is 53 2026-02-20T11:17:30.768 INFO:tasks.rados.rados.0.trial043.stdout:393: seq_num 199 ranges {544593=799729,1761799=776065,2948352=396241} 2026-02-20T11:17:30.773 INFO:tasks.rados.rados.0.trial043.stdout:393: writing trial04342147-14 from 544593 to 1344322 tid 1 2026-02-20T11:17:30.778 INFO:tasks.rados.rados.0.trial043.stdout:393: writing trial04342147-14 from 1761799 to 2537864 tid 2 2026-02-20T11:17:30.781 INFO:tasks.rados.rados.0.trial043.stdout:393: writing trial04342147-14 from 2948352 to 3344593 tid 3 2026-02-20T11:17:30.781 INFO:tasks.rados.rados.0.trial043.stdout:394: delete oid 44 current snap is 53 2026-02-20T11:17:30.798 INFO:tasks.rados.rados.0.trial043.stdout:392: expect (ObjNum 119 snap 14 seq_num 119) 2026-02-20T11:17:30.816 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1146228 2026-02-20T11:17:30.830 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1024540 2026-02-20T11:17:30.839 INFO:tasks.rados.rados.0.trial043.stdout:391: finishing write tid 1 to trial04342147-18 2026-02-20T11:17:30.839 INFO:tasks.rados.rados.0.trial043.stdout:391: finishing write tid 2 to trial04342147-18 2026-02-20T11:17:30.839 INFO:tasks.rados.rados.0.trial043.stdout:391: finishing write tid 3 to trial04342147-18 2026-02-20T11:17:30.839 INFO:tasks.rados.rados.0.trial043.stdout:393: finishing write tid 1 to trial04342147-14 2026-02-20T11:17:30.840 INFO:tasks.rados.rados.0.trial043.stdout:391: finishing write tid 4 to trial04342147-18 2026-02-20T11:17:30.840 INFO:tasks.rados.rados.0.trial043.stdout:391: finishing write tid 5 to trial04342147-18 2026-02-20T11:17:30.840 INFO:tasks.rados.rados.0.trial043.stdout:391: oid 18 updating version 0 to 33 2026-02-20T11:17:30.840 INFO:tasks.rados.rados.0.trial043.stdout:391: oid 18 version 33 is already newer than 33 2026-02-20T11:17:30.840 INFO:tasks.rados.rados.0.trial043.stdout:391: oid 18 version 33 is already newer than 33 2026-02-20T11:17:30.840 INFO:tasks.rados.rados.0.trial043.stdout:391: oid 18 version 33 is already newer than 33 2026-02-20T11:17:30.841 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 18 v 33 (ObjNum 198 snap 53 seq_num 198) dirty exists 2026-02-20T11:17:30.841 INFO:tasks.rados.rados.0.trial043.stdout:391: left oid 18 (ObjNum 198 snap 53 seq_num 198) 2026-02-20T11:17:30.850 INFO:tasks.rados.rados.0.trial043.stdout:393: finishing write tid 2 to trial04342147-14 2026-02-20T11:17:30.862 INFO:tasks.rados.rados.0.trial043.stdout:393: finishing write tid 3 to trial04342147-14 2026-02-20T11:17:30.868 INFO:tasks.rados.rados.0.trial043.stdout:393: finishing write tid 4 to trial04342147-14 2026-02-20T11:17:30.871 INFO:tasks.rados.rados.0.trial043.stdout:393: finishing write tid 5 to trial04342147-14 2026-02-20T11:17:30.871 INFO:tasks.rados.rados.0.trial043.stdout:393: oid 14 updating version 0 to 100 2026-02-20T11:17:30.871 INFO:tasks.rados.rados.0.trial043.stdout:393: oid 14 version 100 is already newer than 100 2026-02-20T11:17:30.871 INFO:tasks.rados.rados.0.trial043.stdout:393: oid 14 version 100 is already newer than 97 2026-02-20T11:17:30.871 INFO:tasks.rados.rados.0.trial043.stdout:393: oid 14 version 100 is already newer than 100 2026-02-20T11:17:30.871 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 14 v 100 (ObjNum 199 snap 53 seq_num 199) dirty exists 2026-02-20T11:17:30.871 INFO:tasks.rados.rados.0.trial043.stdout:393: left oid 14 (ObjNum 199 snap 53 seq_num 199) 2026-02-20T11:17:30.882 INFO:tasks.rados.rados.0.trial043.stdout:391: done (3 left) 2026-02-20T11:17:30.882 INFO:tasks.rados.rados.0.trial043.stdout:392: done (2 left) 2026-02-20T11:17:30.882 INFO:tasks.rados.rados.0.trial043.stdout:393: done (1 left) 2026-02-20T11:17:30.882 INFO:tasks.rados.rados.0.trial043.stdout:394: done (0 left) 2026-02-20T11:17:30.882 INFO:tasks.rados.rados.0.trial043.stdout:395: read oid 40 snap 39 2026-02-20T11:17:30.882 INFO:tasks.rados.rados.0.trial043.stdout:395: expect (ObjNum 103 snap 13 seq_num 103) 2026-02-20T11:17:30.882 INFO:tasks.rados.rados.0.trial043.stdout:396: read oid 23 snap 52 2026-02-20T11:17:30.883 INFO:tasks.rados.rados.0.trial043.stdout:396: expect (ObjNum 165 snap 38 seq_num 165) 2026-02-20T11:17:30.883 INFO:tasks.rados.rados.0.trial043.stdout:397: snap_create 2026-02-20T11:17:30.883 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:17:30.898 INFO:tasks.rados.rados.0.trial043.stdout:395: expect (ObjNum 103 snap 13 seq_num 103) 2026-02-20T11:17:30.898 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:17:30.984 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2581042 2026-02-20T11:17:31.024 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1483850 2026-02-20T11:17:31.064 INFO:tasks.rados.rados.0.trial043.stdout:395: done (1 left) 2026-02-20T11:17:31.064 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:17:31.065 INFO:tasks.rados.rados.0.trial043.stdout:396: expect (ObjNum 165 snap 38 seq_num 165) 2026-02-20T11:17:31.068 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 765113 2026-02-20T11:17:31.141 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:17:31.184 INFO:tasks.rados.rados.0.trial043.stdout:396: done (0 left) 2026-02-20T11:17:31.757 INFO:tasks.rados.rados.0.trial043.stdout:397: done (0 left) 2026-02-20T11:17:31.757 INFO:tasks.rados.rados.0.trial043.stdout:398: snap_create 2026-02-20T11:17:31.942 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:17:32.020 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:32.018+0000 7fa09a8cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:32.022 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:32.022+0000 7fa09a8cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:32.048 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:17:32.065 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:17:32.066 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:17:32.583 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:17:32.654 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:17:32.656 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:17:32.657 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:17:32.761 INFO:tasks.rados.rados.0.trial043.stdout:398: done (0 left) 2026-02-20T11:17:32.761 INFO:tasks.rados.rados.0.trial043.stdout:399: write (excl) oid 33 current snap is 55 2026-02-20T11:17:32.762 INFO:tasks.rados.rados.0.trial043.stdout:399: seq_num 201 ranges {578207=550227,1762410=779734,3286845=91362} 2026-02-20T11:17:32.766 INFO:tasks.rados.rados.0.trial043.stdout:399: writing trial04342147-33 from 578207 to 1128434 tid 1 2026-02-20T11:17:32.772 INFO:tasks.rados.rados.0.trial043.stdout:399: writing trial04342147-33 from 1762410 to 2542144 tid 2 2026-02-20T11:17:32.775 INFO:tasks.rados.rados.0.trial043.stdout:399: writing trial04342147-33 from 3286845 to 3378207 tid 3 2026-02-20T11:17:32.775 INFO:tasks.rados.rados.0.trial043.stdout:400: write oid 9 current snap is 55 2026-02-20T11:17:32.775 INFO:tasks.rados.rados.0.trial043.stdout:400: seq_num 202 ranges {595014=625476,1820892=523392,2797915=596476} 2026-02-20T11:17:32.779 INFO:tasks.rados.rados.0.trial043.stdout:400: writing trial04342147-9 from 595014 to 1220490 tid 1 2026-02-20T11:17:32.784 INFO:tasks.rados.rados.0.trial043.stdout:400: writing trial04342147-9 from 1820892 to 2344284 tid 2 2026-02-20T11:17:32.788 INFO:tasks.rados.rados.0.trial043.stdout:400: writing trial04342147-9 from 2797915 to 3394391 tid 3 2026-02-20T11:17:32.789 INFO:tasks.rados.rados.0.trial043.stdout:401: write (excl) oid 10 current snap is 55 2026-02-20T11:17:32.789 INFO:tasks.rados.rados.0.trial043.stdout:401: seq_num 203 ranges {611821=700725,1879374=783403,3341691=70130} 2026-02-20T11:17:32.794 INFO:tasks.rados.rados.0.trial043.stdout:401: writing trial04342147-10 from 611821 to 1312546 tid 1 2026-02-20T11:17:32.799 INFO:tasks.rados.rados.0.trial043.stdout:401: writing trial04342147-10 from 1879374 to 2662777 tid 2 2026-02-20T11:17:32.802 INFO:tasks.rados.rados.0.trial043.stdout:401: writing trial04342147-10 from 3341691 to 3411821 tid 3 2026-02-20T11:17:32.802 INFO:tasks.rados.rados.0.trial043.stdout:402: write (excl) oid 18 current snap is 55 2026-02-20T11:17:32.802 INFO:tasks.rados.rados.0.trial043.stdout:402: seq_num 204 ranges {628628=775974,1937856=527061,3252761=175867} 2026-02-20T11:17:32.807 INFO:tasks.rados.rados.0.trial043.stdout:402: writing trial04342147-18 from 628628 to 1404602 tid 1 2026-02-20T11:17:32.811 INFO:tasks.rados.rados.0.trial043.stdout:402: writing trial04342147-18 from 1937856 to 2464917 tid 2 2026-02-20T11:17:32.815 INFO:tasks.rados.rados.0.trial043.stdout:402: writing trial04342147-18 from 3252761 to 3428628 tid 3 2026-02-20T11:17:32.815 INFO:tasks.rados.rados.0.trial043.stdout:403: snap_remove snap 49 2026-02-20T11:17:33.765 INFO:tasks.rados.rados.0.trial043.stdout:403: done (4 left) 2026-02-20T11:17:33.765 INFO:tasks.rados.rados.0.trial043.stdout:404: read oid 30 snap 25 2026-02-20T11:17:33.765 INFO:tasks.rados.rados.0.trial043.stdout:404: expect deleted 2026-02-20T11:17:33.765 INFO:tasks.rados.rados.0.trial043.stdout:405: snap_create 2026-02-20T11:17:33.765 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 5 2026-02-20T11:17:33.765 INFO:tasks.rados.rados.0.trial043.stdout:399: finishing write tid 1 to trial04342147-33 2026-02-20T11:17:33.766 INFO:tasks.rados.rados.0.trial043.stdout:400: finishing write tid 1 to trial04342147-9 2026-02-20T11:17:33.766 INFO:tasks.rados.rados.0.trial043.stdout:401: finishing write tid 1 to trial04342147-10 2026-02-20T11:17:33.766 INFO:tasks.rados.rados.0.trial043.stdout:400: finishing write tid 2 to trial04342147-9 2026-02-20T11:17:33.766 INFO:tasks.rados.rados.0.trial043.stdout:399: finishing write tid 2 to trial04342147-33 2026-02-20T11:17:33.766 INFO:tasks.rados.rados.0.trial043.stdout:401: finishing write tid 2 to trial04342147-10 2026-02-20T11:17:33.766 INFO:tasks.rados.rados.0.trial043.stdout:400: finishing write tid 3 to trial04342147-9 2026-02-20T11:17:33.766 INFO:tasks.rados.rados.0.trial043.stdout:401: finishing write tid 3 to trial04342147-10 2026-02-20T11:17:33.766 INFO:tasks.rados.rados.0.trial043.stdout:399: finishing write tid 3 to trial04342147-33 2026-02-20T11:17:33.766 INFO:tasks.rados.rados.0.trial043.stdout:401: finishing write tid 4 to trial04342147-10 2026-02-20T11:17:33.766 INFO:tasks.rados.rados.0.trial043.stdout:401: finishing write tid 5 to trial04342147-10 2026-02-20T11:17:33.767 INFO:tasks.rados.rados.0.trial043.stdout:401: oid 10 updating version 0 to 54 2026-02-20T11:17:33.767 INFO:tasks.rados.rados.0.trial043.stdout:401: oid 10 version 54 is already newer than 51 2026-02-20T11:17:33.767 INFO:tasks.rados.rados.0.trial043.stdout:401: oid 10 version 54 is already newer than 54 2026-02-20T11:17:33.767 INFO:tasks.rados.rados.0.trial043.stdout:401: oid 10 version 54 is already newer than 54 2026-02-20T11:17:33.767 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 10 v 54 (ObjNum 203 snap 55 seq_num 203) dirty exists 2026-02-20T11:17:33.767 INFO:tasks.rados.rados.0.trial043.stdout:401: left oid 10 (ObjNum 203 snap 55 seq_num 203) 2026-02-20T11:17:33.767 INFO:tasks.rados.rados.0.trial043.stdout:399: finishing write tid 4 to trial04342147-33 2026-02-20T11:17:33.767 INFO:tasks.rados.rados.0.trial043.stdout:401: done (4 left) 2026-02-20T11:17:33.767 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 4 2026-02-20T11:17:33.767 INFO:tasks.rados.rados.0.trial043.stdout:399: finishing write tid 5 to trial04342147-33 2026-02-20T11:17:33.767 INFO:tasks.rados.rados.0.trial043.stdout:399: oid 33 updating version 0 to 83 2026-02-20T11:17:33.768 INFO:tasks.rados.rados.0.trial043.stdout:399: oid 33 updating version 83 to 86 2026-02-20T11:17:33.768 INFO:tasks.rados.rados.0.trial043.stdout:399: oid 33 version 86 is already newer than 86 2026-02-20T11:17:33.768 INFO:tasks.rados.rados.0.trial043.stdout:399: oid 33 version 86 is already newer than 86 2026-02-20T11:17:33.768 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 33 v 86 (ObjNum 201 snap 55 seq_num 201) dirty exists 2026-02-20T11:17:33.768 INFO:tasks.rados.rados.0.trial043.stdout:399: left oid 33 (ObjNum 201 snap 55 seq_num 201) 2026-02-20T11:17:33.768 INFO:tasks.rados.rados.0.trial043.stdout:400: finishing write tid 4 to trial04342147-9 2026-02-20T11:17:33.768 INFO:tasks.rados.rados.0.trial043.stdout:400: finishing write tid 5 to trial04342147-9 2026-02-20T11:17:33.768 INFO:tasks.rados.rados.0.trial043.stdout:400: oid 9 updating version 0 to 78 2026-02-20T11:17:33.768 INFO:tasks.rados.rados.0.trial043.stdout:400: oid 9 version 78 is already newer than 78 2026-02-20T11:17:33.768 INFO:tasks.rados.rados.0.trial043.stdout:400: oid 9 version 78 is already newer than 78 2026-02-20T11:17:33.768 INFO:tasks.rados.rados.0.trial043.stdout:400: oid 9 version 78 is already newer than 78 2026-02-20T11:17:33.768 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 9 v 78 (ObjNum 202 snap 55 seq_num 202) dirty exists 2026-02-20T11:17:33.768 INFO:tasks.rados.rados.0.trial043.stdout:400: left oid 9 (ObjNum 202 snap 55 seq_num 202) 2026-02-20T11:17:33.769 INFO:tasks.rados.rados.0.trial043.stdout:402: finishing write tid 1 to trial04342147-18 2026-02-20T11:17:33.769 INFO:tasks.rados.rados.0.trial043.stdout:399: done (3 left) 2026-02-20T11:17:33.769 INFO:tasks.rados.rados.0.trial043.stdout:400: done (2 left) 2026-02-20T11:17:33.769 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:17:33.769 INFO:tasks.rados.rados.0.trial043.stdout:402: finishing write tid 2 to trial04342147-18 2026-02-20T11:17:33.769 INFO:tasks.rados.rados.0.trial043.stdout:402: finishing write tid 3 to trial04342147-18 2026-02-20T11:17:33.769 INFO:tasks.rados.rados.0.trial043.stdout:402: finishing write tid 4 to trial04342147-18 2026-02-20T11:17:33.769 INFO:tasks.rados.rados.0.trial043.stdout:402: finishing write tid 5 to trial04342147-18 2026-02-20T11:17:33.769 INFO:tasks.rados.rados.0.trial043.stdout:402: oid 18 updating version 0 to 38 2026-02-20T11:17:33.769 INFO:tasks.rados.rados.0.trial043.stdout:402: oid 18 version 38 is already newer than 38 2026-02-20T11:17:33.769 INFO:tasks.rados.rados.0.trial043.stdout:402: oid 18 version 38 is already newer than 35 2026-02-20T11:17:33.769 INFO:tasks.rados.rados.0.trial043.stdout:402: oid 18 version 38 is already newer than 38 2026-02-20T11:17:33.770 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 18 v 38 (ObjNum 204 snap 55 seq_num 204) dirty exists 2026-02-20T11:17:33.770 INFO:tasks.rados.rados.0.trial043.stdout:402: left oid 18 (ObjNum 204 snap 55 seq_num 204) 2026-02-20T11:17:33.770 INFO:tasks.rados.rados.0.trial043.stdout:402: done (1 left) 2026-02-20T11:17:33.770 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:17:33.779 INFO:tasks.rados.rados.0.trial043.stdout:404: done (0 left) 2026-02-20T11:17:34.068 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:17:34.146 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:34.144+0000 7f39132cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:34.148 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:34.147+0000 7f39132cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:34.174 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:17:34.195 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:17:34.196 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:17:34.768 INFO:tasks.rados.rados.0.trial043.stdout:405: done (0 left) 2026-02-20T11:17:34.768 INFO:tasks.rados.rados.0.trial043.stdout:406: write (excl) oid 1 current snap is 56 2026-02-20T11:17:34.768 INFO:tasks.rados.rados.0.trial043.stdout:406: seq_num 205 ranges {645435=451223,1879985=787072,3280184=165251} 2026-02-20T11:17:34.771 INFO:tasks.rados.rados.0.trial043.stdout:406: writing trial04342147-1 from 645435 to 1096658 tid 1 2026-02-20T11:17:34.778 INFO:tasks.rados.rados.0.trial043.stdout:406: writing trial04342147-1 from 1879985 to 2667057 tid 2 2026-02-20T11:17:34.781 INFO:tasks.rados.rados.0.trial043.stdout:406: writing trial04342147-1 from 3280184 to 3445435 tid 3 2026-02-20T11:17:34.782 INFO:tasks.rados.rados.0.trial043.stdout:407: read oid 10 snap 25 2026-02-20T11:17:34.782 INFO:tasks.rados.rados.0.trial043.stdout:407: expect deleted 2026-02-20T11:17:34.782 INFO:tasks.rados.rados.0.trial043.stdout:408: read oid 13 snap -1 2026-02-20T11:17:34.782 INFO:tasks.rados.rados.0.trial043.stdout:408: expect (ObjNum 187 snap 48 seq_num 187) 2026-02-20T11:17:34.782 INFO:tasks.rados.rados.0.trial043.stdout:409: read oid 2 snap -1 2026-02-20T11:17:34.782 INFO:tasks.rados.rados.0.trial043.stdout:409: expect (ObjNum 190 snap 48 seq_num 190) 2026-02-20T11:17:34.782 INFO:tasks.rados.rados.0.trial043.stdout:410: snap_create 2026-02-20T11:17:34.782 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 4 2026-02-20T11:17:34.791 INFO:tasks.rados.rados.0.trial043.stdout:407: done (3 left) 2026-02-20T11:17:34.791 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:17:34.796 INFO:tasks.rados.rados.0.trial043.stdout:408: expect (ObjNum 187 snap 48 seq_num 187) 2026-02-20T11:17:34.866 INFO:tasks.rados.rados.0.trial043.stdout:408: done (2 left) 2026-02-20T11:17:34.866 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:17:34.866 INFO:tasks.rados.rados.0.trial043.stdout:409: expect (ObjNum 190 snap 48 seq_num 190) 2026-02-20T11:17:34.887 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2166474 2026-02-20T11:17:34.905 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:17:35.002 INFO:tasks.rados.rados.0.trial043.stdout:406: finishing write tid 1 to trial04342147-1 2026-02-20T11:17:35.002 INFO:tasks.rados.rados.0.trial043.stdout:406: finishing write tid 2 to trial04342147-1 2026-02-20T11:17:35.002 INFO:tasks.rados.rados.0.trial043.stdout:409: done (1 left) 2026-02-20T11:17:35.003 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:17:35.003 INFO:tasks.rados.rados.0.trial043.stdout:406: finishing write tid 3 to trial04342147-1 2026-02-20T11:17:35.003 INFO:tasks.rados.rados.0.trial043.stdout:406: finishing write tid 4 to trial04342147-1 2026-02-20T11:17:35.003 INFO:tasks.rados.rados.0.trial043.stdout:406: finishing write tid 5 to trial04342147-1 2026-02-20T11:17:35.003 INFO:tasks.rados.rados.0.trial043.stdout:406: oid 1 updating version 0 to 45 2026-02-20T11:17:35.003 INFO:tasks.rados.rados.0.trial043.stdout:406: oid 1 updating version 45 to 48 2026-02-20T11:17:35.003 INFO:tasks.rados.rados.0.trial043.stdout:406: oid 1 version 48 is already newer than 48 2026-02-20T11:17:35.003 INFO:tasks.rados.rados.0.trial043.stdout:406: oid 1 version 48 is already newer than 48 2026-02-20T11:17:35.003 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 1 v 48 (ObjNum 205 snap 56 seq_num 205) dirty exists 2026-02-20T11:17:35.003 INFO:tasks.rados.rados.0.trial043.stdout:406: left oid 1 (ObjNum 205 snap 56 seq_num 205) 2026-02-20T11:17:35.003 INFO:tasks.rados.rados.0.trial043.stdout:406: done (0 left) 2026-02-20T11:17:35.411 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~110s 2026-02-20T11:17:35.770 INFO:tasks.rados.rados.0.trial043.stdout:410: done (0 left) 2026-02-20T11:17:35.771 INFO:tasks.rados.rados.0.trial043.stdout:411: read oid 23 snap -1 2026-02-20T11:17:35.771 INFO:tasks.rados.rados.0.trial043.stdout:411: expect (ObjNum 165 snap 38 seq_num 165) 2026-02-20T11:17:35.771 INFO:tasks.rados.rados.0.trial043.stdout:412: snap_create 2026-02-20T11:17:35.771 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:17:35.808 INFO:tasks.rados.rados.0.trial043.stdout:411: expect (ObjNum 165 snap 38 seq_num 165) 2026-02-20T11:17:35.808 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 305801 2026-02-20T11:17:35.871 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1974657 2026-02-20T11:17:35.921 INFO:tasks.rados.rados.0.trial043.stdout:411: done (0 left) 2026-02-20T11:17:36.197 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:17:36.273 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:36.271+0000 7f39b8ec2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:36.275 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:36.274+0000 7f39b8ec2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:36.301 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:17:36.317 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:17:36.318 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:17:36.775 INFO:tasks.rados.rados.0.trial043.stdout:412: done (0 left) 2026-02-20T11:17:36.775 INFO:tasks.rados.rados.0.trial043.stdout:413: write oid 28 current snap is 58 2026-02-20T11:17:36.775 INFO:tasks.rados.rados.0.trial043.stdout:413: seq_num 206 ranges {662242=642825,2054820=530730,3307607=154635} 2026-02-20T11:17:36.780 INFO:tasks.rados.rados.0.trial043.stdout:413: writing trial04342147-28 from 662242 to 1305067 tid 1 2026-02-20T11:17:36.785 INFO:tasks.rados.rados.0.trial043.stdout:413: writing trial04342147-28 from 2054820 to 2585550 tid 2 2026-02-20T11:17:36.789 INFO:tasks.rados.rados.0.trial043.stdout:413: writing trial04342147-28 from 3307607 to 3462242 tid 3 2026-02-20T11:17:36.789 INFO:tasks.rados.rados.0.trial043.stdout:414: write oid 1 current snap is 58 2026-02-20T11:17:36.789 INFO:tasks.rados.rados.0.trial043.stdout:414: seq_num 207 ranges {679049=718074,2113302=674388,3335030=144019} 2026-02-20T11:17:36.794 INFO:tasks.rados.rados.0.trial043.stdout:414: writing trial04342147-1 from 679049 to 1397123 tid 1 2026-02-20T11:17:36.800 INFO:tasks.rados.rados.0.trial043.stdout:414: writing trial04342147-1 from 2113302 to 2787690 tid 2 2026-02-20T11:17:36.803 INFO:tasks.rados.rados.0.trial043.stdout:414: writing trial04342147-1 from 3335030 to 3479049 tid 3 2026-02-20T11:17:36.803 INFO:tasks.rados.rados.0.trial043.stdout:415: delete oid 44 current snap is 58 2026-02-20T11:17:36.805 INFO:tasks.rados.rados.0.trial043.stdout:415: done (2 left) 2026-02-20T11:17:36.805 INFO:tasks.rados.rados.0.trial043.stdout:416: write oid 5 current snap is 58 2026-02-20T11:17:36.806 INFO:tasks.rados.rados.0.trial043.stdout:416: seq_num 209 ranges {712663=468572,1713913=678057,2873523=591733} 2026-02-20T11:17:36.810 INFO:tasks.rados.rados.0.trial043.stdout:416: writing trial04342147-5 from 712663 to 1181235 tid 1 2026-02-20T11:17:36.815 INFO:tasks.rados.rados.0.trial043.stdout:416: writing trial04342147-5 from 1713913 to 2391970 tid 2 2026-02-20T11:17:36.819 INFO:tasks.rados.rados.0.trial043.stdout:416: writing trial04342147-5 from 2873523 to 3465256 tid 3 2026-02-20T11:17:36.819 INFO:tasks.rados.rados.0.trial043.stdout:417: read oid 43 snap 56 2026-02-20T11:17:36.819 INFO:tasks.rados.rados.0.trial043.stdout:417: expect (ObjNum 176 snap 43 seq_num 176) 2026-02-20T11:17:36.819 INFO:tasks.rados.rados.0.trial043.stdout:418: delete oid 48 current snap is 58 2026-02-20T11:17:36.840 INFO:tasks.rados.rados.0.trial043.stdout:417: expect (ObjNum 176 snap 43 seq_num 176) 2026-02-20T11:17:36.840 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:17:36.892 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2776371 2026-02-20T11:17:36.923 INFO:tasks.rados.rados.0.trial043.stdout:417: done (4 left) 2026-02-20T11:17:36.923 INFO:tasks.rados.rados.0.trial043.stdout:418: done (3 left) 2026-02-20T11:17:36.923 INFO:tasks.rados.rados.0.trial043.stdout:419: snap_remove snap 53 2026-02-20T11:17:37.657 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:17:37.730 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:17:37.733 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:17:37.734 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:17:37.777 INFO:tasks.rados.rados.0.trial043.stdout:419: done (3 left) 2026-02-20T11:17:37.777 INFO:tasks.rados.rados.0.trial043.stdout:420: delete oid 35 current snap is 58 2026-02-20T11:17:37.777 INFO:tasks.rados.rados.0.trial043.stdout:416: finishing write tid 1 to trial04342147-5 2026-02-20T11:17:37.777 INFO:tasks.rados.rados.0.trial043.stdout:416: finishing write tid 2 to trial04342147-5 2026-02-20T11:17:37.777 INFO:tasks.rados.rados.0.trial043.stdout:416: finishing write tid 3 to trial04342147-5 2026-02-20T11:17:37.777 INFO:tasks.rados.rados.0.trial043.stdout:416: finishing write tid 4 to trial04342147-5 2026-02-20T11:17:37.777 INFO:tasks.rados.rados.0.trial043.stdout:416: finishing write tid 5 to trial04342147-5 2026-02-20T11:17:37.777 INFO:tasks.rados.rados.0.trial043.stdout:416: oid 5 updating version 0 to 58 2026-02-20T11:17:37.778 INFO:tasks.rados.rados.0.trial043.stdout:416: oid 5 version 58 is already newer than 58 2026-02-20T11:17:37.778 INFO:tasks.rados.rados.0.trial043.stdout:416: oid 5 version 58 is already newer than 58 2026-02-20T11:17:37.778 INFO:tasks.rados.rados.0.trial043.stdout:416: oid 5 version 58 is already newer than 58 2026-02-20T11:17:37.778 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 5 v 58 (ObjNum 209 snap 58 seq_num 209) dirty exists 2026-02-20T11:17:37.778 INFO:tasks.rados.rados.0.trial043.stdout:416: left oid 5 (ObjNum 209 snap 58 seq_num 209) 2026-02-20T11:17:37.778 INFO:tasks.rados.rados.0.trial043.stdout:413: finishing write tid 1 to trial04342147-28 2026-02-20T11:17:37.778 INFO:tasks.rados.rados.0.trial043.stdout:413: finishing write tid 2 to trial04342147-28 2026-02-20T11:17:37.778 INFO:tasks.rados.rados.0.trial043.stdout:413: finishing write tid 3 to trial04342147-28 2026-02-20T11:17:37.778 INFO:tasks.rados.rados.0.trial043.stdout:413: finishing write tid 4 to trial04342147-28 2026-02-20T11:17:37.778 INFO:tasks.rados.rados.0.trial043.stdout:413: finishing write tid 5 to trial04342147-28 2026-02-20T11:17:37.778 INFO:tasks.rados.rados.0.trial043.stdout:413: oid 28 updating version 0 to 53 2026-02-20T11:17:37.778 INFO:tasks.rados.rados.0.trial043.stdout:413: oid 28 version 53 is already newer than 53 2026-02-20T11:17:37.779 INFO:tasks.rados.rados.0.trial043.stdout:413: oid 28 version 53 is already newer than 53 2026-02-20T11:17:37.779 INFO:tasks.rados.rados.0.trial043.stdout:413: oid 28 version 53 is already newer than 53 2026-02-20T11:17:37.779 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 28 v 53 (ObjNum 206 snap 58 seq_num 206) dirty exists 2026-02-20T11:17:37.779 INFO:tasks.rados.rados.0.trial043.stdout:413: left oid 28 (ObjNum 206 snap 58 seq_num 206) 2026-02-20T11:17:37.779 INFO:tasks.rados.rados.0.trial043.stdout:414: finishing write tid 1 to trial04342147-1 2026-02-20T11:17:37.779 INFO:tasks.rados.rados.0.trial043.stdout:414: finishing write tid 2 to trial04342147-1 2026-02-20T11:17:37.779 INFO:tasks.rados.rados.0.trial043.stdout:414: finishing write tid 3 to trial04342147-1 2026-02-20T11:17:37.779 INFO:tasks.rados.rados.0.trial043.stdout:414: finishing write tid 4 to trial04342147-1 2026-02-20T11:17:37.779 INFO:tasks.rados.rados.0.trial043.stdout:414: finishing write tid 5 to trial04342147-1 2026-02-20T11:17:37.779 INFO:tasks.rados.rados.0.trial043.stdout:414: oid 1 updating version 0 to 58 2026-02-20T11:17:37.779 INFO:tasks.rados.rados.0.trial043.stdout:414: oid 1 version 58 is already newer than 58 2026-02-20T11:17:37.779 INFO:tasks.rados.rados.0.trial043.stdout:414: oid 1 version 58 is already newer than 58 2026-02-20T11:17:37.780 INFO:tasks.rados.rados.0.trial043.stdout:414: oid 1 version 58 is already newer than 58 2026-02-20T11:17:37.780 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 1 v 58 (ObjNum 207 snap 58 seq_num 207) dirty exists 2026-02-20T11:17:37.780 INFO:tasks.rados.rados.0.trial043.stdout:414: left oid 1 (ObjNum 207 snap 58 seq_num 207) 2026-02-20T11:17:37.795 INFO:tasks.rados.rados.0.trial043.stdout:413: done (3 left) 2026-02-20T11:17:37.795 INFO:tasks.rados.rados.0.trial043.stdout:414: done (2 left) 2026-02-20T11:17:37.796 INFO:tasks.rados.rados.0.trial043.stdout:416: done (1 left) 2026-02-20T11:17:37.796 INFO:tasks.rados.rados.0.trial043.stdout:420: done (0 left) 2026-02-20T11:17:37.796 INFO:tasks.rados.rados.0.trial043.stdout:421: read oid 39 snap 52 2026-02-20T11:17:37.796 INFO:tasks.rados.rados.0.trial043.stdout:421: expect deleted 2026-02-20T11:17:37.796 INFO:tasks.rados.rados.0.trial043.stdout:422: read oid 24 snap -1 2026-02-20T11:17:37.796 INFO:tasks.rados.rados.0.trial043.stdout:422: expect (ObjNum 186 snap 48 seq_num 186) 2026-02-20T11:17:37.796 INFO:tasks.rados.rados.0.trial043.stdout:423: write oid 35 current snap is 58 2026-02-20T11:17:37.796 INFO:tasks.rados.rados.0.trial043.stdout:423: seq_num 212 ranges {763084=694319,1889359=541737,2955792=541902} 2026-02-20T11:17:37.800 INFO:tasks.rados.rados.0.trial043.stdout:423: writing trial04342147-35 from 763084 to 1457403 tid 1 2026-02-20T11:17:37.804 INFO:tasks.rados.rados.0.trial043.stdout:423: writing trial04342147-35 from 1889359 to 2431096 tid 2 2026-02-20T11:17:37.808 INFO:tasks.rados.rados.0.trial043.stdout:423: writing trial04342147-35 from 2955792 to 3497694 tid 3 2026-02-20T11:17:37.808 INFO:tasks.rados.rados.0.trial043.stdout:424: write oid 45 current snap is 58 2026-02-20T11:17:37.809 INFO:tasks.rados.rados.0.trial043.stdout:424: seq_num 213 ranges {779891=485921,1947841=685395,3383215=196676} 2026-02-20T11:17:37.813 INFO:tasks.rados.rados.0.trial043.stdout:424: writing trial04342147-45 from 779891 to 1265812 tid 1 2026-02-20T11:17:37.818 INFO:tasks.rados.rados.0.trial043.stdout:424: writing trial04342147-45 from 1947841 to 2633236 tid 2 2026-02-20T11:17:37.822 INFO:tasks.rados.rados.0.trial043.stdout:424: writing trial04342147-45 from 3383215 to 3579891 tid 3 2026-02-20T11:17:37.822 INFO:tasks.rados.rados.0.trial043.stdout:425: snap_remove snap 57 2026-02-20T11:17:38.319 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:17:38.395 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:38.394+0000 7f6b80835640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:38.398 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:38.398+0000 7f6b80835640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:38.423 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:17:38.439 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:17:38.440 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:17:38.753 INFO:tasks.rados.rados.0.trial043.stdout:425: done (4 left) 2026-02-20T11:17:38.753 INFO:tasks.rados.rados.0.trial043.stdout:426: read oid 2 snap -1 2026-02-20T11:17:38.753 INFO:tasks.rados.rados.0.trial043.stdout:426: expect (ObjNum 190 snap 48 seq_num 190) 2026-02-20T11:17:38.753 INFO:tasks.rados.rados.0.trial043.stdout:427: write oid 47 current snap is 58 2026-02-20T11:17:38.753 INFO:tasks.rados.rados.0.trial043.stdout:427: seq_num 214 ranges {796698=561170,2006323=545406,3126991=469707} 2026-02-20T11:17:38.757 INFO:tasks.rados.rados.0.trial043.stdout:427: writing trial04342147-47 from 796698 to 1357868 tid 1 2026-02-20T11:17:38.762 INFO:tasks.rados.rados.0.trial043.stdout:427: writing trial04342147-47 from 2006323 to 2551729 tid 2 2026-02-20T11:17:38.767 INFO:tasks.rados.rados.0.trial043.stdout:427: writing trial04342147-47 from 3126991 to 3596698 tid 3 2026-02-20T11:17:38.767 INFO:tasks.rados.rados.0.trial043.stdout:428: snap_create 2026-02-20T11:17:38.767 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 6 2026-02-20T11:17:38.767 INFO:tasks.rados.rados.0.trial043.stdout:423: finishing write tid 1 to trial04342147-35 2026-02-20T11:17:38.767 INFO:tasks.rados.rados.0.trial043.stdout:423: finishing write tid 2 to trial04342147-35 2026-02-20T11:17:38.767 INFO:tasks.rados.rados.0.trial043.stdout:422: expect (ObjNum 186 snap 48 seq_num 186) 2026-02-20T11:17:38.782 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2416807 2026-02-20T11:17:38.793 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:17:38.839 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1870512 2026-02-20T11:17:38.857 INFO:tasks.rados.rados.0.trial043.stdout:423: finishing write tid 3 to trial04342147-35 2026-02-20T11:17:38.858 INFO:tasks.rados.rados.0.trial043.stdout:421: done (5 left) 2026-02-20T11:17:38.858 INFO:tasks.rados.rados.0.trial043.stdout:422: done (4 left) 2026-02-20T11:17:38.858 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 4 2026-02-20T11:17:38.858 INFO:tasks.rados.rados.0.trial043.stdout:423: finishing write tid 4 to trial04342147-35 2026-02-20T11:17:38.858 INFO:tasks.rados.rados.0.trial043.stdout:423: finishing write tid 5 to trial04342147-35 2026-02-20T11:17:38.858 INFO:tasks.rados.rados.0.trial043.stdout:423: oid 35 updating version 0 to 118 2026-02-20T11:17:38.858 INFO:tasks.rados.rados.0.trial043.stdout:423: oid 35 version 118 is already newer than 118 2026-02-20T11:17:38.858 INFO:tasks.rados.rados.0.trial043.stdout:423: oid 35 version 118 is already newer than 118 2026-02-20T11:17:38.858 INFO:tasks.rados.rados.0.trial043.stdout:423: oid 35 version 118 is already newer than 118 2026-02-20T11:17:38.858 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 35 v 118 (ObjNum 212 snap 58 seq_num 212) dirty exists 2026-02-20T11:17:38.858 INFO:tasks.rados.rados.0.trial043.stdout:423: left oid 35 (ObjNum 212 snap 58 seq_num 212) 2026-02-20T11:17:38.858 INFO:tasks.rados.rados.0.trial043.stdout:424: finishing write tid 1 to trial04342147-45 2026-02-20T11:17:38.858 INFO:tasks.rados.rados.0.trial043.stdout:424: finishing write tid 2 to trial04342147-45 2026-02-20T11:17:38.858 INFO:tasks.rados.rados.0.trial043.stdout:423: done (3 left) 2026-02-20T11:17:38.858 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:17:38.858 INFO:tasks.rados.rados.0.trial043.stdout:424: finishing write tid 3 to trial04342147-45 2026-02-20T11:17:38.858 INFO:tasks.rados.rados.0.trial043.stdout:424: finishing write tid 4 to trial04342147-45 2026-02-20T11:17:38.858 INFO:tasks.rados.rados.0.trial043.stdout:424: finishing write tid 5 to trial04342147-45 2026-02-20T11:17:38.858 INFO:tasks.rados.rados.0.trial043.stdout:424: oid 45 updating version 0 to 123 2026-02-20T11:17:38.858 INFO:tasks.rados.rados.0.trial043.stdout:424: oid 45 version 123 is already newer than 123 2026-02-20T11:17:38.858 INFO:tasks.rados.rados.0.trial043.stdout:424: oid 45 version 123 is already newer than 123 2026-02-20T11:17:38.858 INFO:tasks.rados.rados.0.trial043.stdout:424: oid 45 version 123 is already newer than 123 2026-02-20T11:17:38.858 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 45 v 123 (ObjNum 213 snap 58 seq_num 213) dirty exists 2026-02-20T11:17:38.858 INFO:tasks.rados.rados.0.trial043.stdout:424: left oid 45 (ObjNum 213 snap 58 seq_num 213) 2026-02-20T11:17:38.858 INFO:tasks.rados.rados.0.trial043.stdout:424: done (2 left) 2026-02-20T11:17:38.858 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:17:38.858 INFO:tasks.rados.rados.0.trial043.stdout:426: expect (ObjNum 190 snap 48 seq_num 190) 2026-02-20T11:17:38.879 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2105983 2026-02-20T11:17:38.958 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1768741 2026-02-20T11:17:38.986 INFO:tasks.rados.rados.0.trial043.stdout:427: finishing write tid 1 to trial04342147-47 2026-02-20T11:17:38.986 INFO:tasks.rados.rados.0.trial043.stdout:427: finishing write tid 2 to trial04342147-47 2026-02-20T11:17:38.986 INFO:tasks.rados.rados.0.trial043.stdout:426: done (1 left) 2026-02-20T11:17:38.986 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:17:38.986 INFO:tasks.rados.rados.0.trial043.stdout:427: finishing write tid 3 to trial04342147-47 2026-02-20T11:17:38.986 INFO:tasks.rados.rados.0.trial043.stdout:427: finishing write tid 4 to trial04342147-47 2026-02-20T11:17:38.986 INFO:tasks.rados.rados.0.trial043.stdout:427: finishing write tid 5 to trial04342147-47 2026-02-20T11:17:38.986 INFO:tasks.rados.rados.0.trial043.stdout:427: oid 47 updating version 0 to 67 2026-02-20T11:17:38.986 INFO:tasks.rados.rados.0.trial043.stdout:427: oid 47 version 67 is already newer than 67 2026-02-20T11:17:38.986 INFO:tasks.rados.rados.0.trial043.stdout:427: oid 47 version 67 is already newer than 67 2026-02-20T11:17:38.986 INFO:tasks.rados.rados.0.trial043.stdout:427: oid 47 version 67 is already newer than 67 2026-02-20T11:17:38.986 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 47 v 67 (ObjNum 214 snap 58 seq_num 214) dirty exists 2026-02-20T11:17:38.986 INFO:tasks.rados.rados.0.trial043.stdout:427: left oid 47 (ObjNum 214 snap 58 seq_num 214) 2026-02-20T11:17:38.986 INFO:tasks.rados.rados.0.trial043.stdout:427: done (0 left) 2026-02-20T11:17:39.783 INFO:tasks.rados.rados.0.trial043.stdout:428: done (0 left) 2026-02-20T11:17:39.784 INFO:tasks.rados.rados.0.trial043.stdout:429: write oid 22 current snap is 59 2026-02-20T11:17:39.784 INFO:tasks.rados.rados.0.trial043.stdout:429: seq_num 215 ranges {413505=636419,1664805=689064,3038061=492071} 2026-02-20T11:17:39.787 INFO:tasks.rados.rados.0.trial043.stdout:429: writing trial04342147-22 from 413505 to 1049924 tid 1 2026-02-20T11:17:39.792 INFO:tasks.rados.rados.0.trial043.stdout:429: writing trial04342147-22 from 1664805 to 2353869 tid 2 2026-02-20T11:17:39.797 INFO:tasks.rados.rados.0.trial043.stdout:429: writing trial04342147-22 from 3038061 to 3530132 tid 3 2026-02-20T11:17:39.797 INFO:tasks.rados.rados.0.trial043.stdout:430: snap_remove snap 55 2026-02-20T11:17:40.441 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:17:40.518 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:40.516+0000 7fcb39964640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:40.521 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:40.520+0000 7fcb39964640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:40.548 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:17:40.565 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:17:40.565 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:17:40.787 INFO:tasks.rados.rados.0.trial043.stdout:430: done (1 left) 2026-02-20T11:17:40.787 INFO:tasks.rados.rados.0.trial043.stdout:431: snap_create 2026-02-20T11:17:40.787 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:17:40.787 INFO:tasks.rados.rados.0.trial043.stdout:429: finishing write tid 1 to trial04342147-22 2026-02-20T11:17:40.787 INFO:tasks.rados.rados.0.trial043.stdout:429: finishing write tid 2 to trial04342147-22 2026-02-20T11:17:40.787 INFO:tasks.rados.rados.0.trial043.stdout:429: finishing write tid 3 to trial04342147-22 2026-02-20T11:17:40.787 INFO:tasks.rados.rados.0.trial043.stdout:429: finishing write tid 4 to trial04342147-22 2026-02-20T11:17:40.787 INFO:tasks.rados.rados.0.trial043.stdout:429: finishing write tid 5 to trial04342147-22 2026-02-20T11:17:40.788 INFO:tasks.rados.rados.0.trial043.stdout:429: oid 22 updating version 0 to 49 2026-02-20T11:17:40.788 INFO:tasks.rados.rados.0.trial043.stdout:429: oid 22 version 49 is already newer than 49 2026-02-20T11:17:40.788 INFO:tasks.rados.rados.0.trial043.stdout:429: oid 22 version 49 is already newer than 49 2026-02-20T11:17:40.788 INFO:tasks.rados.rados.0.trial043.stdout:429: oid 22 version 49 is already newer than 49 2026-02-20T11:17:40.788 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 22 v 49 (ObjNum 215 snap 59 seq_num 215) dirty exists 2026-02-20T11:17:40.788 INFO:tasks.rados.rados.0.trial043.stdout:429: left oid 22 (ObjNum 215 snap 59 seq_num 215) 2026-02-20T11:17:40.788 INFO:tasks.rados.rados.0.trial043.stdout:429: done (0 left) 2026-02-20T11:17:40.915 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~116s 2026-02-20T11:17:41.789 INFO:tasks.rados.rados.0.trial043.stdout:431: done (0 left) 2026-02-20T11:17:41.790 INFO:tasks.rados.rados.0.trial043.stdout:432: snap_create 2026-02-20T11:17:42.566 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:17:42.644 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:42.643+0000 7f2bbf65a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:42.646 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:42.646+0000 7f2bbf65a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:42.672 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:17:42.688 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:17:42.689 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:17:42.735 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:17:42.792 INFO:tasks.rados.rados.0.trial043.stdout:432: done (0 left) 2026-02-20T11:17:42.792 INFO:tasks.rados.rados.0.trial043.stdout:433: snap_remove snap 25 2026-02-20T11:17:42.809 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:17:42.812 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:17:42.813 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:17:43.795 INFO:tasks.rados.rados.0.trial043.stdout:433: done (0 left) 2026-02-20T11:17:43.795 INFO:tasks.rados.rados.0.trial043.stdout:434: write oid 46 current snap is 61 2026-02-20T11:17:43.795 INFO:tasks.rados.rados.0.trial043.stdout:434: seq_num 216 ranges {430312=711668,1723287=549075,2781837=703343} 2026-02-20T11:17:43.799 INFO:tasks.rados.rados.0.trial043.stdout:434: writing trial04342147-46 from 430312 to 1141980 tid 1 2026-02-20T11:17:43.804 INFO:tasks.rados.rados.0.trial043.stdout:434: writing trial04342147-46 from 1723287 to 2272362 tid 2 2026-02-20T11:17:43.808 INFO:tasks.rados.rados.0.trial043.stdout:434: writing trial04342147-46 from 2781837 to 3485180 tid 3 2026-02-20T11:17:43.808 INFO:tasks.rados.rados.0.trial043.stdout:435: snap_remove snap 59 2026-02-20T11:17:44.690 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:17:44.769 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:44.768+0000 7f4cc6bcc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:44.771 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:44.772+0000 7f4cc6bcc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:44.798 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:17:44.800 INFO:tasks.rados.rados.0.trial043.stdout:435: done (1 left) 2026-02-20T11:17:44.801 INFO:tasks.rados.rados.0.trial043.stdout:436: delete oid 37 current snap is 61 2026-02-20T11:17:44.801 INFO:tasks.rados.rados.0.trial043.stdout:434: finishing write tid 1 to trial04342147-46 2026-02-20T11:17:44.801 INFO:tasks.rados.rados.0.trial043.stdout:434: finishing write tid 2 to trial04342147-46 2026-02-20T11:17:44.801 INFO:tasks.rados.rados.0.trial043.stdout:434: finishing write tid 3 to trial04342147-46 2026-02-20T11:17:44.801 INFO:tasks.rados.rados.0.trial043.stdout:434: finishing write tid 4 to trial04342147-46 2026-02-20T11:17:44.801 INFO:tasks.rados.rados.0.trial043.stdout:434: finishing write tid 5 to trial04342147-46 2026-02-20T11:17:44.801 INFO:tasks.rados.rados.0.trial043.stdout:434: oid 46 updating version 0 to 13 2026-02-20T11:17:44.801 INFO:tasks.rados.rados.0.trial043.stdout:434: oid 46 version 13 is already newer than 13 2026-02-20T11:17:44.801 INFO:tasks.rados.rados.0.trial043.stdout:434: oid 46 version 13 is already newer than 13 2026-02-20T11:17:44.801 INFO:tasks.rados.rados.0.trial043.stdout:434: oid 46 version 13 is already newer than 13 2026-02-20T11:17:44.802 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 46 v 13 (ObjNum 216 snap 61 seq_num 216) dirty exists 2026-02-20T11:17:44.802 INFO:tasks.rados.rados.0.trial043.stdout:434: left oid 46 (ObjNum 216 snap 61 seq_num 216) 2026-02-20T11:17:44.827 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:17:44.828 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:17:44.845 INFO:tasks.rados.rados.0.trial043.stdout:434: done (1 left) 2026-02-20T11:17:44.845 INFO:tasks.rados.rados.0.trial043.stdout:436: done (0 left) 2026-02-20T11:17:44.846 INFO:tasks.rados.rados.0.trial043.stdout:437: delete oid 48 current snap is 61 2026-02-20T11:17:44.848 INFO:tasks.rados.rados.0.trial043.stdout:437: done (0 left) 2026-02-20T11:17:44.848 INFO:tasks.rados.rados.0.trial043.stdout:438: read oid 9 snap -1 2026-02-20T11:17:44.848 INFO:tasks.rados.rados.0.trial043.stdout:438: expect (ObjNum 202 snap 55 seq_num 202) 2026-02-20T11:17:44.849 INFO:tasks.rados.rados.0.trial043.stdout:439: read oid 33 snap 34 2026-02-20T11:17:44.849 INFO:tasks.rados.rados.0.trial043.stdout:439: expect deleted 2026-02-20T11:17:44.849 INFO:tasks.rados.rados.0.trial043.stdout:440: write oid 19 current snap is 61 2026-02-20T11:17:44.849 INFO:tasks.rados.rados.0.trial043.stdout:440: seq_num 219 ranges {480733=537415,1782380=696402,3031400=537159} 2026-02-20T11:17:44.851 INFO:tasks.rados.rados.0.trial043.stdout:440: writing trial04342147-19 from 480733 to 1018148 tid 1 2026-02-20T11:17:44.856 INFO:tasks.rados.rados.0.trial043.stdout:440: writing trial04342147-19 from 1782380 to 2478782 tid 2 2026-02-20T11:17:44.861 INFO:tasks.rados.rados.0.trial043.stdout:440: writing trial04342147-19 from 3031400 to 3568559 tid 3 2026-02-20T11:17:44.861 INFO:tasks.rados.rados.0.trial043.stdout:441: write oid 16 current snap is 61 2026-02-20T11:17:44.861 INFO:tasks.rados.rados.0.trial043.stdout:441: seq_num 220 ranges {497540=612664,1840862=440060,3058823=464784} 2026-02-20T11:17:44.865 INFO:tasks.rados.rados.0.trial043.stdout:441: writing trial04342147-16 from 497540 to 1110204 tid 1 2026-02-20T11:17:44.870 INFO:tasks.rados.rados.0.trial043.stdout:441: writing trial04342147-16 from 1840862 to 2280922 tid 2 2026-02-20T11:17:44.875 INFO:tasks.rados.rados.0.trial043.stdout:441: writing trial04342147-16 from 3058823 to 3523607 tid 3 2026-02-20T11:17:44.875 INFO:tasks.rados.rados.0.trial043.stdout:442: read oid 18 snap 58 2026-02-20T11:17:44.875 INFO:tasks.rados.rados.0.trial043.stdout:442: expect (ObjNum 204 snap 55 seq_num 204) 2026-02-20T11:17:44.875 INFO:tasks.rados.rados.0.trial043.stdout:443: snap_create 2026-02-20T11:17:44.875 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 5 2026-02-20T11:17:44.876 INFO:tasks.rados.rados.0.trial043.stdout:438: expect (ObjNum 202 snap 55 seq_num 202) 2026-02-20T11:17:44.924 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 849821 2026-02-20T11:17:44.965 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:17:45.014 INFO:tasks.rados.rados.0.trial043.stdout:438: done (4 left) 2026-02-20T11:17:45.015 INFO:tasks.rados.rados.0.trial043.stdout:439: done (3 left) 2026-02-20T11:17:45.015 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:17:45.015 INFO:tasks.rados.rados.0.trial043.stdout:440: finishing write tid 1 to trial04342147-19 2026-02-20T11:17:45.015 INFO:tasks.rados.rados.0.trial043.stdout:440: finishing write tid 2 to trial04342147-19 2026-02-20T11:17:45.015 INFO:tasks.rados.rados.0.trial043.stdout:442: expect (ObjNum 204 snap 55 seq_num 204) 2026-02-20T11:17:45.015 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:17:45.118 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:17:45.172 INFO:tasks.rados.rados.0.trial043.stdout:441: finishing write tid 1 to trial04342147-16 2026-02-20T11:17:45.172 INFO:tasks.rados.rados.0.trial043.stdout:440: finishing write tid 3 to trial04342147-19 2026-02-20T11:17:45.172 INFO:tasks.rados.rados.0.trial043.stdout:442: done (2 left) 2026-02-20T11:17:45.173 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:17:45.173 INFO:tasks.rados.rados.0.trial043.stdout:441: finishing write tid 2 to trial04342147-16 2026-02-20T11:17:45.173 INFO:tasks.rados.rados.0.trial043.stdout:440: finishing write tid 4 to trial04342147-19 2026-02-20T11:17:45.173 INFO:tasks.rados.rados.0.trial043.stdout:440: finishing write tid 5 to trial04342147-19 2026-02-20T11:17:45.173 INFO:tasks.rados.rados.0.trial043.stdout:440: oid 19 updating version 0 to 24 2026-02-20T11:17:45.173 INFO:tasks.rados.rados.0.trial043.stdout:440: oid 19 version 24 is already newer than 24 2026-02-20T11:17:45.174 INFO:tasks.rados.rados.0.trial043.stdout:440: oid 19 version 24 is already newer than 24 2026-02-20T11:17:45.174 INFO:tasks.rados.rados.0.trial043.stdout:440: oid 19 version 24 is already newer than 24 2026-02-20T11:17:45.174 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 19 v 24 (ObjNum 219 snap 61 seq_num 219) dirty exists 2026-02-20T11:17:45.174 INFO:tasks.rados.rados.0.trial043.stdout:440: left oid 19 (ObjNum 219 snap 61 seq_num 219) 2026-02-20T11:17:45.174 INFO:tasks.rados.rados.0.trial043.stdout:441: finishing write tid 3 to trial04342147-16 2026-02-20T11:17:45.174 INFO:tasks.rados.rados.0.trial043.stdout:441: finishing write tid 4 to trial04342147-16 2026-02-20T11:17:45.175 INFO:tasks.rados.rados.0.trial043.stdout:441: finishing write tid 5 to trial04342147-16 2026-02-20T11:17:45.175 INFO:tasks.rados.rados.0.trial043.stdout:441: oid 16 updating version 0 to 99 2026-02-20T11:17:45.175 INFO:tasks.rados.rados.0.trial043.stdout:441: oid 16 version 99 is already newer than 99 2026-02-20T11:17:45.175 INFO:tasks.rados.rados.0.trial043.stdout:441: oid 16 version 99 is already newer than 99 2026-02-20T11:17:45.175 INFO:tasks.rados.rados.0.trial043.stdout:441: oid 16 version 99 is already newer than 99 2026-02-20T11:17:45.175 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 16 v 99 (ObjNum 220 snap 61 seq_num 220) dirty exists 2026-02-20T11:17:45.176 INFO:tasks.rados.rados.0.trial043.stdout:441: left oid 16 (ObjNum 220 snap 61 seq_num 220) 2026-02-20T11:17:45.176 INFO:tasks.rados.rados.0.trial043.stdout:440: done (1 left) 2026-02-20T11:17:45.176 INFO:tasks.rados.rados.0.trial043.stdout:441: done (0 left) 2026-02-20T11:17:45.803 INFO:tasks.rados.rados.0.trial043.stdout:443: done (0 left) 2026-02-20T11:17:45.803 INFO:tasks.rados.rados.0.trial043.stdout:444: snap_remove snap 34 2026-02-20T11:17:46.420 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~121s 2026-02-20T11:17:46.809 INFO:tasks.rados.rados.0.trial043.stdout:444: done (0 left) 2026-02-20T11:17:46.809 INFO:tasks.rados.rados.0.trial043.stdout:445: write (excl) oid 30 current snap is 62 2026-02-20T11:17:46.809 INFO:tasks.rados.rados.0.trial043.stdout:445: seq_num 221 ranges {514347=404266,1499344=700071,2686246=676056} 2026-02-20T11:17:46.812 INFO:tasks.rados.rados.0.trial043.stdout:445: writing trial04342147-30 from 514347 to 918613 tid 1 2026-02-20T11:17:46.817 INFO:tasks.rados.rados.0.trial043.stdout:445: writing trial04342147-30 from 1499344 to 2199415 tid 2 2026-02-20T11:17:46.821 INFO:tasks.rados.rados.0.trial043.stdout:445: writing trial04342147-30 from 2686246 to 3362302 tid 3 2026-02-20T11:17:46.821 INFO:tasks.rados.rados.0.trial043.stdout:446: snap_create 2026-02-20T11:17:46.822 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:17:46.829 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:17:46.836 INFO:tasks.rados.rados.0.trial043.stdout:445: finishing write tid 1 to trial04342147-30 2026-02-20T11:17:46.855 INFO:tasks.rados.rados.0.trial043.stdout:445: finishing write tid 2 to trial04342147-30 2026-02-20T11:17:46.861 INFO:tasks.rados.rados.0.trial043.stdout:445: finishing write tid 3 to trial04342147-30 2026-02-20T11:17:46.867 INFO:tasks.rados.rados.0.trial043.stdout:445: finishing write tid 4 to trial04342147-30 2026-02-20T11:17:46.869 INFO:tasks.rados.rados.0.trial043.stdout:445: finishing write tid 5 to trial04342147-30 2026-02-20T11:17:46.869 INFO:tasks.rados.rados.0.trial043.stdout:445: oid 30 updating version 0 to 28 2026-02-20T11:17:46.869 INFO:tasks.rados.rados.0.trial043.stdout:445: oid 30 version 28 is already newer than 25 2026-02-20T11:17:46.869 INFO:tasks.rados.rados.0.trial043.stdout:445: oid 30 version 28 is already newer than 28 2026-02-20T11:17:46.869 INFO:tasks.rados.rados.0.trial043.stdout:445: oid 30 version 28 is already newer than 28 2026-02-20T11:17:46.869 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 30 v 28 (ObjNum 221 snap 62 seq_num 221) dirty exists 2026-02-20T11:17:46.870 INFO:tasks.rados.rados.0.trial043.stdout:445: left oid 30 (ObjNum 221 snap 62 seq_num 221) 2026-02-20T11:17:46.870 INFO:tasks.rados.rados.0.trial043.stdout:445: done (0 left) 2026-02-20T11:17:46.904 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:46.903+0000 7fe49a85a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:46.906 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:46.907+0000 7fe49a85a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:46.932 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:17:46.948 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:17:46.949 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:17:47.811 INFO:tasks.rados.rados.0.trial043.stdout:446: done (0 left) 2026-02-20T11:17:47.812 INFO:tasks.rados.rados.0.trial043.stdout:447: write (excl) oid 2 current snap is 63 2026-02-20T11:17:47.812 INFO:tasks.rados.rados.0.trial043.stdout:447: seq_num 222 ranges {531154=479515,1557826=443729,2713669=487328,3679295=51859} 2026-02-20T11:17:47.813 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:17:47.815 INFO:tasks.rados.rados.0.trial043.stdout:447: writing trial04342147-2 from 531154 to 1010669 tid 1 2026-02-20T11:17:47.819 INFO:tasks.rados.rados.0.trial043.stdout:447: writing trial04342147-2 from 1557826 to 2001555 tid 2 2026-02-20T11:17:47.824 INFO:tasks.rados.rados.0.trial043.stdout:447: writing trial04342147-2 from 2713669 to 3200997 tid 3 2026-02-20T11:17:47.826 INFO:tasks.rados.rados.0.trial043.stdout:447: writing trial04342147-2 from 3679295 to 3731154 tid 4 2026-02-20T11:17:47.826 INFO:tasks.rados.rados.0.trial043.stdout:448: read oid 35 snap -1 2026-02-20T11:17:47.826 INFO:tasks.rados.rados.0.trial043.stdout:448: expect (ObjNum 212 snap 58 seq_num 212) 2026-02-20T11:17:47.826 INFO:tasks.rados.rados.0.trial043.stdout:449: snap_remove snap 46 2026-02-20T11:17:47.885 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:17:47.887 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:17:47.888 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:17:48.755 INFO:tasks.rados.rados.0.trial043.stdout:449: done (2 left) 2026-02-20T11:17:48.755 INFO:tasks.rados.rados.0.trial043.stdout:450: write (excl) oid 9 current snap is 63 2026-02-20T11:17:48.755 INFO:tasks.rados.rados.0.trial043.stdout:450: seq_num 223 ranges {547961=554764,1616308=703740,2741092=698600} 2026-02-20T11:17:48.759 INFO:tasks.rados.rados.0.trial043.stdout:450: writing trial04342147-9 from 547961 to 1102725 tid 1 2026-02-20T11:17:48.764 INFO:tasks.rados.rados.0.trial043.stdout:450: writing trial04342147-9 from 1616308 to 2320048 tid 2 2026-02-20T11:17:48.769 INFO:tasks.rados.rados.0.trial043.stdout:450: writing trial04342147-9 from 2741092 to 3439692 tid 3 2026-02-20T11:17:48.769 INFO:tasks.rados.rados.0.trial043.stdout:451: read oid 47 snap -1 2026-02-20T11:17:48.769 INFO:tasks.rados.rados.0.trial043.stdout:451: expect (ObjNum 214 snap 58 seq_num 214) 2026-02-20T11:17:48.769 INFO:tasks.rados.rados.0.trial043.stdout:452: read oid 41 snap -1 2026-02-20T11:17:48.769 INFO:tasks.rados.rados.0.trial043.stdout:452: expect (ObjNum 148 snap 27 seq_num 148) 2026-02-20T11:17:48.769 INFO:tasks.rados.rados.0.trial043.stdout:453: write oid 3 current snap is 63 2026-02-20T11:17:48.769 INFO:tasks.rados.rados.0.trial043.stdout:448: expect (ObjNum 212 snap 58 seq_num 212) 2026-02-20T11:17:48.855 INFO:tasks.rados.rados.0.trial043.stdout:447: finishing write tid 1 to trial04342147-2 2026-02-20T11:17:48.855 INFO:tasks.rados.rados.0.trial043.stdout:447: finishing write tid 2 to trial04342147-2 2026-02-20T11:17:48.855 INFO:tasks.rados.rados.0.trial043.stdout:453: seq_num 224 ranges {564768=630013,1674790=447398,2768515=626225} 2026-02-20T11:17:48.859 INFO:tasks.rados.rados.0.trial043.stdout:453: writing trial04342147-3 from 564768 to 1194781 tid 1 2026-02-20T11:17:48.863 INFO:tasks.rados.rados.0.trial043.stdout:453: writing trial04342147-3 from 1674790 to 2122188 tid 2 2026-02-20T11:17:48.868 INFO:tasks.rados.rados.0.trial043.stdout:453: writing trial04342147-3 from 2768515 to 3394740 tid 3 2026-02-20T11:17:48.869 INFO:tasks.rados.rados.0.trial043.stdout:448: done (5 left) 2026-02-20T11:17:48.869 INFO:tasks.rados.rados.0.trial043.stdout:454: write (excl) oid 12 current snap is 63 2026-02-20T11:17:48.869 INFO:tasks.rados.rados.0.trial043.stdout:447: finishing write tid 3 to trial04342147-2 2026-02-20T11:17:48.869 INFO:tasks.rados.rados.0.trial043.stdout:454: seq_num 225 ranges {581575=705262,1733272=707409,3195938=437497} 2026-02-20T11:17:48.873 INFO:tasks.rados.rados.0.trial043.stdout:454: writing trial04342147-12 from 581575 to 1286837 tid 1 2026-02-20T11:17:48.877 INFO:tasks.rados.rados.0.trial043.stdout:454: writing trial04342147-12 from 1733272 to 2440681 tid 2 2026-02-20T11:17:48.882 INFO:tasks.rados.rados.0.trial043.stdout:454: writing trial04342147-12 from 3195938 to 3633435 tid 3 2026-02-20T11:17:48.882 INFO:tasks.rados.rados.0.trial043.stdout:455: write (excl) oid 4 current snap is 63 2026-02-20T11:17:48.882 INFO:tasks.rados.rados.0.trial043.stdout:447: finishing write tid 4 to trial04342147-2 2026-02-20T11:17:48.882 INFO:tasks.rados.rados.0.trial043.stdout:455: seq_num 226 ranges {598382=780511,2075401=451067,3107008=648769} 2026-02-20T11:17:48.887 INFO:tasks.rados.rados.0.trial043.stdout:455: writing trial04342147-4 from 598382 to 1378893 tid 1 2026-02-20T11:17:48.891 INFO:tasks.rados.rados.0.trial043.stdout:455: writing trial04342147-4 from 2075401 to 2526468 tid 2 2026-02-20T11:17:48.896 INFO:tasks.rados.rados.0.trial043.stdout:455: writing trial04342147-4 from 3107008 to 3755777 tid 3 2026-02-20T11:17:48.896 INFO:tasks.rados.rados.0.trial043.stdout:456: read oid 30 snap -1 2026-02-20T11:17:48.896 INFO:tasks.rados.rados.0.trial043.stdout:456: expect (ObjNum 221 snap 62 seq_num 221) 2026-02-20T11:17:48.896 INFO:tasks.rados.rados.0.trial043.stdout:457: read oid 11 snap 36 2026-02-20T11:17:48.896 INFO:tasks.rados.rados.0.trial043.stdout:457: expect deleted 2026-02-20T11:17:48.896 INFO:tasks.rados.rados.0.trial043.stdout:458: snap_remove snap 62 2026-02-20T11:17:48.950 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:17:49.026 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:49.025+0000 7f90f3562640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:49.028 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:49.029+0000 7f90f22d9640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:49.053 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:17:49.069 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:17:49.069 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:17:49.819 INFO:tasks.rados.rados.0.trial043.stdout:458: done (9 left) 2026-02-20T11:17:49.819 INFO:tasks.rados.rados.0.trial043.stdout:459: read oid 43 snap 39 2026-02-20T11:17:49.819 INFO:tasks.rados.rados.0.trial043.stdout:459: expect (ObjNum 42 snap 0 seq_num 42) 2026-02-20T11:17:49.819 INFO:tasks.rados.rados.0.trial043.stdout:460: snap_create 2026-02-20T11:17:49.819 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 10 2026-02-20T11:17:49.819 INFO:tasks.rados.rados.0.trial043.stdout:447: finishing write tid 5 to trial04342147-2 2026-02-20T11:17:49.819 INFO:tasks.rados.rados.0.trial043.stdout:447: finishing write tid 6 to trial04342147-2 2026-02-20T11:17:49.819 INFO:tasks.rados.rados.0.trial043.stdout:447: oid 2 updating version 0 to 69 2026-02-20T11:17:49.819 INFO:tasks.rados.rados.0.trial043.stdout:447: oid 2 updating version 69 to 73 2026-02-20T11:17:49.819 INFO:tasks.rados.rados.0.trial043.stdout:447: oid 2 version 73 is already newer than 73 2026-02-20T11:17:49.820 INFO:tasks.rados.rados.0.trial043.stdout:447: oid 2 version 73 is already newer than 73 2026-02-20T11:17:49.820 INFO:tasks.rados.rados.0.trial043.stdout:447: oid 2 version 73 is already newer than 73 2026-02-20T11:17:49.820 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 2 v 73 (ObjNum 222 snap 63 seq_num 222) dirty exists 2026-02-20T11:17:49.820 INFO:tasks.rados.rados.0.trial043.stdout:447: left oid 2 (ObjNum 222 snap 63 seq_num 222) 2026-02-20T11:17:49.820 INFO:tasks.rados.rados.0.trial043.stdout:447: done (9 left) 2026-02-20T11:17:49.820 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 9 2026-02-20T11:17:49.820 INFO:tasks.rados.rados.0.trial043.stdout:451: expect (ObjNum 214 snap 58 seq_num 214) 2026-02-20T11:17:49.820 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:17:49.916 INFO:tasks.rados.rados.0.trial043.stdout:450: finishing write tid 1 to trial04342147-9 2026-02-20T11:17:49.916 INFO:tasks.rados.rados.0.trial043.stdout:450: finishing write tid 2 to trial04342147-9 2026-02-20T11:17:49.916 INFO:tasks.rados.rados.0.trial043.stdout:451: done (8 left) 2026-02-20T11:17:49.917 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 8 2026-02-20T11:17:49.917 INFO:tasks.rados.rados.0.trial043.stdout:452: expect (ObjNum 148 snap 27 seq_num 148) 2026-02-20T11:17:49.933 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 294503 2026-02-20T11:17:49.965 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1422477 2026-02-20T11:17:49.981 INFO:tasks.rados.rados.0.trial043.stdout:450: finishing write tid 3 to trial04342147-9 2026-02-20T11:17:49.981 INFO:tasks.rados.rados.0.trial043.stdout:450: finishing write tid 4 to trial04342147-9 2026-02-20T11:17:49.981 INFO:tasks.rados.rados.0.trial043.stdout:452: done (7 left) 2026-02-20T11:17:49.981 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 7 2026-02-20T11:17:49.981 INFO:tasks.rados.rados.0.trial043.stdout:450: finishing write tid 5 to trial04342147-9 2026-02-20T11:17:49.981 INFO:tasks.rados.rados.0.trial043.stdout:450: oid 9 updating version 0 to 93 2026-02-20T11:17:49.981 INFO:tasks.rados.rados.0.trial043.stdout:450: oid 9 version 93 is already newer than 93 2026-02-20T11:17:49.981 INFO:tasks.rados.rados.0.trial043.stdout:450: oid 9 version 93 is already newer than 90 2026-02-20T11:17:49.981 INFO:tasks.rados.rados.0.trial043.stdout:450: oid 9 version 93 is already newer than 93 2026-02-20T11:17:49.982 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 9 v 93 (ObjNum 223 snap 63 seq_num 223) dirty exists 2026-02-20T11:17:49.982 INFO:tasks.rados.rados.0.trial043.stdout:450: left oid 9 (ObjNum 223 snap 63 seq_num 223) 2026-02-20T11:17:49.982 INFO:tasks.rados.rados.0.trial043.stdout:454: finishing write tid 1 to trial04342147-12 2026-02-20T11:17:49.982 INFO:tasks.rados.rados.0.trial043.stdout:450: done (6 left) 2026-02-20T11:17:49.982 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 6 2026-02-20T11:17:49.982 INFO:tasks.rados.rados.0.trial043.stdout:453: finishing write tid 1 to trial04342147-3 2026-02-20T11:17:49.982 INFO:tasks.rados.rados.0.trial043.stdout:457: done (5 left) 2026-02-20T11:17:49.982 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 5 2026-02-20T11:17:49.982 INFO:tasks.rados.rados.0.trial043.stdout:454: finishing write tid 2 to trial04342147-12 2026-02-20T11:17:49.982 INFO:tasks.rados.rados.0.trial043.stdout:456: expect (ObjNum 221 snap 62 seq_num 221) 2026-02-20T11:17:50.012 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:17:50.071 INFO:tasks.rados.rados.0.trial043.stdout:453: finishing write tid 2 to trial04342147-3 2026-02-20T11:17:50.071 INFO:tasks.rados.rados.0.trial043.stdout:454: finishing write tid 3 to trial04342147-12 2026-02-20T11:17:50.071 INFO:tasks.rados.rados.0.trial043.stdout:456: done (4 left) 2026-02-20T11:17:50.071 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 4 2026-02-20T11:17:50.071 INFO:tasks.rados.rados.0.trial043.stdout:455: finishing write tid 1 to trial04342147-4 2026-02-20T11:17:50.072 INFO:tasks.rados.rados.0.trial043.stdout:454: finishing write tid 4 to trial04342147-12 2026-02-20T11:17:50.072 INFO:tasks.rados.rados.0.trial043.stdout:453: finishing write tid 3 to trial04342147-3 2026-02-20T11:17:50.072 INFO:tasks.rados.rados.0.trial043.stdout:454: finishing write tid 5 to trial04342147-12 2026-02-20T11:17:50.072 INFO:tasks.rados.rados.0.trial043.stdout:454: oid 12 updating version 0 to 77 2026-02-20T11:17:50.072 INFO:tasks.rados.rados.0.trial043.stdout:454: oid 12 version 77 is already newer than 77 2026-02-20T11:17:50.072 INFO:tasks.rados.rados.0.trial043.stdout:454: oid 12 version 77 is already newer than 74 2026-02-20T11:17:50.072 INFO:tasks.rados.rados.0.trial043.stdout:454: oid 12 version 77 is already newer than 77 2026-02-20T11:17:50.072 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 12 v 77 (ObjNum 225 snap 63 seq_num 225) dirty exists 2026-02-20T11:17:50.073 INFO:tasks.rados.rados.0.trial043.stdout:454: left oid 12 (ObjNum 225 snap 63 seq_num 225) 2026-02-20T11:17:50.073 INFO:tasks.rados.rados.0.trial043.stdout:453: finishing write tid 4 to trial04342147-3 2026-02-20T11:17:50.073 INFO:tasks.rados.rados.0.trial043.stdout:453: finishing write tid 5 to trial04342147-3 2026-02-20T11:17:50.073 INFO:tasks.rados.rados.0.trial043.stdout:453: oid 3 updating version 0 to 130 2026-02-20T11:17:50.073 INFO:tasks.rados.rados.0.trial043.stdout:453: oid 3 version 130 is already newer than 130 2026-02-20T11:17:50.073 INFO:tasks.rados.rados.0.trial043.stdout:453: oid 3 version 130 is already newer than 130 2026-02-20T11:17:50.073 INFO:tasks.rados.rados.0.trial043.stdout:453: oid 3 version 130 is already newer than 130 2026-02-20T11:17:50.073 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 3 v 130 (ObjNum 224 snap 63 seq_num 224) dirty exists 2026-02-20T11:17:50.074 INFO:tasks.rados.rados.0.trial043.stdout:453: left oid 3 (ObjNum 224 snap 63 seq_num 224) 2026-02-20T11:17:50.074 INFO:tasks.rados.rados.0.trial043.stdout:455: finishing write tid 2 to trial04342147-4 2026-02-20T11:17:50.074 INFO:tasks.rados.rados.0.trial043.stdout:455: finishing write tid 3 to trial04342147-4 2026-02-20T11:17:50.074 INFO:tasks.rados.rados.0.trial043.stdout:455: finishing write tid 4 to trial04342147-4 2026-02-20T11:17:50.074 INFO:tasks.rados.rados.0.trial043.stdout:455: finishing write tid 5 to trial04342147-4 2026-02-20T11:17:50.074 INFO:tasks.rados.rados.0.trial043.stdout:455: oid 4 updating version 0 to 54 2026-02-20T11:17:50.074 INFO:tasks.rados.rados.0.trial043.stdout:455: oid 4 version 54 is already newer than 51 2026-02-20T11:17:50.074 INFO:tasks.rados.rados.0.trial043.stdout:455: oid 4 version 54 is already newer than 54 2026-02-20T11:17:50.075 INFO:tasks.rados.rados.0.trial043.stdout:455: oid 4 version 54 is already newer than 54 2026-02-20T11:17:50.075 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 4 v 54 (ObjNum 226 snap 63 seq_num 226) dirty exists 2026-02-20T11:17:50.075 INFO:tasks.rados.rados.0.trial043.stdout:455: left oid 4 (ObjNum 226 snap 63 seq_num 226) 2026-02-20T11:17:50.075 INFO:tasks.rados.rados.0.trial043.stdout:453: done (3 left) 2026-02-20T11:17:50.075 INFO:tasks.rados.rados.0.trial043.stdout:454: done (2 left) 2026-02-20T11:17:50.075 INFO:tasks.rados.rados.0.trial043.stdout:455: done (1 left) 2026-02-20T11:17:50.075 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:17:50.075 INFO:tasks.rados.rados.0.trial043.stdout:459: expect (ObjNum 42 snap 0 seq_num 42) 2026-02-20T11:17:50.076 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 503061 2026-02-20T11:17:50.120 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2406836 2026-02-20T11:17:50.125 INFO:tasks.rados.rados.0.trial043.stdout:459: done (0 left) 2026-02-20T11:17:50.822 INFO:tasks.rados.rados.0.trial043.stdout:460: done (0 left) 2026-02-20T11:17:50.822 INFO:tasks.rados.rados.0.trial043.stdout:461: delete oid 39 current snap is 64 2026-02-20T11:17:50.842 INFO:tasks.rados.rados.0.trial043.stdout:461: done (0 left) 2026-02-20T11:17:50.842 INFO:tasks.rados.rados.0.trial043.stdout:462: read oid 4 snap -1 2026-02-20T11:17:50.842 INFO:tasks.rados.rados.0.trial043.stdout:462: expect (ObjNum 226 snap 63 seq_num 226) 2026-02-20T11:17:50.842 INFO:tasks.rados.rados.0.trial043.stdout:463: write oid 29 current snap is 64 2026-02-20T11:17:50.842 INFO:tasks.rados.rados.0.trial043.stdout:463: seq_num 228 ranges {631996=531009,1792365=454736,2761854=671313} 2026-02-20T11:17:50.846 INFO:tasks.rados.rados.0.trial043.stdout:463: writing trial04342147-29 from 631996 to 1163005 tid 1 2026-02-20T11:17:50.850 INFO:tasks.rados.rados.0.trial043.stdout:463: writing trial04342147-29 from 1792365 to 2247101 tid 2 2026-02-20T11:17:50.855 INFO:tasks.rados.rados.0.trial043.stdout:463: writing trial04342147-29 from 2761854 to 3433167 tid 3 2026-02-20T11:17:50.855 INFO:tasks.rados.rados.0.trial043.stdout:464: write oid 26 current snap is 64 2026-02-20T11:17:50.856 INFO:tasks.rados.rados.0.trial043.stdout:464: seq_num 229 ranges {648803=722611,1967200=714747,3421983=426820} 2026-02-20T11:17:50.860 INFO:tasks.rados.rados.0.trial043.stdout:464: writing trial04342147-26 from 648803 to 1371414 tid 1 2026-02-20T11:17:50.865 INFO:tasks.rados.rados.0.trial043.stdout:464: writing trial04342147-26 from 1967200 to 2681947 tid 2 2026-02-20T11:17:50.870 INFO:tasks.rados.rados.0.trial043.stdout:464: writing trial04342147-26 from 3421983 to 3848803 tid 3 2026-02-20T11:17:50.870 INFO:tasks.rados.rados.0.trial043.stdout:465: delete oid 31 current snap is 64 2026-02-20T11:17:50.902 INFO:tasks.rados.rados.0.trial043.stdout:465: done (3 left) 2026-02-20T11:17:50.902 INFO:tasks.rados.rados.0.trial043.stdout:466: write oid 45 current snap is 64 2026-02-20T11:17:50.902 INFO:tasks.rados.rados.0.trial043.stdout:466: seq_num 231 ranges {682417=473109,1567811=602063,2844123=621482} 2026-02-20T11:17:50.906 INFO:tasks.rados.rados.0.trial043.stdout:466: writing trial04342147-45 from 682417 to 1155526 tid 1 2026-02-20T11:17:50.910 INFO:tasks.rados.rados.0.trial043.stdout:466: writing trial04342147-45 from 1567811 to 2169874 tid 2 2026-02-20T11:17:50.915 INFO:tasks.rados.rados.0.trial043.stdout:466: writing trial04342147-45 from 2844123 to 3465605 tid 3 2026-02-20T11:17:50.915 INFO:tasks.rados.rados.0.trial043.stdout:467: read oid 46 snap -1 2026-02-20T11:17:50.915 INFO:tasks.rados.rados.0.trial043.stdout:467: expect (ObjNum 216 snap 61 seq_num 216) 2026-02-20T11:17:50.915 INFO:tasks.rados.rados.0.trial043.stdout:468: snap_remove snap 51 2026-02-20T11:17:51.070 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:17:51.146 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:51.145+0000 7fb28d81a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:51.148 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:51.149+0000 7fb28d81a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:51.179 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:17:51.196 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:17:51.197 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:17:51.826 INFO:tasks.rados.rados.0.trial043.stdout:468: done (5 left) 2026-02-20T11:17:51.826 INFO:tasks.rados.rados.0.trial043.stdout:469: read oid 48 snap -1 2026-02-20T11:17:51.826 INFO:tasks.rados.rados.0.trial043.stdout:469: expect deleted 2026-02-20T11:17:51.826 INFO:tasks.rados.rados.0.trial043.stdout:470: delete oid 14 current snap is 64 2026-02-20T11:17:51.826 INFO:tasks.rados.rados.0.trial043.stdout:467: expect (ObjNum 216 snap 61 seq_num 216) 2026-02-20T11:17:51.836 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2872734 2026-02-20T11:17:51.887 INFO:tasks.rados.rados.0.trial043.stdout:463: finishing write tid 1 to trial04342147-29 2026-02-20T11:17:51.888 INFO:tasks.rados.rados.0.trial043.stdout:462: expect (ObjNum 226 snap 63 seq_num 226) 2026-02-20T11:17:51.888 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:17:51.925 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~127s 2026-02-20T11:17:51.990 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 3107316 2026-02-20T11:17:52.064 INFO:tasks.rados.rados.0.trial043.stdout:463: finishing write tid 2 to trial04342147-29 2026-02-20T11:17:52.064 INFO:tasks.rados.rados.0.trial043.stdout:466: finishing write tid 1 to trial04342147-45 2026-02-20T11:17:52.064 INFO:tasks.rados.rados.0.trial043.stdout:462: done (6 left) 2026-02-20T11:17:52.064 INFO:tasks.rados.rados.0.trial043.stdout:467: done (5 left) 2026-02-20T11:17:52.065 INFO:tasks.rados.rados.0.trial043.stdout:470: done (4 left) 2026-02-20T11:17:52.065 INFO:tasks.rados.rados.0.trial043.stdout:471: write oid 35 current snap is 64 2026-02-20T11:17:52.065 INFO:tasks.rados.rados.0.trial043.stdout:463: finishing write tid 3 to trial04342147-29 2026-02-20T11:17:52.065 INFO:tasks.rados.rados.0.trial043.stdout:471: seq_num 233 ranges {716031=623607,2084775=605732,3298969=617062} 2026-02-20T11:17:52.068 INFO:tasks.rados.rados.0.trial043.stdout:471: writing trial04342147-35 from 716031 to 1339638 tid 1 2026-02-20T11:17:52.073 INFO:tasks.rados.rados.0.trial043.stdout:471: writing trial04342147-35 from 2084775 to 2690507 tid 2 2026-02-20T11:17:52.079 INFO:tasks.rados.rados.0.trial043.stdout:471: writing trial04342147-35 from 3298969 to 3916031 tid 3 2026-02-20T11:17:52.080 INFO:tasks.rados.rados.0.trial043.stdout:472: delete oid 21 current snap is 64 2026-02-20T11:17:52.080 INFO:tasks.rados.rados.0.trial043.stdout:466: finishing write tid 2 to trial04342147-45 2026-02-20T11:17:52.080 INFO:tasks.rados.rados.0.trial043.stdout:463: finishing write tid 4 to trial04342147-29 2026-02-20T11:17:52.080 INFO:tasks.rados.rados.0.trial043.stdout:466: finishing write tid 3 to trial04342147-45 2026-02-20T11:17:52.080 INFO:tasks.rados.rados.0.trial043.stdout:463: finishing write tid 5 to trial04342147-29 2026-02-20T11:17:52.080 INFO:tasks.rados.rados.0.trial043.stdout:463: oid 29 updating version 0 to 116 2026-02-20T11:17:52.080 INFO:tasks.rados.rados.0.trial043.stdout:463: oid 29 version 116 is already newer than 116 2026-02-20T11:17:52.080 INFO:tasks.rados.rados.0.trial043.stdout:463: oid 29 version 116 is already newer than 116 2026-02-20T11:17:52.081 INFO:tasks.rados.rados.0.trial043.stdout:463: oid 29 version 116 is already newer than 116 2026-02-20T11:17:52.081 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 29 v 116 (ObjNum 228 snap 64 seq_num 228) dirty exists 2026-02-20T11:17:52.081 INFO:tasks.rados.rados.0.trial043.stdout:463: left oid 29 (ObjNum 228 snap 64 seq_num 228) 2026-02-20T11:17:52.081 INFO:tasks.rados.rados.0.trial043.stdout:466: finishing write tid 4 to trial04342147-45 2026-02-20T11:17:52.081 INFO:tasks.rados.rados.0.trial043.stdout:466: finishing write tid 5 to trial04342147-45 2026-02-20T11:17:52.081 INFO:tasks.rados.rados.0.trial043.stdout:466: oid 45 updating version 0 to 135 2026-02-20T11:17:52.081 INFO:tasks.rados.rados.0.trial043.stdout:466: oid 45 version 135 is already newer than 135 2026-02-20T11:17:52.081 INFO:tasks.rados.rados.0.trial043.stdout:466: oid 45 version 135 is already newer than 135 2026-02-20T11:17:52.081 INFO:tasks.rados.rados.0.trial043.stdout:466: oid 45 version 135 is already newer than 135 2026-02-20T11:17:52.081 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 45 v 135 (ObjNum 231 snap 64 seq_num 231) dirty exists 2026-02-20T11:17:52.082 INFO:tasks.rados.rados.0.trial043.stdout:466: left oid 45 (ObjNum 231 snap 64 seq_num 231) 2026-02-20T11:17:52.082 INFO:tasks.rados.rados.0.trial043.stdout:464: finishing write tid 1 to trial04342147-26 2026-02-20T11:17:52.082 INFO:tasks.rados.rados.0.trial043.stdout:464: finishing write tid 2 to trial04342147-26 2026-02-20T11:17:52.082 INFO:tasks.rados.rados.0.trial043.stdout:464: finishing write tid 3 to trial04342147-26 2026-02-20T11:17:52.082 INFO:tasks.rados.rados.0.trial043.stdout:464: finishing write tid 4 to trial04342147-26 2026-02-20T11:17:52.082 INFO:tasks.rados.rados.0.trial043.stdout:464: finishing write tid 5 to trial04342147-26 2026-02-20T11:17:52.082 INFO:tasks.rados.rados.0.trial043.stdout:464: oid 26 updating version 0 to 121 2026-02-20T11:17:52.082 INFO:tasks.rados.rados.0.trial043.stdout:464: oid 26 version 121 is already newer than 121 2026-02-20T11:17:52.082 INFO:tasks.rados.rados.0.trial043.stdout:464: oid 26 version 121 is already newer than 121 2026-02-20T11:17:52.082 INFO:tasks.rados.rados.0.trial043.stdout:464: oid 26 version 121 is already newer than 121 2026-02-20T11:17:52.082 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 26 v 121 (ObjNum 229 snap 64 seq_num 229) dirty exists 2026-02-20T11:17:52.083 INFO:tasks.rados.rados.0.trial043.stdout:464: left oid 26 (ObjNum 229 snap 64 seq_num 229) 2026-02-20T11:17:52.092 INFO:tasks.rados.rados.0.trial043.stdout:463: done (5 left) 2026-02-20T11:17:52.092 INFO:tasks.rados.rados.0.trial043.stdout:464: done (4 left) 2026-02-20T11:17:52.092 INFO:tasks.rados.rados.0.trial043.stdout:466: done (3 left) 2026-02-20T11:17:52.093 INFO:tasks.rados.rados.0.trial043.stdout:469: done (2 left) 2026-02-20T11:17:52.093 INFO:tasks.rados.rados.0.trial043.stdout:472: done (1 left) 2026-02-20T11:17:52.093 INFO:tasks.rados.rados.0.trial043.stdout:473: delete oid 45 current snap is 64 2026-02-20T11:17:52.100 INFO:tasks.rados.rados.0.trial043.stdout:471: finishing write tid 1 to trial04342147-35 2026-02-20T11:17:52.108 INFO:tasks.rados.rados.0.trial043.stdout:471: finishing write tid 2 to trial04342147-35 2026-02-20T11:17:52.114 INFO:tasks.rados.rados.0.trial043.stdout:471: finishing write tid 3 to trial04342147-35 2026-02-20T11:17:52.117 INFO:tasks.rados.rados.0.trial043.stdout:471: finishing write tid 4 to trial04342147-35 2026-02-20T11:17:52.120 INFO:tasks.rados.rados.0.trial043.stdout:471: finishing write tid 5 to trial04342147-35 2026-02-20T11:17:52.120 INFO:tasks.rados.rados.0.trial043.stdout:471: oid 35 updating version 0 to 140 2026-02-20T11:17:52.120 INFO:tasks.rados.rados.0.trial043.stdout:471: oid 35 version 140 is already newer than 140 2026-02-20T11:17:52.120 INFO:tasks.rados.rados.0.trial043.stdout:471: oid 35 version 140 is already newer than 140 2026-02-20T11:17:52.120 INFO:tasks.rados.rados.0.trial043.stdout:471: oid 35 version 140 is already newer than 140 2026-02-20T11:17:52.120 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 35 v 140 (ObjNum 233 snap 64 seq_num 233) dirty exists 2026-02-20T11:17:52.120 INFO:tasks.rados.rados.0.trial043.stdout:471: left oid 35 (ObjNum 233 snap 64 seq_num 233) 2026-02-20T11:17:52.127 INFO:tasks.rados.rados.0.trial043.stdout:471: done (1 left) 2026-02-20T11:17:52.127 INFO:tasks.rados.rados.0.trial043.stdout:473: done (0 left) 2026-02-20T11:17:52.127 INFO:tasks.rados.rados.0.trial043.stdout:474: read oid 41 snap 36 2026-02-20T11:17:52.127 INFO:tasks.rados.rados.0.trial043.stdout:474: expect (ObjNum 148 snap 27 seq_num 148) 2026-02-20T11:17:52.127 INFO:tasks.rados.rados.0.trial043.stdout:475: snap_create 2026-02-20T11:17:52.127 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:17:52.142 INFO:tasks.rados.rados.0.trial043.stdout:474: expect (ObjNum 148 snap 27 seq_num 148) 2026-02-20T11:17:52.172 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 867243 2026-02-20T11:17:52.192 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:17:52.219 INFO:tasks.rados.rados.0.trial043.stdout:474: done (0 left) 2026-02-20T11:17:52.830 INFO:tasks.rados.rados.0.trial043.stdout:475: done (0 left) 2026-02-20T11:17:52.830 INFO:tasks.rados.rados.0.trial043.stdout:476: write oid 37 current snap is 65 2026-02-20T11:17:52.830 INFO:tasks.rados.rados.0.trial043.stdout:476: seq_num 236 ranges {766452=565707,1860221=469412,2981238=594195} 2026-02-20T11:17:52.835 INFO:tasks.rados.rados.0.trial043.stdout:476: writing trial04342147-37 from 766452 to 1332159 tid 1 2026-02-20T11:17:52.839 INFO:tasks.rados.rados.0.trial043.stdout:476: writing trial04342147-37 from 1860221 to 2329633 tid 2 2026-02-20T11:17:52.844 INFO:tasks.rados.rados.0.trial043.stdout:476: writing trial04342147-37 from 2981238 to 3575433 tid 3 2026-02-20T11:17:52.844 INFO:tasks.rados.rados.0.trial043.stdout:477: write oid 45 current snap is 65 2026-02-20T11:17:52.844 INFO:tasks.rados.rados.0.trial043.stdout:477: seq_num 237 ranges {783259=640956,1918703=613070,3008661=405467,3821939=161320} 2026-02-20T11:17:52.849 INFO:tasks.rados.rados.0.trial043.stdout:477: writing trial04342147-45 from 783259 to 1424215 tid 1 2026-02-20T11:17:52.854 INFO:tasks.rados.rados.0.trial043.stdout:477: writing trial04342147-45 from 1918703 to 2531773 tid 2 2026-02-20T11:17:52.857 INFO:tasks.rados.rados.0.trial043.stdout:477: writing trial04342147-45 from 3008661 to 3414128 tid 3 2026-02-20T11:17:52.859 INFO:tasks.rados.rados.0.trial043.stdout:477: writing trial04342147-45 from 3821939 to 3983259 tid 4 2026-02-20T11:17:52.859 INFO:tasks.rados.rados.0.trial043.stdout:478: snap_create 2026-02-20T11:17:52.859 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:17:52.869 INFO:tasks.rados.rados.0.trial043.stdout:476: finishing write tid 1 to trial04342147-37 2026-02-20T11:17:52.877 INFO:tasks.rados.rados.0.trial043.stdout:476: finishing write tid 2 to trial04342147-37 2026-02-20T11:17:52.884 INFO:tasks.rados.rados.0.trial043.stdout:476: finishing write tid 3 to trial04342147-37 2026-02-20T11:17:52.889 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:17:52.890 INFO:tasks.rados.rados.0.trial043.stdout:476: finishing write tid 4 to trial04342147-37 2026-02-20T11:17:52.892 INFO:tasks.rados.rados.0.trial043.stdout:476: finishing write tid 5 to trial04342147-37 2026-02-20T11:17:52.892 INFO:tasks.rados.rados.0.trial043.stdout:476: oid 37 updating version 0 to 145 2026-02-20T11:17:52.892 INFO:tasks.rados.rados.0.trial043.stdout:476: oid 37 version 145 is already newer than 145 2026-02-20T11:17:52.892 INFO:tasks.rados.rados.0.trial043.stdout:476: oid 37 version 145 is already newer than 145 2026-02-20T11:17:52.892 INFO:tasks.rados.rados.0.trial043.stdout:476: oid 37 version 145 is already newer than 145 2026-02-20T11:17:52.892 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 37 v 145 (ObjNum 236 snap 65 seq_num 236) dirty exists 2026-02-20T11:17:52.892 INFO:tasks.rados.rados.0.trial043.stdout:476: left oid 37 (ObjNum 236 snap 65 seq_num 236) 2026-02-20T11:17:52.892 INFO:tasks.rados.rados.0.trial043.stdout:476: done (1 left) 2026-02-20T11:17:52.892 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:17:52.898 INFO:tasks.rados.rados.0.trial043.stdout:477: finishing write tid 1 to trial04342147-45 2026-02-20T11:17:52.904 INFO:tasks.rados.rados.0.trial043.stdout:477: finishing write tid 2 to trial04342147-45 2026-02-20T11:17:52.910 INFO:tasks.rados.rados.0.trial043.stdout:477: finishing write tid 3 to trial04342147-45 2026-02-20T11:17:52.915 INFO:tasks.rados.rados.0.trial043.stdout:477: finishing write tid 4 to trial04342147-45 2026-02-20T11:17:52.918 INFO:tasks.rados.rados.0.trial043.stdout:477: finishing write tid 5 to trial04342147-45 2026-02-20T11:17:52.920 INFO:tasks.rados.rados.0.trial043.stdout:477: finishing write tid 6 to trial04342147-45 2026-02-20T11:17:52.920 INFO:tasks.rados.rados.0.trial043.stdout:477: oid 45 updating version 0 to 150 2026-02-20T11:17:52.920 INFO:tasks.rados.rados.0.trial043.stdout:477: oid 45 version 150 is already newer than 150 2026-02-20T11:17:52.920 INFO:tasks.rados.rados.0.trial043.stdout:477: oid 45 version 150 is already newer than 150 2026-02-20T11:17:52.920 INFO:tasks.rados.rados.0.trial043.stdout:477: oid 45 version 150 is already newer than 150 2026-02-20T11:17:52.920 INFO:tasks.rados.rados.0.trial043.stdout:477: oid 45 version 150 is already newer than 150 2026-02-20T11:17:52.920 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 45 v 150 (ObjNum 237 snap 65 seq_num 237) dirty exists 2026-02-20T11:17:52.920 INFO:tasks.rados.rados.0.trial043.stdout:477: left oid 45 (ObjNum 237 snap 65 seq_num 237) 2026-02-20T11:17:52.920 INFO:tasks.rados.rados.0.trial043.stdout:477: done (0 left) 2026-02-20T11:17:52.960 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:17:52.962 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:17:52.962 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:17:53.197 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:17:53.273 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:53.273+0000 7f75a445a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:53.276 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:53.276+0000 7f75a445a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:53.303 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:17:53.321 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:17:53.321 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:17:53.833 INFO:tasks.rados.rados.0.trial043.stdout:478: done (0 left) 2026-02-20T11:17:53.833 INFO:tasks.rados.rados.0.trial043.stdout:479: delete oid 13 current snap is 66 2026-02-20T11:17:53.876 INFO:tasks.rados.rados.0.trial043.stdout:479: done (0 left) 2026-02-20T11:17:53.876 INFO:tasks.rados.rados.0.trial043.stdout:480: delete oid 48 current snap is 66 2026-02-20T11:17:53.878 INFO:tasks.rados.rados.0.trial043.stdout:480: done (0 left) 2026-02-20T11:17:53.878 INFO:tasks.rados.rados.0.trial043.stdout:481: read oid 40 snap -1 2026-02-20T11:17:53.878 INFO:tasks.rados.rados.0.trial043.stdout:481: expect (ObjNum 103 snap 13 seq_num 103) 2026-02-20T11:17:53.878 INFO:tasks.rados.rados.0.trial043.stdout:482: read oid 9 snap -1 2026-02-20T11:17:53.878 INFO:tasks.rados.rados.0.trial043.stdout:482: expect (ObjNum 223 snap 63 seq_num 223) 2026-02-20T11:17:53.878 INFO:tasks.rados.rados.0.trial043.stdout:483: write (excl) oid 24 current snap is 66 2026-02-20T11:17:53.878 INFO:tasks.rados.rados.0.trial043.stdout:483: seq_num 240 ranges {433680=466703,1577796=455884} 2026-02-20T11:17:53.882 INFO:tasks.rados.rados.0.trial043.stdout:483: writing trial04342147-24 from 433680 to 900383 tid 1 2026-02-20T11:17:53.886 INFO:tasks.rados.rados.0.trial043.stdout:483: writing trial04342147-24 from 1577796 to 2033680 tid 2 2026-02-20T11:17:53.886 INFO:tasks.rados.rados.0.trial043.stdout:484: delete oid 16 current snap is 66 2026-02-20T11:17:53.900 INFO:tasks.rados.rados.0.trial043.stdout:481: expect (ObjNum 103 snap 13 seq_num 103) 2026-02-20T11:17:54.055 INFO:tasks.rados.rados.0.trial043.stdout:481: done (3 left) 2026-02-20T11:17:54.055 INFO:tasks.rados.rados.0.trial043.stdout:484: done (2 left) 2026-02-20T11:17:54.055 INFO:tasks.rados.rados.0.trial043.stdout:485: read oid 47 snap 36 2026-02-20T11:17:54.055 INFO:tasks.rados.rados.0.trial043.stdout:485: expect (ObjNum 119 snap 14 seq_num 119) 2026-02-20T11:17:54.055 INFO:tasks.rados.rados.0.trial043.stdout:486: snap_remove snap 63 2026-02-20T11:17:54.838 INFO:tasks.rados.rados.0.trial043.stdout:482: expect (ObjNum 223 snap 63 seq_num 223) 2026-02-20T11:17:54.955 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1027771 2026-02-20T11:17:55.007 INFO:tasks.rados.rados.0.trial043.stdout:483: finishing write tid 1 to trial04342147-24 2026-02-20T11:17:55.007 INFO:tasks.rados.rados.0.trial043.stdout:483: finishing write tid 2 to trial04342147-24 2026-02-20T11:17:55.007 INFO:tasks.rados.rados.0.trial043.stdout:482: done (3 left) 2026-02-20T11:17:55.007 INFO:tasks.rados.rados.0.trial043.stdout:486: done (2 left) 2026-02-20T11:17:55.007 INFO:tasks.rados.rados.0.trial043.stdout:487: read oid 38 snap -1 2026-02-20T11:17:55.007 INFO:tasks.rados.rados.0.trial043.stdout:487: expect (ObjNum 162 snap 36 seq_num 162) 2026-02-20T11:17:55.007 INFO:tasks.rados.rados.0.trial043.stdout:488: write (excl) oid 5 current snap is 66 2026-02-20T11:17:55.007 INFO:tasks.rados.rados.0.trial043.stdout:483: finishing write tid 3 to trial04342147-24 2026-02-20T11:17:55.007 INFO:tasks.rados.rados.0.trial043.stdout:483: finishing write tid 4 to trial04342147-24 2026-02-20T11:17:55.007 INFO:tasks.rados.rados.0.trial043.stdout:483: oid 24 updating version 0 to 38 2026-02-20T11:17:55.007 INFO:tasks.rados.rados.0.trial043.stdout:483: oid 24 version 38 is already newer than 36 2026-02-20T11:17:55.007 INFO:tasks.rados.rados.0.trial043.stdout:483: oid 24 version 38 is already newer than 38 2026-02-20T11:17:55.007 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 24 v 38 (ObjNum 240 snap 66 seq_num 240) dirty exists 2026-02-20T11:17:55.007 INFO:tasks.rados.rados.0.trial043.stdout:483: left oid 24 (ObjNum 240 snap 66 seq_num 240) 2026-02-20T11:17:55.007 INFO:tasks.rados.rados.0.trial043.stdout:488: seq_num 242 ranges {467294=617201,1578407=488887} 2026-02-20T11:17:55.011 INFO:tasks.rados.rados.0.trial043.stdout:488: writing trial04342147-5 from 467294 to 1084495 tid 1 2026-02-20T11:17:55.015 INFO:tasks.rados.rados.0.trial043.stdout:488: writing trial04342147-5 from 1578407 to 2067294 tid 2 2026-02-20T11:17:55.015 INFO:tasks.rados.rados.0.trial043.stdout:483: done (3 left) 2026-02-20T11:17:55.015 INFO:tasks.rados.rados.0.trial043.stdout:489: snap_remove snap 58 2026-02-20T11:17:55.322 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:17:55.399 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:55.398+0000 7fec35f93640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:55.401 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:55.402+0000 7fec35f93640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:55.429 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:17:55.446 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:17:55.447 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:17:55.842 INFO:tasks.rados.rados.0.trial043.stdout:489: done (3 left) 2026-02-20T11:17:55.842 INFO:tasks.rados.rados.0.trial043.stdout:490: write (excl) oid 15 current snap is 66 2026-02-20T11:17:55.842 INFO:tasks.rados.rados.0.trial043.stdout:490: seq_num 243 ranges {484101=692450,1636889=447212} 2026-02-20T11:17:55.846 INFO:tasks.rados.rados.0.trial043.stdout:490: writing trial04342147-15 from 484101 to 1176551 tid 1 2026-02-20T11:17:55.850 INFO:tasks.rados.rados.0.trial043.stdout:490: writing trial04342147-15 from 1636889 to 2084101 tid 2 2026-02-20T11:17:55.850 INFO:tasks.rados.rados.0.trial043.stdout:491: write oid 6 current snap is 66 2026-02-20T11:17:55.850 INFO:tasks.rados.rados.0.trial043.stdout:491: seq_num 244 ranges {500908=484052,1411724=689184} 2026-02-20T11:17:55.853 INFO:tasks.rados.rados.0.trial043.stdout:491: writing trial04342147-6 from 500908 to 984960 tid 1 2026-02-20T11:17:55.858 INFO:tasks.rados.rados.0.trial043.stdout:491: writing trial04342147-6 from 1411724 to 2100908 tid 2 2026-02-20T11:17:55.858 INFO:tasks.rados.rados.0.trial043.stdout:492: read oid 31 snap -1 2026-02-20T11:17:55.858 INFO:tasks.rados.rados.0.trial043.stdout:492: expect deleted 2026-02-20T11:17:55.858 INFO:tasks.rados.rados.0.trial043.stdout:485: expect (ObjNum 119 snap 14 seq_num 119) 2026-02-20T11:17:55.874 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:17:55.907 INFO:tasks.rados.rados.0.trial043.stdout:493: read oid 24 snap -1 2026-02-20T11:17:55.907 INFO:tasks.rados.rados.0.trial043.stdout:493: expect (ObjNum 240 snap 66 seq_num 240) 2026-02-20T11:17:55.907 INFO:tasks.rados.rados.0.trial043.stdout:485: done (6 left) 2026-02-20T11:17:55.907 INFO:tasks.rados.rados.0.trial043.stdout:494: read oid 32 snap -1 2026-02-20T11:17:55.908 INFO:tasks.rados.rados.0.trial043.stdout:494: expect deleted 2026-02-20T11:17:55.908 INFO:tasks.rados.rados.0.trial043.stdout:495: snap_remove snap 52 2026-02-20T11:17:56.848 INFO:tasks.rados.rados.0.trial043.stdout:495: done (7 left) 2026-02-20T11:17:56.848 INFO:tasks.rados.rados.0.trial043.stdout:496: write oid 49 current snap is 66 2026-02-20T11:17:56.848 INFO:tasks.rados.rados.0.trial043.stdout:496: seq_num 245 ranges {517715=559301,1870206=247509} 2026-02-20T11:17:56.852 INFO:tasks.rados.rados.0.trial043.stdout:496: writing trial04342147-49 from 517715 to 1077016 tid 1 2026-02-20T11:17:56.856 INFO:tasks.rados.rados.0.trial043.stdout:496: writing trial04342147-49 from 1870206 to 2117715 tid 2 2026-02-20T11:17:56.856 INFO:tasks.rados.rados.0.trial043.stdout:497: read oid 9 snap -1 2026-02-20T11:17:56.856 INFO:tasks.rados.rados.0.trial043.stdout:497: expect (ObjNum 223 snap 63 seq_num 223) 2026-02-20T11:17:56.856 INFO:tasks.rados.rados.0.trial043.stdout:498: read oid 7 snap -1 2026-02-20T11:17:56.856 INFO:tasks.rados.rados.0.trial043.stdout:498: expect (ObjNum 63 snap 3 seq_num 63) 2026-02-20T11:17:56.856 INFO:tasks.rados.rados.0.trial043.stdout:499: snap_remove snap 50 2026-02-20T11:17:57.430 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~132s 2026-02-20T11:17:57.448 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:17:57.525 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:57.524+0000 7f7fe030b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:57.527 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:57.528+0000 7f7fe030b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:57.555 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:17:57.571 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:17:57.572 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:17:57.850 INFO:tasks.rados.rados.0.trial043.stdout:499: done (10 left) 2026-02-20T11:17:57.851 INFO:tasks.rados.rados.0.trial043.stdout:500: delete oid 3 current snap is 66 2026-02-20T11:17:57.851 INFO:tasks.rados.rados.0.trial043.stdout:487: expect (ObjNum 162 snap 36 seq_num 162) 2026-02-20T11:17:57.895 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:17:57.920 INFO:tasks.rados.rados.0.trial043.stdout:488: finishing write tid 1 to trial04342147-5 2026-02-20T11:17:57.920 INFO:tasks.rados.rados.0.trial043.stdout:488: finishing write tid 2 to trial04342147-5 2026-02-20T11:17:57.920 INFO:tasks.rados.rados.0.trial043.stdout:487: done (10 left) 2026-02-20T11:17:57.920 INFO:tasks.rados.rados.0.trial043.stdout:500: done (9 left) 2026-02-20T11:17:57.920 INFO:tasks.rados.rados.0.trial043.stdout:501: delete oid 26 current snap is 66 2026-02-20T11:17:57.921 INFO:tasks.rados.rados.0.trial043.stdout:488: finishing write tid 3 to trial04342147-5 2026-02-20T11:17:57.921 INFO:tasks.rados.rados.0.trial043.stdout:488: finishing write tid 4 to trial04342147-5 2026-02-20T11:17:57.921 INFO:tasks.rados.rados.0.trial043.stdout:488: oid 5 updating version 0 to 60 2026-02-20T11:17:57.921 INFO:tasks.rados.rados.0.trial043.stdout:488: oid 5 updating version 60 to 62 2026-02-20T11:17:57.921 INFO:tasks.rados.rados.0.trial043.stdout:488: oid 5 version 62 is already newer than 62 2026-02-20T11:17:57.921 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 5 v 62 (ObjNum 242 snap 66 seq_num 242) dirty exists 2026-02-20T11:17:57.921 INFO:tasks.rados.rados.0.trial043.stdout:488: left oid 5 (ObjNum 242 snap 66 seq_num 242) 2026-02-20T11:17:57.921 INFO:tasks.rados.rados.0.trial043.stdout:491: finishing write tid 1 to trial04342147-6 2026-02-20T11:17:57.921 INFO:tasks.rados.rados.0.trial043.stdout:490: finishing write tid 1 to trial04342147-15 2026-02-20T11:17:57.921 INFO:tasks.rados.rados.0.trial043.stdout:491: finishing write tid 2 to trial04342147-6 2026-02-20T11:17:57.921 INFO:tasks.rados.rados.0.trial043.stdout:491: finishing write tid 3 to trial04342147-6 2026-02-20T11:17:57.921 INFO:tasks.rados.rados.0.trial043.stdout:491: finishing write tid 4 to trial04342147-6 2026-02-20T11:17:57.922 INFO:tasks.rados.rados.0.trial043.stdout:491: oid 6 updating version 0 to 21 2026-02-20T11:17:57.922 INFO:tasks.rados.rados.0.trial043.stdout:491: oid 6 version 21 is already newer than 21 2026-02-20T11:17:57.922 INFO:tasks.rados.rados.0.trial043.stdout:491: oid 6 version 21 is already newer than 21 2026-02-20T11:17:57.922 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 6 v 21 (ObjNum 244 snap 66 seq_num 244) dirty exists 2026-02-20T11:17:57.922 INFO:tasks.rados.rados.0.trial043.stdout:491: left oid 6 (ObjNum 244 snap 66 seq_num 244) 2026-02-20T11:17:57.922 INFO:tasks.rados.rados.0.trial043.stdout:490: finishing write tid 2 to trial04342147-15 2026-02-20T11:17:57.922 INFO:tasks.rados.rados.0.trial043.stdout:490: finishing write tid 3 to trial04342147-15 2026-02-20T11:17:57.922 INFO:tasks.rados.rados.0.trial043.stdout:490: finishing write tid 4 to trial04342147-15 2026-02-20T11:17:57.922 INFO:tasks.rados.rados.0.trial043.stdout:490: oid 15 updating version 0 to 42 2026-02-20T11:17:57.922 INFO:tasks.rados.rados.0.trial043.stdout:490: oid 15 version 42 is already newer than 42 2026-02-20T11:17:57.922 INFO:tasks.rados.rados.0.trial043.stdout:490: oid 15 version 42 is already newer than 40 2026-02-20T11:17:57.922 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 15 v 42 (ObjNum 243 snap 66 seq_num 243) dirty exists 2026-02-20T11:17:57.923 INFO:tasks.rados.rados.0.trial043.stdout:490: left oid 15 (ObjNum 243 snap 66 seq_num 243) 2026-02-20T11:17:57.923 INFO:tasks.rados.rados.0.trial043.stdout:493: expect (ObjNum 240 snap 66 seq_num 240) 2026-02-20T11:17:57.923 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 638769 2026-02-20T11:17:57.943 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1130278 2026-02-20T11:17:57.955 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 437775 2026-02-20T11:17:57.964 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:17:57.974 INFO:tasks.rados.rados.0.trial043.stdout:488: done (9 left) 2026-02-20T11:17:57.974 INFO:tasks.rados.rados.0.trial043.stdout:490: done (8 left) 2026-02-20T11:17:57.974 INFO:tasks.rados.rados.0.trial043.stdout:491: done (7 left) 2026-02-20T11:17:57.974 INFO:tasks.rados.rados.0.trial043.stdout:492: done (6 left) 2026-02-20T11:17:57.974 INFO:tasks.rados.rados.0.trial043.stdout:493: done (5 left) 2026-02-20T11:17:57.974 INFO:tasks.rados.rados.0.trial043.stdout:494: done (4 left) 2026-02-20T11:17:57.974 INFO:tasks.rados.rados.0.trial043.stdout:501: done (3 left) 2026-02-20T11:17:57.974 INFO:tasks.rados.rados.0.trial043.stdout:502: read oid 24 snap 36 2026-02-20T11:17:57.974 INFO:tasks.rados.rados.0.trial043.stdout:502: expect (ObjNum 152 snap 31 seq_num 152) 2026-02-20T11:17:57.975 INFO:tasks.rados.rados.0.trial043.stdout:503: read oid 44 snap -1 2026-02-20T11:17:57.975 INFO:tasks.rados.rados.0.trial043.stdout:503: expect deleted 2026-02-20T11:17:57.975 INFO:tasks.rados.rados.0.trial043.stdout:504: write (excl) oid 29 current snap is 66 2026-02-20T11:17:57.975 INFO:tasks.rados.rados.0.trial043.stdout:498: expect (ObjNum 63 snap 3 seq_num 63) 2026-02-20T11:17:58.026 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2599794 2026-02-20T11:17:58.033 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 67075 2026-02-20T11:17:58.034 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:17:58.037 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:17:58.037 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:17:58.068 INFO:tasks.rados.rados.0.trial043.stdout:497: expect (ObjNum 223 snap 63 seq_num 223) 2026-02-20T11:17:58.106 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2925072 2026-02-20T11:17:58.116 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:17:58.241 INFO:tasks.rados.rados.0.trial043.stdout:496: finishing write tid 1 to trial04342147-49 2026-02-20T11:17:58.241 INFO:tasks.rados.rados.0.trial043.stdout:496: finishing write tid 2 to trial04342147-49 2026-02-20T11:17:58.241 INFO:tasks.rados.rados.0.trial043.stdout:496: finishing write tid 3 to trial04342147-49 2026-02-20T11:17:58.241 INFO:tasks.rados.rados.0.trial043.stdout:496: finishing write tid 4 to trial04342147-49 2026-02-20T11:17:58.241 INFO:tasks.rados.rados.0.trial043.stdout:496: oid 49 updating version 0 to 47 2026-02-20T11:17:58.241 INFO:tasks.rados.rados.0.trial043.stdout:496: oid 49 version 47 is already newer than 47 2026-02-20T11:17:58.242 INFO:tasks.rados.rados.0.trial043.stdout:496: oid 49 version 47 is already newer than 47 2026-02-20T11:17:58.242 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 49 v 47 (ObjNum 245 snap 66 seq_num 245) dirty exists 2026-02-20T11:17:58.242 INFO:tasks.rados.rados.0.trial043.stdout:496: left oid 49 (ObjNum 245 snap 66 seq_num 245) 2026-02-20T11:17:58.242 INFO:tasks.rados.rados.0.trial043.stdout:504: seq_num 248 ranges {568136=785048,1929299=238837} 2026-02-20T11:17:58.246 INFO:tasks.rados.rados.0.trial043.stdout:504: writing trial04342147-29 from 568136 to 1353184 tid 1 2026-02-20T11:17:58.249 INFO:tasks.rados.rados.0.trial043.stdout:504: writing trial04342147-29 from 1929299 to 2168136 tid 2 2026-02-20T11:17:58.250 INFO:tasks.rados.rados.0.trial043.stdout:496: done (5 left) 2026-02-20T11:17:58.250 INFO:tasks.rados.rados.0.trial043.stdout:497: done (4 left) 2026-02-20T11:17:58.250 INFO:tasks.rados.rados.0.trial043.stdout:498: done (3 left) 2026-02-20T11:17:58.250 INFO:tasks.rados.rados.0.trial043.stdout:505: read oid 42 snap 54 2026-02-20T11:17:58.250 INFO:tasks.rados.rados.0.trial043.stdout:505: expect deleted 2026-02-20T11:17:58.250 INFO:tasks.rados.rados.0.trial043.stdout:506: read oid 49 snap -1 2026-02-20T11:17:58.250 INFO:tasks.rados.rados.0.trial043.stdout:506: expect (ObjNum 245 snap 66 seq_num 245) 2026-02-20T11:17:58.250 INFO:tasks.rados.rados.0.trial043.stdout:507: snap_create 2026-02-20T11:17:58.250 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 5 2026-02-20T11:17:58.250 INFO:tasks.rados.rados.0.trial043.stdout:502: expect (ObjNum 152 snap 31 seq_num 152) 2026-02-20T11:17:58.250 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:17:58.284 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2402658 2026-02-20T11:17:58.315 INFO:tasks.rados.rados.0.trial043.stdout:502: done (4 left) 2026-02-20T11:17:58.315 INFO:tasks.rados.rados.0.trial043.stdout:503: done (3 left) 2026-02-20T11:17:58.315 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:17:58.315 INFO:tasks.rados.rados.0.trial043.stdout:505: done (2 left) 2026-02-20T11:17:58.315 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:17:58.315 INFO:tasks.rados.rados.0.trial043.stdout:506: expect (ObjNum 245 snap 66 seq_num 245) 2026-02-20T11:17:58.320 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 610657 2026-02-20T11:17:58.376 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:17:58.414 INFO:tasks.rados.rados.0.trial043.stdout:504: finishing write tid 1 to trial04342147-29 2026-02-20T11:17:58.414 INFO:tasks.rados.rados.0.trial043.stdout:504: finishing write tid 2 to trial04342147-29 2026-02-20T11:17:58.414 INFO:tasks.rados.rados.0.trial043.stdout:504: finishing write tid 3 to trial04342147-29 2026-02-20T11:17:58.414 INFO:tasks.rados.rados.0.trial043.stdout:506: done (1 left) 2026-02-20T11:17:58.414 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:17:58.414 INFO:tasks.rados.rados.0.trial043.stdout:504: finishing write tid 4 to trial04342147-29 2026-02-20T11:17:58.415 INFO:tasks.rados.rados.0.trial043.stdout:504: oid 29 updating version 0 to 136 2026-02-20T11:17:58.415 INFO:tasks.rados.rados.0.trial043.stdout:504: oid 29 version 136 is already newer than 134 2026-02-20T11:17:58.415 INFO:tasks.rados.rados.0.trial043.stdout:504: oid 29 version 136 is already newer than 136 2026-02-20T11:17:58.415 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 29 v 136 (ObjNum 248 snap 66 seq_num 248) dirty exists 2026-02-20T11:17:58.415 INFO:tasks.rados.rados.0.trial043.stdout:504: left oid 29 (ObjNum 248 snap 66 seq_num 248) 2026-02-20T11:17:58.415 INFO:tasks.rados.rados.0.trial043.stdout:504: done (0 left) 2026-02-20T11:17:58.757 INFO:tasks.rados.rados.0.trial043.stdout:507: done (0 left) 2026-02-20T11:17:58.758 INFO:tasks.rados.rados.0.trial043.stdout:508: read oid 9 snap -1 2026-02-20T11:17:58.758 INFO:tasks.rados.rados.0.trial043.stdout:508: expect (ObjNum 223 snap 63 seq_num 223) 2026-02-20T11:17:58.758 INFO:tasks.rados.rados.0.trial043.stdout:509: snap_create 2026-02-20T11:17:58.758 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:17:58.795 INFO:tasks.rados.rados.0.trial043.stdout:508: expect (ObjNum 223 snap 63 seq_num 223) 2026-02-20T11:17:58.904 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:17:58.966 INFO:tasks.rados.rados.0.trial043.stdout:508: done (0 left) 2026-02-20T11:17:59.573 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:17:59.647 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:59.646+0000 7efdada5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:59.650 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:17:59.650+0000 7efdada5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:17:59.679 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:17:59.696 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:17:59.696 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:17:59.858 INFO:tasks.rados.rados.0.trial043.stdout:509: done (0 left) 2026-02-20T11:17:59.858 INFO:tasks.rados.rados.0.trial043.stdout:510: write (excl) oid 24 current snap is 68 2026-02-20T11:17:59.858 INFO:tasks.rados.rados.0.trial043.stdout:510: seq_num 249 ranges {584943=460297,1587781=597162} 2026-02-20T11:17:59.861 INFO:tasks.rados.rados.0.trial043.stdout:510: writing trial04342147-24 from 584943 to 1045240 tid 1 2026-02-20T11:17:59.866 INFO:tasks.rados.rados.0.trial043.stdout:510: writing trial04342147-24 from 1587781 to 2184943 tid 2 2026-02-20T11:17:59.866 INFO:tasks.rados.rados.0.trial043.stdout:511: read oid 4 snap -1 2026-02-20T11:17:59.866 INFO:tasks.rados.rados.0.trial043.stdout:511: expect (ObjNum 226 snap 63 seq_num 226) 2026-02-20T11:17:59.866 INFO:tasks.rados.rados.0.trial043.stdout:512: snap_remove snap 36 2026-02-20T11:18:00.863 INFO:tasks.rados.rados.0.trial043.stdout:512: done (2 left) 2026-02-20T11:18:00.863 INFO:tasks.rados.rados.0.trial043.stdout:513: snap_create 2026-02-20T11:18:00.863 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:18:00.863 INFO:tasks.rados.rados.0.trial043.stdout:510: finishing write tid 1 to trial04342147-24 2026-02-20T11:18:00.863 INFO:tasks.rados.rados.0.trial043.stdout:510: finishing write tid 2 to trial04342147-24 2026-02-20T11:18:00.863 INFO:tasks.rados.rados.0.trial043.stdout:510: finishing write tid 3 to trial04342147-24 2026-02-20T11:18:00.863 INFO:tasks.rados.rados.0.trial043.stdout:510: finishing write tid 4 to trial04342147-24 2026-02-20T11:18:00.864 INFO:tasks.rados.rados.0.trial043.stdout:510: oid 24 updating version 0 to 46 2026-02-20T11:18:00.864 INFO:tasks.rados.rados.0.trial043.stdout:510: oid 24 version 46 is already newer than 46 2026-02-20T11:18:00.864 INFO:tasks.rados.rados.0.trial043.stdout:510: oid 24 version 46 is already newer than 44 2026-02-20T11:18:00.864 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 24 v 46 (ObjNum 249 snap 68 seq_num 249) dirty exists 2026-02-20T11:18:00.864 INFO:tasks.rados.rados.0.trial043.stdout:510: left oid 24 (ObjNum 249 snap 68 seq_num 249) 2026-02-20T11:18:00.864 INFO:tasks.rados.rados.0.trial043.stdout:511: expect (ObjNum 226 snap 63 seq_num 226) 2026-02-20T11:18:00.988 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:01.050 INFO:tasks.rados.rados.0.trial043.stdout:510: done (1 left) 2026-02-20T11:18:01.050 INFO:tasks.rados.rados.0.trial043.stdout:511: done (0 left) 2026-02-20T11:18:01.697 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:18:01.773 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:01.772+0000 7fe1bba11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:01.776 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:01.776+0000 7fe1bba11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:01.806 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:18:01.828 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:18:01.828 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:18:01.864 INFO:tasks.rados.rados.0.trial043.stdout:513: done (0 left) 2026-02-20T11:18:01.864 INFO:tasks.rados.rados.0.trial043.stdout:514: read oid 46 snap -1 2026-02-20T11:18:01.864 INFO:tasks.rados.rados.0.trial043.stdout:514: expect (ObjNum 216 snap 61 seq_num 216) 2026-02-20T11:18:01.864 INFO:tasks.rados.rados.0.trial043.stdout:515: read oid 12 snap 68 2026-02-20T11:18:01.865 INFO:tasks.rados.rados.0.trial043.stdout:515: expect (ObjNum 225 snap 63 seq_num 225) 2026-02-20T11:18:01.865 INFO:tasks.rados.rados.0.trial043.stdout:516: delete oid 18 current snap is 69 2026-02-20T11:18:01.895 INFO:tasks.rados.rados.0.trial043.stdout:515: expect (ObjNum 225 snap 63 seq_num 225) 2026-02-20T11:18:01.895 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:01.933 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2757559 2026-02-20T11:18:01.949 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 931852 2026-02-20T11:18:01.973 INFO:tasks.rados.rados.0.trial043.stdout:515: done (2 left) 2026-02-20T11:18:01.973 INFO:tasks.rados.rados.0.trial043.stdout:516: done (1 left) 2026-02-20T11:18:01.973 INFO:tasks.rados.rados.0.trial043.stdout:517: snap_remove snap 65 2026-02-20T11:18:02.869 INFO:tasks.rados.rados.0.trial043.stdout:517: done (1 left) 2026-02-20T11:18:02.869 INFO:tasks.rados.rados.0.trial043.stdout:518: snap_create 2026-02-20T11:18:02.869 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:18:02.870 INFO:tasks.rados.rados.0.trial043.stdout:514: expect (ObjNum 216 snap 61 seq_num 216) 2026-02-20T11:18:02.934 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~138s 2026-02-20T11:18:02.962 INFO:tasks.rados.rados.0.trial043.stdout:514: done (0 left) 2026-02-20T11:18:03.039 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:18:03.108 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:18:03.111 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:18:03.111 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:18:03.829 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:18:03.871 INFO:tasks.rados.rados.0.trial043.stdout:518: done (0 left) 2026-02-20T11:18:03.871 INFO:tasks.rados.rados.0.trial043.stdout:519: read oid 19 snap -1 2026-02-20T11:18:03.871 INFO:tasks.rados.rados.0.trial043.stdout:519: expect (ObjNum 219 snap 61 seq_num 219) 2026-02-20T11:18:03.871 INFO:tasks.rados.rados.0.trial043.stdout:520: read oid 27 snap -1 2026-02-20T11:18:03.871 INFO:tasks.rados.rados.0.trial043.stdout:520: expect (ObjNum 180 snap 48 seq_num 180) 2026-02-20T11:18:03.872 INFO:tasks.rados.rados.0.trial043.stdout:521: write oid 14 current snap is 70 2026-02-20T11:18:03.872 INFO:tasks.rados.rados.0.trial043.stdout:521: seq_num 251 ranges {618557=727148,2104745=113812} 2026-02-20T11:18:03.876 INFO:tasks.rados.rados.0.trial043.stdout:521: writing trial04342147-14 from 618557 to 1345705 tid 1 2026-02-20T11:18:03.879 INFO:tasks.rados.rados.0.trial043.stdout:521: writing trial04342147-14 from 2104745 to 2218557 tid 2 2026-02-20T11:18:03.879 INFO:tasks.rados.rados.0.trial043.stdout:522: write oid 45 current snap is 70 2026-02-20T11:18:03.879 INFO:tasks.rados.rados.0.trial043.stdout:522: seq_num 252 ranges {635364=402397,1763227=472137} 2026-02-20T11:18:03.883 INFO:tasks.rados.rados.0.trial043.stdout:522: writing trial04342147-45 from 635364 to 1037761 tid 1 2026-02-20T11:18:03.889 INFO:tasks.rados.rados.0.trial043.stdout:522: writing trial04342147-45 from 1763227 to 2235364 tid 2 2026-02-20T11:18:03.889 INFO:tasks.rados.rados.0.trial043.stdout:523: snap_create 2026-02-20T11:18:03.889 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 4 2026-02-20T11:18:03.909 INFO:tasks.rados.rados.0.trial043.stdout:520: expect (ObjNum 180 snap 48 seq_num 180) 2026-02-20T11:18:03.910 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:03.910+0000 7f5c5db0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:03.913 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:03.913+0000 7f5c5db0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:03.956 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:18:03.960 INFO:tasks.rados.rados.0.trial043.stdout:520: done (3 left) 2026-02-20T11:18:03.961 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:18:03.961 INFO:tasks.rados.rados.0.trial043.stdout:519: expect (ObjNum 219 snap 61 seq_num 219) 2026-02-20T11:18:03.965 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 789225 2026-02-20T11:18:03.971 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:18:03.971 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:18:04.076 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:04.136 INFO:tasks.rados.rados.0.trial043.stdout:522: finishing write tid 1 to trial04342147-45 2026-02-20T11:18:04.136 INFO:tasks.rados.rados.0.trial043.stdout:521: finishing write tid 1 to trial04342147-14 2026-02-20T11:18:04.137 INFO:tasks.rados.rados.0.trial043.stdout:519: done (2 left) 2026-02-20T11:18:04.137 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:18:04.137 INFO:tasks.rados.rados.0.trial043.stdout:522: finishing write tid 2 to trial04342147-45 2026-02-20T11:18:04.137 INFO:tasks.rados.rados.0.trial043.stdout:521: finishing write tid 2 to trial04342147-14 2026-02-20T11:18:04.137 INFO:tasks.rados.rados.0.trial043.stdout:522: finishing write tid 3 to trial04342147-45 2026-02-20T11:18:04.137 INFO:tasks.rados.rados.0.trial043.stdout:521: finishing write tid 3 to trial04342147-14 2026-02-20T11:18:04.137 INFO:tasks.rados.rados.0.trial043.stdout:522: finishing write tid 4 to trial04342147-45 2026-02-20T11:18:04.137 INFO:tasks.rados.rados.0.trial043.stdout:522: oid 45 updating version 0 to 158 2026-02-20T11:18:04.137 INFO:tasks.rados.rados.0.trial043.stdout:522: oid 45 version 158 is already newer than 158 2026-02-20T11:18:04.137 INFO:tasks.rados.rados.0.trial043.stdout:522: oid 45 version 158 is already newer than 158 2026-02-20T11:18:04.137 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 45 v 158 (ObjNum 252 snap 70 seq_num 252) dirty exists 2026-02-20T11:18:04.138 INFO:tasks.rados.rados.0.trial043.stdout:522: left oid 45 (ObjNum 252 snap 70 seq_num 252) 2026-02-20T11:18:04.138 INFO:tasks.rados.rados.0.trial043.stdout:521: finishing write tid 4 to trial04342147-14 2026-02-20T11:18:04.138 INFO:tasks.rados.rados.0.trial043.stdout:521: oid 14 updating version 0 to 152 2026-02-20T11:18:04.138 INFO:tasks.rados.rados.0.trial043.stdout:521: oid 14 version 152 is already newer than 152 2026-02-20T11:18:04.138 INFO:tasks.rados.rados.0.trial043.stdout:521: oid 14 version 152 is already newer than 152 2026-02-20T11:18:04.138 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 14 v 152 (ObjNum 251 snap 70 seq_num 251) dirty exists 2026-02-20T11:18:04.138 INFO:tasks.rados.rados.0.trial043.stdout:521: left oid 14 (ObjNum 251 snap 70 seq_num 251) 2026-02-20T11:18:04.138 INFO:tasks.rados.rados.0.trial043.stdout:521: done (1 left) 2026-02-20T11:18:04.138 INFO:tasks.rados.rados.0.trial043.stdout:522: done (0 left) 2026-02-20T11:18:04.876 INFO:tasks.rados.rados.0.trial043.stdout:523: done (0 left) 2026-02-20T11:18:04.876 INFO:tasks.rados.rados.0.trial043.stdout:524: read oid 10 snap -1 2026-02-20T11:18:04.877 INFO:tasks.rados.rados.0.trial043.stdout:524: expect (ObjNum 203 snap 55 seq_num 203) 2026-02-20T11:18:04.877 INFO:tasks.rados.rados.0.trial043.stdout:525: write (excl) oid 19 current snap is 71 2026-02-20T11:18:04.877 INFO:tasks.rados.rados.0.trial043.stdout:525: seq_num 253 ranges {652171=477646,1821709=430462} 2026-02-20T11:18:04.880 INFO:tasks.rados.rados.0.trial043.stdout:525: writing trial04342147-19 from 652171 to 1129817 tid 1 2026-02-20T11:18:04.885 INFO:tasks.rados.rados.0.trial043.stdout:525: writing trial04342147-19 from 1821709 to 2252171 tid 2 2026-02-20T11:18:04.885 INFO:tasks.rados.rados.0.trial043.stdout:526: read oid 49 snap -1 2026-02-20T11:18:04.886 INFO:tasks.rados.rados.0.trial043.stdout:526: expect (ObjNum 245 snap 66 seq_num 245) 2026-02-20T11:18:04.886 INFO:tasks.rados.rados.0.trial043.stdout:527: read oid 6 snap -1 2026-02-20T11:18:04.886 INFO:tasks.rados.rados.0.trial043.stdout:527: expect (ObjNum 244 snap 66 seq_num 244) 2026-02-20T11:18:04.886 INFO:tasks.rados.rados.0.trial043.stdout:528: snap_remove snap 48 2026-02-20T11:18:05.878 INFO:tasks.rados.rados.0.trial043.stdout:528: done (4 left) 2026-02-20T11:18:05.878 INFO:tasks.rados.rados.0.trial043.stdout:529: read oid 44 snap -1 2026-02-20T11:18:05.878 INFO:tasks.rados.rados.0.trial043.stdout:529: expect deleted 2026-02-20T11:18:05.878 INFO:tasks.rados.rados.0.trial043.stdout:530: delete oid 9 current snap is 71 2026-02-20T11:18:05.878 INFO:tasks.rados.rados.0.trial043.stdout:524: expect (ObjNum 203 snap 55 seq_num 203) 2026-02-20T11:18:05.918 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:05.957 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 318430 2026-02-20T11:18:05.972 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:18:05.994 INFO:tasks.rados.rados.0.trial043.stdout:526: expect (ObjNum 245 snap 66 seq_num 245) 2026-02-20T11:18:05.994 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:06.037 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 842551 2026-02-20T11:18:06.046 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:06.045+0000 7f4f5ab0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:06.049 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:06.049+0000 7f4f5ab0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:06.074 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:18:06.090 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:18:06.091 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:18:06.102 INFO:tasks.rados.rados.0.trial043.stdout:527: expect (ObjNum 244 snap 66 seq_num 244) 2026-02-20T11:18:06.103 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:06.119 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:06.137 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:06.155 INFO:tasks.rados.rados.0.trial043.stdout:525: finishing write tid 1 to trial04342147-19 2026-02-20T11:18:06.155 INFO:tasks.rados.rados.0.trial043.stdout:525: finishing write tid 2 to trial04342147-19 2026-02-20T11:18:06.155 INFO:tasks.rados.rados.0.trial043.stdout:525: finishing write tid 3 to trial04342147-19 2026-02-20T11:18:06.155 INFO:tasks.rados.rados.0.trial043.stdout:525: finishing write tid 4 to trial04342147-19 2026-02-20T11:18:06.155 INFO:tasks.rados.rados.0.trial043.stdout:525: oid 19 updating version 0 to 30 2026-02-20T11:18:06.155 INFO:tasks.rados.rados.0.trial043.stdout:525: oid 19 updating version 30 to 32 2026-02-20T11:18:06.155 INFO:tasks.rados.rados.0.trial043.stdout:525: oid 19 version 32 is already newer than 32 2026-02-20T11:18:06.155 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 19 v 32 (ObjNum 253 snap 71 seq_num 253) dirty exists 2026-02-20T11:18:06.155 INFO:tasks.rados.rados.0.trial043.stdout:525: left oid 19 (ObjNum 253 snap 71 seq_num 253) 2026-02-20T11:18:06.155 INFO:tasks.rados.rados.0.trial043.stdout:524: done (5 left) 2026-02-20T11:18:06.155 INFO:tasks.rados.rados.0.trial043.stdout:525: done (4 left) 2026-02-20T11:18:06.155 INFO:tasks.rados.rados.0.trial043.stdout:526: done (3 left) 2026-02-20T11:18:06.156 INFO:tasks.rados.rados.0.trial043.stdout:527: done (2 left) 2026-02-20T11:18:06.156 INFO:tasks.rados.rados.0.trial043.stdout:530: done (1 left) 2026-02-20T11:18:06.156 INFO:tasks.rados.rados.0.trial043.stdout:531: write (excl) oid 43 current snap is 71 2026-02-20T11:18:06.156 INFO:tasks.rados.rados.0.trial043.stdout:531: seq_num 255 ranges {685785=628144,1822320=463465} 2026-02-20T11:18:06.160 INFO:tasks.rados.rados.0.trial043.stdout:531: writing trial04342147-43 from 685785 to 1313929 tid 1 2026-02-20T11:18:06.164 INFO:tasks.rados.rados.0.trial043.stdout:531: writing trial04342147-43 from 1822320 to 2285785 tid 2 2026-02-20T11:18:06.164 INFO:tasks.rados.rados.0.trial043.stdout:529: done (1 left) 2026-02-20T11:18:06.164 INFO:tasks.rados.rados.0.trial043.stdout:532: read oid 12 snap -1 2026-02-20T11:18:06.164 INFO:tasks.rados.rados.0.trial043.stdout:532: expect (ObjNum 225 snap 63 seq_num 225) 2026-02-20T11:18:06.164 INFO:tasks.rados.rados.0.trial043.stdout:533: write oid 50 current snap is 71 2026-02-20T11:18:06.164 INFO:tasks.rados.rados.0.trial043.stdout:533: seq_num 256 ranges {702592=703393,1880802=421790} 2026-02-20T11:18:06.169 INFO:tasks.rados.rados.0.trial043.stdout:533: writing trial04342147-50 from 702592 to 1405985 tid 1 2026-02-20T11:18:06.173 INFO:tasks.rados.rados.0.trial043.stdout:533: writing trial04342147-50 from 1880802 to 2302592 tid 2 2026-02-20T11:18:06.173 INFO:tasks.rados.rados.0.trial043.stdout:534: snap_remove snap 38 2026-02-20T11:18:06.889 INFO:tasks.rados.rados.0.trial043.stdout:534: done (3 left) 2026-02-20T11:18:06.889 INFO:tasks.rados.rados.0.trial043.stdout:535: write (excl) oid 24 current snap is 71 2026-02-20T11:18:06.889 INFO:tasks.rados.rados.0.trial043.stdout:535: seq_num 257 ranges {719399=778642,1939284=380115} 2026-02-20T11:18:06.895 INFO:tasks.rados.rados.0.trial043.stdout:535: writing trial04342147-24 from 719399 to 1498041 tid 1 2026-02-20T11:18:06.899 INFO:tasks.rados.rados.0.trial043.stdout:535: writing trial04342147-24 from 1939284 to 2319399 tid 2 2026-02-20T11:18:06.899 INFO:tasks.rados.rados.0.trial043.stdout:536: write oid 47 current snap is 71 2026-02-20T11:18:06.899 INFO:tasks.rados.rados.0.trial043.stdout:536: seq_num 258 ranges {736206=453891,1881413=454793} 2026-02-20T11:18:06.904 INFO:tasks.rados.rados.0.trial043.stdout:536: writing trial04342147-47 from 736206 to 1190097 tid 1 2026-02-20T11:18:06.909 INFO:tasks.rados.rados.0.trial043.stdout:536: writing trial04342147-47 from 1881413 to 2336206 tid 2 2026-02-20T11:18:06.909 INFO:tasks.rados.rados.0.trial043.stdout:537: read oid 18 snap 39 2026-02-20T11:18:06.909 INFO:tasks.rados.rados.0.trial043.stdout:537: expect (ObjNum 17 snap 0 seq_num 17) 2026-02-20T11:18:06.909 INFO:tasks.rados.rados.0.trial043.stdout:538: snap_remove snap 70 2026-02-20T11:18:07.891 INFO:tasks.rados.rados.0.trial043.stdout:538: done (6 left) 2026-02-20T11:18:07.891 INFO:tasks.rados.rados.0.trial043.stdout:539: snap_create 2026-02-20T11:18:07.891 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 6 2026-02-20T11:18:07.891 INFO:tasks.rados.rados.0.trial043.stdout:532: expect (ObjNum 225 snap 63 seq_num 225) 2026-02-20T11:18:07.897 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2739751 2026-02-20T11:18:07.962 INFO:tasks.rados.rados.0.trial043.stdout:531: finishing write tid 1 to trial04342147-43 2026-02-20T11:18:07.962 INFO:tasks.rados.rados.0.trial043.stdout:531: finishing write tid 2 to trial04342147-43 2026-02-20T11:18:07.962 INFO:tasks.rados.rados.0.trial043.stdout:531: finishing write tid 3 to trial04342147-43 2026-02-20T11:18:07.962 INFO:tasks.rados.rados.0.trial043.stdout:531: finishing write tid 4 to trial04342147-43 2026-02-20T11:18:07.962 INFO:tasks.rados.rados.0.trial043.stdout:531: oid 43 updating version 0 to 154 2026-02-20T11:18:07.962 INFO:tasks.rados.rados.0.trial043.stdout:531: oid 43 version 154 is already newer than 154 2026-02-20T11:18:07.963 INFO:tasks.rados.rados.0.trial043.stdout:531: oid 43 version 154 is already newer than 152 2026-02-20T11:18:07.963 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 43 v 154 (ObjNum 255 snap 71 seq_num 255) dirty exists 2026-02-20T11:18:07.963 INFO:tasks.rados.rados.0.trial043.stdout:531: left oid 43 (ObjNum 255 snap 71 seq_num 255) 2026-02-20T11:18:07.963 INFO:tasks.rados.rados.0.trial043.stdout:533: finishing write tid 1 to trial04342147-50 2026-02-20T11:18:07.963 INFO:tasks.rados.rados.0.trial043.stdout:533: finishing write tid 2 to trial04342147-50 2026-02-20T11:18:07.963 INFO:tasks.rados.rados.0.trial043.stdout:533: finishing write tid 3 to trial04342147-50 2026-02-20T11:18:07.963 INFO:tasks.rados.rados.0.trial043.stdout:533: finishing write tid 4 to trial04342147-50 2026-02-20T11:18:07.963 INFO:tasks.rados.rados.0.trial043.stdout:533: oid 50 updating version 0 to 58 2026-02-20T11:18:07.963 INFO:tasks.rados.rados.0.trial043.stdout:533: oid 50 version 58 is already newer than 58 2026-02-20T11:18:07.963 INFO:tasks.rados.rados.0.trial043.stdout:533: oid 50 version 58 is already newer than 58 2026-02-20T11:18:07.963 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 50 v 58 (ObjNum 256 snap 71 seq_num 256) dirty exists 2026-02-20T11:18:07.964 INFO:tasks.rados.rados.0.trial043.stdout:533: left oid 50 (ObjNum 256 snap 71 seq_num 256) 2026-02-20T11:18:07.964 INFO:tasks.rados.rados.0.trial043.stdout:531: done (5 left) 2026-02-20T11:18:07.964 INFO:tasks.rados.rados.0.trial043.stdout:532: done (4 left) 2026-02-20T11:18:07.964 INFO:tasks.rados.rados.0.trial043.stdout:533: done (3 left) 2026-02-20T11:18:07.964 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:18:07.964 INFO:tasks.rados.rados.0.trial043.stdout:535: finishing write tid 1 to trial04342147-24 2026-02-20T11:18:07.964 INFO:tasks.rados.rados.0.trial043.stdout:537: expect (ObjNum 17 snap 0 seq_num 17) 2026-02-20T11:18:07.964 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:07.980 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 408659 2026-02-20T11:18:08.017 INFO:tasks.rados.rados.0.trial043.stdout:535: finishing write tid 2 to trial04342147-24 2026-02-20T11:18:08.017 INFO:tasks.rados.rados.0.trial043.stdout:536: finishing write tid 1 to trial04342147-47 2026-02-20T11:18:08.017 INFO:tasks.rados.rados.0.trial043.stdout:535: finishing write tid 3 to trial04342147-24 2026-02-20T11:18:08.017 INFO:tasks.rados.rados.0.trial043.stdout:535: finishing write tid 4 to trial04342147-24 2026-02-20T11:18:08.017 INFO:tasks.rados.rados.0.trial043.stdout:535: oid 24 updating version 0 to 50 2026-02-20T11:18:08.017 INFO:tasks.rados.rados.0.trial043.stdout:535: oid 24 version 50 is already newer than 48 2026-02-20T11:18:08.018 INFO:tasks.rados.rados.0.trial043.stdout:535: oid 24 version 50 is already newer than 50 2026-02-20T11:18:08.018 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 24 v 50 (ObjNum 257 snap 71 seq_num 257) dirty exists 2026-02-20T11:18:08.018 INFO:tasks.rados.rados.0.trial043.stdout:535: left oid 24 (ObjNum 257 snap 71 seq_num 257) 2026-02-20T11:18:08.018 INFO:tasks.rados.rados.0.trial043.stdout:536: finishing write tid 2 to trial04342147-47 2026-02-20T11:18:08.018 INFO:tasks.rados.rados.0.trial043.stdout:536: finishing write tid 3 to trial04342147-47 2026-02-20T11:18:08.018 INFO:tasks.rados.rados.0.trial043.stdout:536: finishing write tid 4 to trial04342147-47 2026-02-20T11:18:08.018 INFO:tasks.rados.rados.0.trial043.stdout:536: oid 47 updating version 0 to 81 2026-02-20T11:18:08.018 INFO:tasks.rados.rados.0.trial043.stdout:536: oid 47 version 81 is already newer than 81 2026-02-20T11:18:08.018 INFO:tasks.rados.rados.0.trial043.stdout:536: oid 47 version 81 is already newer than 81 2026-02-20T11:18:08.018 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 47 v 81 (ObjNum 258 snap 71 seq_num 258) dirty exists 2026-02-20T11:18:08.018 INFO:tasks.rados.rados.0.trial043.stdout:536: left oid 47 (ObjNum 258 snap 71 seq_num 258) 2026-02-20T11:18:08.018 INFO:tasks.rados.rados.0.trial043.stdout:535: done (2 left) 2026-02-20T11:18:08.019 INFO:tasks.rados.rados.0.trial043.stdout:536: done (1 left) 2026-02-20T11:18:08.019 INFO:tasks.rados.rados.0.trial043.stdout:537: done (0 left) 2026-02-20T11:18:08.092 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:18:08.112 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:18:08.170 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:08.169+0000 7fc7a0411640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:08.173 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:08.173+0000 7fc7a0411640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:08.184 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:18:08.186 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:18:08.187 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:18:08.198 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:18:08.213 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:18:08.214 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:18:08.439 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~143s 2026-02-20T11:18:08.759 INFO:tasks.rados.rados.0.trial043.stdout:539: done (0 left) 2026-02-20T11:18:08.759 INFO:tasks.rados.rados.0.trial043.stdout:540: delete oid 2 current snap is 72 2026-02-20T11:18:08.802 INFO:tasks.rados.rados.0.trial043.stdout:540: done (0 left) 2026-02-20T11:18:08.802 INFO:tasks.rados.rados.0.trial043.stdout:541: read oid 14 snap -1 2026-02-20T11:18:08.803 INFO:tasks.rados.rados.0.trial043.stdout:541: expect (ObjNum 251 snap 70 seq_num 251) 2026-02-20T11:18:08.803 INFO:tasks.rados.rados.0.trial043.stdout:542: write oid 31 current snap is 72 2026-02-20T11:18:08.803 INFO:tasks.rados.rados.0.trial043.stdout:542: seq_num 260 ranges {769820=720742,2114730=255090} 2026-02-20T11:18:08.808 INFO:tasks.rados.rados.0.trial043.stdout:542: writing trial04342147-31 from 769820 to 1490562 tid 1 2026-02-20T11:18:08.811 INFO:tasks.rados.rados.0.trial043.stdout:542: writing trial04342147-31 from 2114730 to 2369820 tid 2 2026-02-20T11:18:08.811 INFO:tasks.rados.rados.0.trial043.stdout:543: snap_create 2026-02-20T11:18:08.812 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:18:08.818 INFO:tasks.rados.rados.0.trial043.stdout:541: expect (ObjNum 251 snap 70 seq_num 251) 2026-02-20T11:18:08.840 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 687059 2026-02-20T11:18:08.853 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:08.872 INFO:tasks.rados.rados.0.trial043.stdout:542: finishing write tid 1 to trial04342147-31 2026-02-20T11:18:08.873 INFO:tasks.rados.rados.0.trial043.stdout:542: finishing write tid 2 to trial04342147-31 2026-02-20T11:18:08.873 INFO:tasks.rados.rados.0.trial043.stdout:542: finishing write tid 3 to trial04342147-31 2026-02-20T11:18:08.873 INFO:tasks.rados.rados.0.trial043.stdout:542: finishing write tid 4 to trial04342147-31 2026-02-20T11:18:08.873 INFO:tasks.rados.rados.0.trial043.stdout:542: oid 31 updating version 0 to 50 2026-02-20T11:18:08.873 INFO:tasks.rados.rados.0.trial043.stdout:542: oid 31 version 50 is already newer than 50 2026-02-20T11:18:08.873 INFO:tasks.rados.rados.0.trial043.stdout:542: oid 31 version 50 is already newer than 50 2026-02-20T11:18:08.873 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 31 v 50 (ObjNum 260 snap 72 seq_num 260) dirty exists 2026-02-20T11:18:08.873 INFO:tasks.rados.rados.0.trial043.stdout:542: left oid 31 (ObjNum 260 snap 72 seq_num 260) 2026-02-20T11:18:08.873 INFO:tasks.rados.rados.0.trial043.stdout:541: done (1 left) 2026-02-20T11:18:08.874 INFO:tasks.rados.rados.0.trial043.stdout:542: done (0 left) 2026-02-20T11:18:09.760 INFO:tasks.rados.rados.0.trial043.stdout:543: done (0 left) 2026-02-20T11:18:09.761 INFO:tasks.rados.rados.0.trial043.stdout:544: read oid 15 snap 69 2026-02-20T11:18:09.761 INFO:tasks.rados.rados.0.trial043.stdout:544: expect (ObjNum 243 snap 66 seq_num 243) 2026-02-20T11:18:09.761 INFO:tasks.rados.rados.0.trial043.stdout:545: read oid 50 snap -1 2026-02-20T11:18:09.761 INFO:tasks.rados.rados.0.trial043.stdout:545: expect (ObjNum 256 snap 71 seq_num 256) 2026-02-20T11:18:09.761 INFO:tasks.rados.rados.0.trial043.stdout:546: snap_create 2026-02-20T11:18:09.761 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:18:09.795 INFO:tasks.rados.rados.0.trial043.stdout:544: expect (ObjNum 243 snap 66 seq_num 243) 2026-02-20T11:18:09.848 INFO:tasks.rados.rados.0.trial043.stdout:544: done (1 left) 2026-02-20T11:18:09.848 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:18:09.848 INFO:tasks.rados.rados.0.trial043.stdout:545: expect (ObjNum 256 snap 71 seq_num 256) 2026-02-20T11:18:09.897 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:09.923 INFO:tasks.rados.rados.0.trial043.stdout:545: done (0 left) 2026-02-20T11:18:10.214 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:18:10.289 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:10.288+0000 7f04a903d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:10.292 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:10.292+0000 7f04a903d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:10.319 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:18:10.336 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:18:10.336 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:18:10.763 INFO:tasks.rados.rados.0.trial043.stdout:546: done (0 left) 2026-02-20T11:18:10.763 INFO:tasks.rados.rados.0.trial043.stdout:547: write (excl) oid 19 current snap is 74 2026-02-20T11:18:10.763 INFO:tasks.rados.rados.0.trial043.stdout:547: seq_num 261 ranges {786627=795991,2173212=213415} 2026-02-20T11:18:10.769 INFO:tasks.rados.rados.0.trial043.stdout:547: writing trial04342147-19 from 786627 to 1582618 tid 1 2026-02-20T11:18:10.773 INFO:tasks.rados.rados.0.trial043.stdout:547: writing trial04342147-19 from 2173212 to 2386627 tid 2 2026-02-20T11:18:10.773 INFO:tasks.rados.rados.0.trial043.stdout:548: write oid 9 current snap is 74 2026-02-20T11:18:10.773 INFO:tasks.rados.rados.0.trial043.stdout:548: seq_num 262 ranges {403434=471240,1315341=400756} 2026-02-20T11:18:10.776 INFO:tasks.rados.rados.0.trial043.stdout:548: writing trial04342147-9 from 403434 to 874674 tid 1 2026-02-20T11:18:10.779 INFO:tasks.rados.rados.0.trial043.stdout:548: writing trial04342147-9 from 1315341 to 1716097 tid 2 2026-02-20T11:18:10.779 INFO:tasks.rados.rados.0.trial043.stdout:549: delete oid 5 current snap is 74 2026-02-20T11:18:10.807 INFO:tasks.rados.rados.0.trial043.stdout:548: finishing write tid 1 to trial04342147-9 2026-02-20T11:18:10.812 INFO:tasks.rados.rados.0.trial043.stdout:549: done (2 left) 2026-02-20T11:18:10.812 INFO:tasks.rados.rados.0.trial043.stdout:550: snap_remove snap 56 2026-02-20T11:18:11.768 INFO:tasks.rados.rados.0.trial043.stdout:550: done (2 left) 2026-02-20T11:18:11.768 INFO:tasks.rados.rados.0.trial043.stdout:551: read oid 6 snap -1 2026-02-20T11:18:11.768 INFO:tasks.rados.rados.0.trial043.stdout:551: expect (ObjNum 244 snap 66 seq_num 244) 2026-02-20T11:18:11.768 INFO:tasks.rados.rados.0.trial043.stdout:552: snap_create 2026-02-20T11:18:11.768 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:18:11.768 INFO:tasks.rados.rados.0.trial043.stdout:548: finishing write tid 2 to trial04342147-9 2026-02-20T11:18:11.768 INFO:tasks.rados.rados.0.trial043.stdout:548: finishing write tid 3 to trial04342147-9 2026-02-20T11:18:11.769 INFO:tasks.rados.rados.0.trial043.stdout:548: finishing write tid 4 to trial04342147-9 2026-02-20T11:18:11.769 INFO:tasks.rados.rados.0.trial043.stdout:548: oid 9 updating version 0 to 116 2026-02-20T11:18:11.769 INFO:tasks.rados.rados.0.trial043.stdout:548: oid 9 version 116 is already newer than 116 2026-02-20T11:18:11.769 INFO:tasks.rados.rados.0.trial043.stdout:548: oid 9 version 116 is already newer than 116 2026-02-20T11:18:11.769 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 9 v 116 (ObjNum 262 snap 74 seq_num 262) dirty exists 2026-02-20T11:18:11.769 INFO:tasks.rados.rados.0.trial043.stdout:548: left oid 9 (ObjNum 262 snap 74 seq_num 262) 2026-02-20T11:18:11.769 INFO:tasks.rados.rados.0.trial043.stdout:547: finishing write tid 1 to trial04342147-19 2026-02-20T11:18:11.769 INFO:tasks.rados.rados.0.trial043.stdout:548: done (2 left) 2026-02-20T11:18:11.769 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:18:11.769 INFO:tasks.rados.rados.0.trial043.stdout:547: finishing write tid 2 to trial04342147-19 2026-02-20T11:18:11.769 INFO:tasks.rados.rados.0.trial043.stdout:547: finishing write tid 3 to trial04342147-19 2026-02-20T11:18:11.769 INFO:tasks.rados.rados.0.trial043.stdout:547: finishing write tid 4 to trial04342147-19 2026-02-20T11:18:11.770 INFO:tasks.rados.rados.0.trial043.stdout:547: oid 19 updating version 0 to 34 2026-02-20T11:18:11.770 INFO:tasks.rados.rados.0.trial043.stdout:547: oid 19 updating version 34 to 36 2026-02-20T11:18:11.770 INFO:tasks.rados.rados.0.trial043.stdout:547: oid 19 version 36 is already newer than 36 2026-02-20T11:18:11.770 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 19 v 36 (ObjNum 261 snap 74 seq_num 261) dirty exists 2026-02-20T11:18:11.770 INFO:tasks.rados.rados.0.trial043.stdout:547: left oid 19 (ObjNum 261 snap 74 seq_num 261) 2026-02-20T11:18:11.770 INFO:tasks.rados.rados.0.trial043.stdout:547: done (1 left) 2026-02-20T11:18:11.770 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:18:11.794 INFO:tasks.rados.rados.0.trial043.stdout:551: expect (ObjNum 244 snap 66 seq_num 244) 2026-02-20T11:18:11.794 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:11.812 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:11.847 INFO:tasks.rados.rados.0.trial043.stdout:551: done (0 left) 2026-02-20T11:18:12.337 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:18:12.413 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:12.412+0000 7fad8f76f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:12.416 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:12.416+0000 7fad8f76f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:12.442 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:18:12.459 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:18:12.460 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:18:12.772 INFO:tasks.rados.rados.0.trial043.stdout:552: done (0 left) 2026-02-20T11:18:12.772 INFO:tasks.rados.rados.0.trial043.stdout:553: write (excl) oid 25 current snap is 75 2026-02-20T11:18:12.772 INFO:tasks.rados.rados.0.trial043.stdout:553: seq_num 264 ranges {437048=621738,1832305=404425} 2026-02-20T11:18:12.775 INFO:tasks.rados.rados.0.trial043.stdout:553: writing trial04342147-25 from 437048 to 1058786 tid 1 2026-02-20T11:18:12.780 INFO:tasks.rados.rados.0.trial043.stdout:553: writing trial04342147-25 from 1832305 to 2236730 tid 2 2026-02-20T11:18:12.780 INFO:tasks.rados.rados.0.trial043.stdout:554: write oid 45 current snap is 75 2026-02-20T11:18:12.780 INFO:tasks.rados.rados.0.trial043.stdout:554: seq_num 265 ranges {453855=696987,1890787=548083} 2026-02-20T11:18:12.784 INFO:tasks.rados.rados.0.trial043.stdout:554: writing trial04342147-45 from 453855 to 1150842 tid 1 2026-02-20T11:18:12.789 INFO:tasks.rados.rados.0.trial043.stdout:554: writing trial04342147-45 from 1890787 to 2438870 tid 2 2026-02-20T11:18:12.789 INFO:tasks.rados.rados.0.trial043.stdout:555: snap_remove snap 69 2026-02-20T11:18:13.187 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:18:13.260 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:18:13.262 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:18:13.262 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:18:13.771 INFO:tasks.rados.rados.0.trial043.stdout:555: done (2 left) 2026-02-20T11:18:13.771 INFO:tasks.rados.rados.0.trial043.stdout:556: delete oid 5 current snap is 75 2026-02-20T11:18:13.771 INFO:tasks.rados.rados.0.trial043.stdout:553: finishing write tid 1 to trial04342147-25 2026-02-20T11:18:13.771 INFO:tasks.rados.rados.0.trial043.stdout:553: finishing write tid 2 to trial04342147-25 2026-02-20T11:18:13.771 INFO:tasks.rados.rados.0.trial043.stdout:553: finishing write tid 3 to trial04342147-25 2026-02-20T11:18:13.771 INFO:tasks.rados.rados.0.trial043.stdout:553: finishing write tid 4 to trial04342147-25 2026-02-20T11:18:13.771 INFO:tasks.rados.rados.0.trial043.stdout:553: oid 25 updating version 0 to 62 2026-02-20T11:18:13.771 INFO:tasks.rados.rados.0.trial043.stdout:553: oid 25 version 62 is already newer than 60 2026-02-20T11:18:13.771 INFO:tasks.rados.rados.0.trial043.stdout:553: oid 25 version 62 is already newer than 62 2026-02-20T11:18:13.771 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 25 v 62 (ObjNum 264 snap 75 seq_num 264) dirty exists 2026-02-20T11:18:13.771 INFO:tasks.rados.rados.0.trial043.stdout:553: left oid 25 (ObjNum 264 snap 75 seq_num 264) 2026-02-20T11:18:13.771 INFO:tasks.rados.rados.0.trial043.stdout:554: finishing write tid 1 to trial04342147-45 2026-02-20T11:18:13.771 INFO:tasks.rados.rados.0.trial043.stdout:554: finishing write tid 2 to trial04342147-45 2026-02-20T11:18:13.771 INFO:tasks.rados.rados.0.trial043.stdout:554: finishing write tid 3 to trial04342147-45 2026-02-20T11:18:13.771 INFO:tasks.rados.rados.0.trial043.stdout:554: finishing write tid 4 to trial04342147-45 2026-02-20T11:18:13.771 INFO:tasks.rados.rados.0.trial043.stdout:554: oid 45 updating version 0 to 162 2026-02-20T11:18:13.771 INFO:tasks.rados.rados.0.trial043.stdout:554: oid 45 version 162 is already newer than 162 2026-02-20T11:18:13.771 INFO:tasks.rados.rados.0.trial043.stdout:554: oid 45 version 162 is already newer than 162 2026-02-20T11:18:13.771 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 45 v 162 (ObjNum 265 snap 75 seq_num 265) dirty exists 2026-02-20T11:18:13.771 INFO:tasks.rados.rados.0.trial043.stdout:554: left oid 45 (ObjNum 265 snap 75 seq_num 265) 2026-02-20T11:18:13.789 INFO:tasks.rados.rados.0.trial043.stdout:553: done (2 left) 2026-02-20T11:18:13.789 INFO:tasks.rados.rados.0.trial043.stdout:554: done (1 left) 2026-02-20T11:18:13.789 INFO:tasks.rados.rados.0.trial043.stdout:556: done (0 left) 2026-02-20T11:18:13.789 INFO:tasks.rados.rados.0.trial043.stdout:557: write (excl) oid 50 current snap is 75 2026-02-20T11:18:13.789 INFO:tasks.rados.rados.0.trial043.stdout:557: seq_num 267 ranges {487469=563838,1607751=551752} 2026-02-20T11:18:13.793 INFO:tasks.rados.rados.0.trial043.stdout:557: writing trial04342147-50 from 487469 to 1051307 tid 1 2026-02-20T11:18:13.797 INFO:tasks.rados.rados.0.trial043.stdout:557: writing trial04342147-50 from 1607751 to 2159503 tid 2 2026-02-20T11:18:13.797 INFO:tasks.rados.rados.0.trial043.stdout:558: snap_create 2026-02-20T11:18:13.797 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:18:13.819 INFO:tasks.rados.rados.0.trial043.stdout:557: finishing write tid 1 to trial04342147-50 2026-02-20T11:18:13.833 INFO:tasks.rados.rados.0.trial043.stdout:557: finishing write tid 2 to trial04342147-50 2026-02-20T11:18:13.837 INFO:tasks.rados.rados.0.trial043.stdout:557: finishing write tid 3 to trial04342147-50 2026-02-20T11:18:13.839 INFO:tasks.rados.rados.0.trial043.stdout:557: finishing write tid 4 to trial04342147-50 2026-02-20T11:18:13.839 INFO:tasks.rados.rados.0.trial043.stdout:557: oid 50 updating version 0 to 62 2026-02-20T11:18:13.839 INFO:tasks.rados.rados.0.trial043.stdout:557: oid 50 version 62 is already newer than 60 2026-02-20T11:18:13.839 INFO:tasks.rados.rados.0.trial043.stdout:557: oid 50 version 62 is already newer than 62 2026-02-20T11:18:13.839 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 50 v 62 (ObjNum 267 snap 75 seq_num 267) dirty exists 2026-02-20T11:18:13.839 INFO:tasks.rados.rados.0.trial043.stdout:557: left oid 50 (ObjNum 267 snap 75 seq_num 267) 2026-02-20T11:18:13.839 INFO:tasks.rados.rados.0.trial043.stdout:557: done (0 left) 2026-02-20T11:18:13.942 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~149s 2026-02-20T11:18:14.460 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:18:14.537 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:14.536+0000 7f26831dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:14.539 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:14.540+0000 7f26831dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:14.564 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:18:14.580 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:18:14.581 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:18:14.777 INFO:tasks.rados.rados.0.trial043.stdout:558: done (0 left) 2026-02-20T11:18:14.777 INFO:tasks.rados.rados.0.trial043.stdout:559: read oid 17 snap -1 2026-02-20T11:18:14.777 INFO:tasks.rados.rados.0.trial043.stdout:559: expect deleted 2026-02-20T11:18:14.778 INFO:tasks.rados.rados.0.trial043.stdout:560: read oid 30 snap -1 2026-02-20T11:18:14.778 INFO:tasks.rados.rados.0.trial043.stdout:560: expect (ObjNum 221 snap 62 seq_num 221) 2026-02-20T11:18:14.778 INFO:tasks.rados.rados.0.trial043.stdout:561: delete oid 9 current snap is 76 2026-02-20T11:18:14.811 INFO:tasks.rados.rados.0.trial043.stdout:560: expect (ObjNum 221 snap 62 seq_num 221) 2026-02-20T11:18:14.822 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2899276 2026-02-20T11:18:14.832 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 819990 2026-02-20T11:18:14.856 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:14.887 INFO:tasks.rados.rados.0.trial043.stdout:559: done (2 left) 2026-02-20T11:18:14.887 INFO:tasks.rados.rados.0.trial043.stdout:560: done (1 left) 2026-02-20T11:18:14.887 INFO:tasks.rados.rados.0.trial043.stdout:561: done (0 left) 2026-02-20T11:18:14.887 INFO:tasks.rados.rados.0.trial043.stdout:562: delete oid 13 current snap is 76 2026-02-20T11:18:14.890 INFO:tasks.rados.rados.0.trial043.stdout:562: done (0 left) 2026-02-20T11:18:14.890 INFO:tasks.rados.rados.0.trial043.stdout:563: read oid 33 snap -1 2026-02-20T11:18:14.890 INFO:tasks.rados.rados.0.trial043.stdout:563: expect (ObjNum 201 snap 55 seq_num 201) 2026-02-20T11:18:14.890 INFO:tasks.rados.rados.0.trial043.stdout:564: read oid 26 snap -1 2026-02-20T11:18:14.890 INFO:tasks.rados.rados.0.trial043.stdout:564: expect deleted 2026-02-20T11:18:14.890 INFO:tasks.rados.rados.0.trial043.stdout:565: delete oid 45 current snap is 76 2026-02-20T11:18:14.913 INFO:tasks.rados.rados.0.trial043.stdout:563: expect (ObjNum 201 snap 55 seq_num 201) 2026-02-20T11:18:14.913 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:14.955 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 805698 2026-02-20T11:18:15.000 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2520018 2026-02-20T11:18:15.014 INFO:tasks.rados.rados.0.trial043.stdout:563: done (2 left) 2026-02-20T11:18:15.014 INFO:tasks.rados.rados.0.trial043.stdout:564: done (1 left) 2026-02-20T11:18:15.014 INFO:tasks.rados.rados.0.trial043.stdout:565: done (0 left) 2026-02-20T11:18:15.014 INFO:tasks.rados.rados.0.trial043.stdout:566: snap_create 2026-02-20T11:18:15.780 INFO:tasks.rados.rados.0.trial043.stdout:566: done (0 left) 2026-02-20T11:18:15.780 INFO:tasks.rados.rados.0.trial043.stdout:567: snap_remove snap 60 2026-02-20T11:18:16.581 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:18:16.658 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:16.656+0000 7f4ceb28b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:16.660 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:16.660+0000 7f4ceb28b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:16.686 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:18:16.703 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:18:16.703 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:18:16.781 INFO:tasks.rados.rados.0.trial043.stdout:567: done (0 left) 2026-02-20T11:18:16.781 INFO:tasks.rados.rados.0.trial043.stdout:568: snap_remove snap 72 2026-02-20T11:18:17.786 INFO:tasks.rados.rados.0.trial043.stdout:568: done (0 left) 2026-02-20T11:18:17.786 INFO:tasks.rados.rados.0.trial043.stdout:569: write oid 26 current snap is 77 2026-02-20T11:18:17.786 INFO:tasks.rados.rados.0.trial043.stdout:569: seq_num 271 ranges {554697=464834,1725326=559090} 2026-02-20T11:18:17.790 INFO:tasks.rados.rados.0.trial043.stdout:569: writing trial04342147-26 from 554697 to 1019531 tid 1 2026-02-20T11:18:17.796 INFO:tasks.rados.rados.0.trial043.stdout:569: writing trial04342147-26 from 1725326 to 2284416 tid 2 2026-02-20T11:18:17.796 INFO:tasks.rados.rados.0.trial043.stdout:570: read oid 29 snap -1 2026-02-20T11:18:17.796 INFO:tasks.rados.rados.0.trial043.stdout:570: expect (ObjNum 248 snap 66 seq_num 248) 2026-02-20T11:18:17.797 INFO:tasks.rados.rados.0.trial043.stdout:571: delete oid 39 current snap is 77 2026-02-20T11:18:17.802 INFO:tasks.rados.rados.0.trial043.stdout:571: done (2 left) 2026-02-20T11:18:17.802 INFO:tasks.rados.rados.0.trial043.stdout:572: write (excl) oid 41 current snap is 77 2026-02-20T11:18:17.802 INFO:tasks.rados.rados.0.trial043.stdout:572: seq_num 273 ranges {588311=615332,1842290=562759} 2026-02-20T11:18:17.806 INFO:tasks.rados.rados.0.trial043.stdout:572: writing trial04342147-41 from 588311 to 1203643 tid 1 2026-02-20T11:18:17.811 INFO:tasks.rados.rados.0.trial043.stdout:572: writing trial04342147-41 from 1842290 to 2405049 tid 2 2026-02-20T11:18:17.811 INFO:tasks.rados.rados.0.trial043.stdout:573: write oid 42 current snap is 77 2026-02-20T11:18:17.811 INFO:tasks.rados.rados.0.trial043.stdout:573: seq_num 274 ranges {605118=406934,1617125=706417} 2026-02-20T11:18:17.815 INFO:tasks.rados.rados.0.trial043.stdout:573: writing trial04342147-42 from 605118 to 1012052 tid 1 2026-02-20T11:18:17.820 INFO:tasks.rados.rados.0.trial043.stdout:573: writing trial04342147-42 from 1617125 to 2323542 tid 2 2026-02-20T11:18:17.820 INFO:tasks.rados.rados.0.trial043.stdout:574: write (excl) oid 46 current snap is 77 2026-02-20T11:18:17.821 INFO:tasks.rados.rados.0.trial043.stdout:574: seq_num 275 ranges {621925=482183,1559254=566428} 2026-02-20T11:18:17.824 INFO:tasks.rados.rados.0.trial043.stdout:574: writing trial04342147-46 from 621925 to 1104108 tid 1 2026-02-20T11:18:17.828 INFO:tasks.rados.rados.0.trial043.stdout:574: writing trial04342147-46 from 1559254 to 2125682 tid 2 2026-02-20T11:18:17.829 INFO:tasks.rados.rados.0.trial043.stdout:575: write (excl) oid 17 current snap is 77 2026-02-20T11:18:17.829 INFO:tasks.rados.rados.0.trial043.stdout:575: seq_num 276 ranges {638732=557432,1617736=710086} 2026-02-20T11:18:17.833 INFO:tasks.rados.rados.0.trial043.stdout:575: writing trial04342147-17 from 638732 to 1196164 tid 1 2026-02-20T11:18:17.837 INFO:tasks.rados.rados.0.trial043.stdout:575: writing trial04342147-17 from 1617736 to 2327822 tid 2 2026-02-20T11:18:17.837 INFO:tasks.rados.rados.0.trial043.stdout:576: write (excl) oid 10 current snap is 77 2026-02-20T11:18:17.838 INFO:tasks.rados.rados.0.trial043.stdout:576: seq_num 277 ranges {655539=632681,2076218=570097} 2026-02-20T11:18:17.842 INFO:tasks.rados.rados.0.trial043.stdout:576: writing trial04342147-10 from 655539 to 1288220 tid 1 2026-02-20T11:18:17.847 INFO:tasks.rados.rados.0.trial043.stdout:576: writing trial04342147-10 from 2076218 to 2646315 tid 2 2026-02-20T11:18:17.847 INFO:tasks.rados.rados.0.trial043.stdout:577: write (excl) oid 9 current snap is 77 2026-02-20T11:18:17.847 INFO:tasks.rados.rados.0.trial043.stdout:577: seq_num 278 ranges {672346=707930,2134700=537646} 2026-02-20T11:18:17.852 INFO:tasks.rados.rados.0.trial043.stdout:577: writing trial04342147-9 from 672346 to 1380276 tid 1 2026-02-20T11:18:17.857 INFO:tasks.rados.rados.0.trial043.stdout:577: writing trial04342147-9 from 2134700 to 2672346 tid 2 2026-02-20T11:18:17.857 INFO:tasks.rados.rados.0.trial043.stdout:578: write (excl) oid 43 current snap is 77 2026-02-20T11:18:17.857 INFO:tasks.rados.rados.0.trial043.stdout:578: seq_num 279 ranges {689153=783179,2076829=457413} 2026-02-20T11:18:17.863 INFO:tasks.rados.rados.0.trial043.stdout:578: writing trial04342147-43 from 689153 to 1472332 tid 1 2026-02-20T11:18:17.867 INFO:tasks.rados.rados.0.trial043.stdout:578: writing trial04342147-43 from 2076829 to 2534242 tid 2 2026-02-20T11:18:17.867 INFO:tasks.rados.rados.0.trial043.stdout:579: read oid 15 snap 75 2026-02-20T11:18:17.867 INFO:tasks.rados.rados.0.trial043.stdout:579: expect (ObjNum 243 snap 66 seq_num 243) 2026-02-20T11:18:17.867 INFO:tasks.rados.rados.0.trial043.stdout:580: read oid 23 snap -1 2026-02-20T11:18:17.867 INFO:tasks.rados.rados.0.trial043.stdout:580: expect (ObjNum 165 snap 38 seq_num 165) 2026-02-20T11:18:17.867 INFO:tasks.rados.rados.0.trial043.stdout:581: snap_remove snap 67 2026-02-20T11:18:18.263 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:18:18.334 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:18:18.337 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:18:18.337 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:18:18.704 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:18:18.764 INFO:tasks.rados.rados.0.trial043.stdout:581: done (11 left) 2026-02-20T11:18:18.764 INFO:tasks.rados.rados.0.trial043.stdout:582: write oid 8 current snap is 77 2026-02-20T11:18:18.764 INFO:tasks.rados.rados.0.trial043.stdout:573: finishing write tid 1 to trial04342147-42 2026-02-20T11:18:18.764 INFO:tasks.rados.rados.0.trial043.stdout:582: seq_num 280 ranges {705960=458428,1735311=717424} 2026-02-20T11:18:18.768 INFO:tasks.rados.rados.0.trial043.stdout:582: writing trial04342147-8 from 705960 to 1164388 tid 1 2026-02-20T11:18:18.774 INFO:tasks.rados.rados.0.trial043.stdout:582: writing trial04342147-8 from 1735311 to 2452735 tid 2 2026-02-20T11:18:18.775 INFO:tasks.rados.rados.0.trial043.stdout:583: read oid 13 snap -1 2026-02-20T11:18:18.775 INFO:tasks.rados.rados.0.trial043.stdout:583: expect deleted 2026-02-20T11:18:18.775 INFO:tasks.rados.rados.0.trial043.stdout:584: read oid 33 snap -1 2026-02-20T11:18:18.775 INFO:tasks.rados.rados.0.trial043.stdout:584: expect (ObjNum 201 snap 55 seq_num 201) 2026-02-20T11:18:18.775 INFO:tasks.rados.rados.0.trial043.stdout:585: read oid 21 snap -1 2026-02-20T11:18:18.775 INFO:tasks.rados.rados.0.trial043.stdout:585: expect deleted 2026-02-20T11:18:18.775 INFO:tasks.rados.rados.0.trial043.stdout:586: snap_remove snap 66 2026-02-20T11:18:18.785 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:18.783+0000 7f53403f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:18.789 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:18.788+0000 7f53403f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:18.816 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:18:18.850 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:18:18.850 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:18:19.445 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~154s 2026-02-20T11:18:19.764 INFO:tasks.rados.rados.0.trial043.stdout:586: done (15 left) 2026-02-20T11:18:19.764 INFO:tasks.rados.rados.0.trial043.stdout:587: write oid 22 current snap is 77 2026-02-20T11:18:19.764 INFO:tasks.rados.rados.0.trial043.stdout:574: finishing write tid 1 to trial04342147-46 2026-02-20T11:18:19.764 INFO:tasks.rados.rados.0.trial043.stdout:587: seq_num 281 ranges {722767=533677,1793793=461082} 2026-02-20T11:18:19.768 INFO:tasks.rados.rados.0.trial043.stdout:587: writing trial04342147-22 from 722767 to 1256444 tid 1 2026-02-20T11:18:19.773 INFO:tasks.rados.rados.0.trial043.stdout:587: writing trial04342147-22 from 1793793 to 2254875 tid 2 2026-02-20T11:18:19.773 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:18:19.773 INFO:tasks.rados.rados.0.trial043.stdout:573: finishing write tid 2 to trial04342147-42 2026-02-20T11:18:19.773 INFO:tasks.rados.rados.0.trial043.stdout:570: expect (ObjNum 248 snap 66 seq_num 248) 2026-02-20T11:18:19.799 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:19.830 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:19.862 INFO:tasks.rados.rados.0.trial043.stdout:573: finishing write tid 3 to trial04342147-42 2026-02-20T11:18:19.862 INFO:tasks.rados.rados.0.trial043.stdout:570: done (15 left) 2026-02-20T11:18:19.862 INFO:tasks.rados.rados.0.trial043.stdout:588: read oid 3 snap -1 2026-02-20T11:18:19.862 INFO:tasks.rados.rados.0.trial043.stdout:588: expect deleted 2026-02-20T11:18:19.863 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:18:19.863 INFO:tasks.rados.rados.0.trial043.stdout:573: finishing write tid 4 to trial04342147-42 2026-02-20T11:18:19.863 INFO:tasks.rados.rados.0.trial043.stdout:573: oid 42 updating version 0 to 86 2026-02-20T11:18:19.863 INFO:tasks.rados.rados.0.trial043.stdout:573: oid 42 version 86 is already newer than 86 2026-02-20T11:18:19.863 INFO:tasks.rados.rados.0.trial043.stdout:573: oid 42 version 86 is already newer than 86 2026-02-20T11:18:19.863 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 42 v 86 (ObjNum 274 snap 77 seq_num 274) dirty exists 2026-02-20T11:18:19.863 INFO:tasks.rados.rados.0.trial043.stdout:573: left oid 42 (ObjNum 274 snap 77 seq_num 274) 2026-02-20T11:18:19.863 INFO:tasks.rados.rados.0.trial043.stdout:574: finishing write tid 2 to trial04342147-46 2026-02-20T11:18:19.863 INFO:tasks.rados.rados.0.trial043.stdout:576: finishing write tid 1 to trial04342147-10 2026-02-20T11:18:19.863 INFO:tasks.rados.rados.0.trial043.stdout:572: finishing write tid 1 to trial04342147-41 2026-02-20T11:18:19.863 INFO:tasks.rados.rados.0.trial043.stdout:574: finishing write tid 3 to trial04342147-46 2026-02-20T11:18:19.864 INFO:tasks.rados.rados.0.trial043.stdout:573: done (15 left) 2026-02-20T11:18:19.864 INFO:tasks.rados.rados.0.trial043.stdout:589: snap_create 2026-02-20T11:18:19.864 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 15 2026-02-20T11:18:19.864 INFO:tasks.rados.rados.0.trial043.stdout:579: expect (ObjNum 243 snap 66 seq_num 243) 2026-02-20T11:18:19.864 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 285415 2026-02-20T11:18:19.888 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1435715 2026-02-20T11:18:19.920 INFO:tasks.rados.rados.0.trial043.stdout:574: finishing write tid 4 to trial04342147-46 2026-02-20T11:18:19.920 INFO:tasks.rados.rados.0.trial043.stdout:574: oid 46 updating version 0 to 17 2026-02-20T11:18:19.920 INFO:tasks.rados.rados.0.trial043.stdout:574: oid 46 version 17 is already newer than 17 2026-02-20T11:18:19.920 INFO:tasks.rados.rados.0.trial043.stdout:574: oid 46 version 17 is already newer than 15 2026-02-20T11:18:19.920 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 46 v 17 (ObjNum 275 snap 77 seq_num 275) dirty exists 2026-02-20T11:18:19.920 INFO:tasks.rados.rados.0.trial043.stdout:574: left oid 46 (ObjNum 275 snap 77 seq_num 275) 2026-02-20T11:18:19.920 INFO:tasks.rados.rados.0.trial043.stdout:576: finishing write tid 2 to trial04342147-10 2026-02-20T11:18:19.920 INFO:tasks.rados.rados.0.trial043.stdout:572: finishing write tid 2 to trial04342147-41 2026-02-20T11:18:19.921 INFO:tasks.rados.rados.0.trial043.stdout:574: done (14 left) 2026-02-20T11:18:19.921 INFO:tasks.rados.rados.0.trial043.stdout:579: done (13 left) 2026-02-20T11:18:19.921 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 13 2026-02-20T11:18:19.921 INFO:tasks.rados.rados.0.trial043.stdout:569: finishing write tid 1 to trial04342147-26 2026-02-20T11:18:19.921 INFO:tasks.rados.rados.0.trial043.stdout:572: finishing write tid 3 to trial04342147-41 2026-02-20T11:18:19.921 INFO:tasks.rados.rados.0.trial043.stdout:572: finishing write tid 4 to trial04342147-41 2026-02-20T11:18:19.921 INFO:tasks.rados.rados.0.trial043.stdout:572: oid 41 updating version 0 to 128 2026-02-20T11:18:19.921 INFO:tasks.rados.rados.0.trial043.stdout:572: oid 41 version 128 is already newer than 126 2026-02-20T11:18:19.921 INFO:tasks.rados.rados.0.trial043.stdout:572: oid 41 version 128 is already newer than 128 2026-02-20T11:18:19.921 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 41 v 128 (ObjNum 273 snap 77 seq_num 273) dirty exists 2026-02-20T11:18:19.921 INFO:tasks.rados.rados.0.trial043.stdout:572: left oid 41 (ObjNum 273 snap 77 seq_num 273) 2026-02-20T11:18:19.921 INFO:tasks.rados.rados.0.trial043.stdout:576: finishing write tid 3 to trial04342147-10 2026-02-20T11:18:19.922 INFO:tasks.rados.rados.0.trial043.stdout:569: finishing write tid 2 to trial04342147-26 2026-02-20T11:18:19.922 INFO:tasks.rados.rados.0.trial043.stdout:572: done (12 left) 2026-02-20T11:18:19.922 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 12 2026-02-20T11:18:19.922 INFO:tasks.rados.rados.0.trial043.stdout:578: finishing write tid 1 to trial04342147-43 2026-02-20T11:18:19.922 INFO:tasks.rados.rados.0.trial043.stdout:576: finishing write tid 4 to trial04342147-10 2026-02-20T11:18:19.922 INFO:tasks.rados.rados.0.trial043.stdout:576: oid 10 updating version 0 to 68 2026-02-20T11:18:19.922 INFO:tasks.rados.rados.0.trial043.stdout:576: oid 10 version 68 is already newer than 68 2026-02-20T11:18:19.922 INFO:tasks.rados.rados.0.trial043.stdout:576: oid 10 version 68 is already newer than 66 2026-02-20T11:18:19.922 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 10 v 68 (ObjNum 277 snap 77 seq_num 277) dirty exists 2026-02-20T11:18:19.922 INFO:tasks.rados.rados.0.trial043.stdout:576: left oid 10 (ObjNum 277 snap 77 seq_num 277) 2026-02-20T11:18:19.922 INFO:tasks.rados.rados.0.trial043.stdout:576: done (11 left) 2026-02-20T11:18:19.922 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 11 2026-02-20T11:18:19.922 INFO:tasks.rados.rados.0.trial043.stdout:577: finishing write tid 1 to trial04342147-9 2026-02-20T11:18:19.923 INFO:tasks.rados.rados.0.trial043.stdout:569: finishing write tid 3 to trial04342147-26 2026-02-20T11:18:19.923 INFO:tasks.rados.rados.0.trial043.stdout:569: finishing write tid 4 to trial04342147-26 2026-02-20T11:18:19.923 INFO:tasks.rados.rados.0.trial043.stdout:569: oid 26 updating version 0 to 175 2026-02-20T11:18:19.923 INFO:tasks.rados.rados.0.trial043.stdout:569: oid 26 version 175 is already newer than 175 2026-02-20T11:18:19.923 INFO:tasks.rados.rados.0.trial043.stdout:569: oid 26 version 175 is already newer than 175 2026-02-20T11:18:19.923 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 26 v 175 (ObjNum 271 snap 77 seq_num 271) dirty exists 2026-02-20T11:18:19.923 INFO:tasks.rados.rados.0.trial043.stdout:569: left oid 26 (ObjNum 271 snap 77 seq_num 271) 2026-02-20T11:18:19.923 INFO:tasks.rados.rados.0.trial043.stdout:575: finishing write tid 1 to trial04342147-17 2026-02-20T11:18:19.923 INFO:tasks.rados.rados.0.trial043.stdout:578: finishing write tid 2 to trial04342147-43 2026-02-20T11:18:19.923 INFO:tasks.rados.rados.0.trial043.stdout:569: done (10 left) 2026-02-20T11:18:19.923 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 10 2026-02-20T11:18:19.923 INFO:tasks.rados.rados.0.trial043.stdout:577: finishing write tid 2 to trial04342147-9 2026-02-20T11:18:19.924 INFO:tasks.rados.rados.0.trial043.stdout:577: finishing write tid 3 to trial04342147-9 2026-02-20T11:18:19.924 INFO:tasks.rados.rados.0.trial043.stdout:577: finishing write tid 4 to trial04342147-9 2026-02-20T11:18:19.924 INFO:tasks.rados.rados.0.trial043.stdout:577: oid 9 updating version 0 to 129 2026-02-20T11:18:19.924 INFO:tasks.rados.rados.0.trial043.stdout:577: oid 9 updating version 129 to 131 2026-02-20T11:18:19.924 INFO:tasks.rados.rados.0.trial043.stdout:577: oid 9 version 131 is already newer than 131 2026-02-20T11:18:19.924 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 9 v 131 (ObjNum 278 snap 77 seq_num 278) dirty exists 2026-02-20T11:18:19.924 INFO:tasks.rados.rados.0.trial043.stdout:577: left oid 9 (ObjNum 278 snap 77 seq_num 278) 2026-02-20T11:18:19.924 INFO:tasks.rados.rados.0.trial043.stdout:580: expect (ObjNum 165 snap 38 seq_num 165) 2026-02-20T11:18:20.043 INFO:tasks.rados.rados.0.trial043.stdout:575: finishing write tid 2 to trial04342147-17 2026-02-20T11:18:20.044 INFO:tasks.rados.rados.0.trial043.stdout:575: finishing write tid 3 to trial04342147-17 2026-02-20T11:18:20.044 INFO:tasks.rados.rados.0.trial043.stdout:577: done (9 left) 2026-02-20T11:18:20.044 INFO:tasks.rados.rados.0.trial043.stdout:580: done (8 left) 2026-02-20T11:18:20.044 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 8 2026-02-20T11:18:20.044 INFO:tasks.rados.rados.0.trial043.stdout:575: finishing write tid 4 to trial04342147-17 2026-02-20T11:18:20.044 INFO:tasks.rados.rados.0.trial043.stdout:575: oid 17 updating version 0 to 178 2026-02-20T11:18:20.044 INFO:tasks.rados.rados.0.trial043.stdout:575: oid 17 version 178 is already newer than 178 2026-02-20T11:18:20.044 INFO:tasks.rados.rados.0.trial043.stdout:575: oid 17 version 178 is already newer than 176 2026-02-20T11:18:20.045 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 17 v 178 (ObjNum 276 snap 77 seq_num 276) dirty exists 2026-02-20T11:18:20.045 INFO:tasks.rados.rados.0.trial043.stdout:575: left oid 17 (ObjNum 276 snap 77 seq_num 276) 2026-02-20T11:18:20.045 INFO:tasks.rados.rados.0.trial043.stdout:578: finishing write tid 3 to trial04342147-43 2026-02-20T11:18:20.045 INFO:tasks.rados.rados.0.trial043.stdout:578: finishing write tid 4 to trial04342147-43 2026-02-20T11:18:20.045 INFO:tasks.rados.rados.0.trial043.stdout:578: oid 43 updating version 0 to 172 2026-02-20T11:18:20.045 INFO:tasks.rados.rados.0.trial043.stdout:578: oid 43 version 172 is already newer than 170 2026-02-20T11:18:20.045 INFO:tasks.rados.rados.0.trial043.stdout:578: oid 43 version 172 is already newer than 172 2026-02-20T11:18:20.045 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 43 v 172 (ObjNum 279 snap 77 seq_num 279) dirty exists 2026-02-20T11:18:20.045 INFO:tasks.rados.rados.0.trial043.stdout:578: left oid 43 (ObjNum 279 snap 77 seq_num 279) 2026-02-20T11:18:20.045 INFO:tasks.rados.rados.0.trial043.stdout:575: done (7 left) 2026-02-20T11:18:20.045 INFO:tasks.rados.rados.0.trial043.stdout:578: done (6 left) 2026-02-20T11:18:20.045 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 6 2026-02-20T11:18:20.046 INFO:tasks.rados.rados.0.trial043.stdout:583: done (5 left) 2026-02-20T11:18:20.046 INFO:tasks.rados.rados.0.trial043.stdout:585: done (4 left) 2026-02-20T11:18:20.046 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 4 2026-02-20T11:18:20.046 INFO:tasks.rados.rados.0.trial043.stdout:584: expect (ObjNum 201 snap 55 seq_num 201) 2026-02-20T11:18:20.072 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:20.142 INFO:tasks.rados.rados.0.trial043.stdout:582: finishing write tid 1 to trial04342147-8 2026-02-20T11:18:20.142 INFO:tasks.rados.rados.0.trial043.stdout:584: done (3 left) 2026-02-20T11:18:20.143 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:18:20.143 INFO:tasks.rados.rados.0.trial043.stdout:582: finishing write tid 2 to trial04342147-8 2026-02-20T11:18:20.143 INFO:tasks.rados.rados.0.trial043.stdout:582: finishing write tid 3 to trial04342147-8 2026-02-20T11:18:20.143 INFO:tasks.rados.rados.0.trial043.stdout:582: finishing write tid 4 to trial04342147-8 2026-02-20T11:18:20.143 INFO:tasks.rados.rados.0.trial043.stdout:582: oid 8 updating version 0 to 66 2026-02-20T11:18:20.143 INFO:tasks.rados.rados.0.trial043.stdout:582: oid 8 version 66 is already newer than 66 2026-02-20T11:18:20.143 INFO:tasks.rados.rados.0.trial043.stdout:582: oid 8 version 66 is already newer than 66 2026-02-20T11:18:20.143 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 8 v 66 (ObjNum 280 snap 77 seq_num 280) dirty exists 2026-02-20T11:18:20.143 INFO:tasks.rados.rados.0.trial043.stdout:582: left oid 8 (ObjNum 280 snap 77 seq_num 280) 2026-02-20T11:18:20.143 INFO:tasks.rados.rados.0.trial043.stdout:587: finishing write tid 1 to trial04342147-22 2026-02-20T11:18:20.143 INFO:tasks.rados.rados.0.trial043.stdout:587: finishing write tid 2 to trial04342147-22 2026-02-20T11:18:20.144 INFO:tasks.rados.rados.0.trial043.stdout:582: done (2 left) 2026-02-20T11:18:20.144 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:18:20.144 INFO:tasks.rados.rados.0.trial043.stdout:587: finishing write tid 3 to trial04342147-22 2026-02-20T11:18:20.144 INFO:tasks.rados.rados.0.trial043.stdout:587: finishing write tid 4 to trial04342147-22 2026-02-20T11:18:20.144 INFO:tasks.rados.rados.0.trial043.stdout:587: oid 22 updating version 0 to 70 2026-02-20T11:18:20.144 INFO:tasks.rados.rados.0.trial043.stdout:587: oid 22 version 70 is already newer than 70 2026-02-20T11:18:20.144 INFO:tasks.rados.rados.0.trial043.stdout:587: oid 22 version 70 is already newer than 70 2026-02-20T11:18:20.144 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 22 v 70 (ObjNum 281 snap 77 seq_num 281) dirty exists 2026-02-20T11:18:20.144 INFO:tasks.rados.rados.0.trial043.stdout:587: left oid 22 (ObjNum 281 snap 77 seq_num 281) 2026-02-20T11:18:20.144 INFO:tasks.rados.rados.0.trial043.stdout:587: done (1 left) 2026-02-20T11:18:20.144 INFO:tasks.rados.rados.0.trial043.stdout:588: done (0 left) 2026-02-20T11:18:20.795 INFO:tasks.rados.rados.0.trial043.stdout:589: done (0 left) 2026-02-20T11:18:20.796 INFO:tasks.rados.rados.0.trial043.stdout:590: write oid 9 current snap is 78 2026-02-20T11:18:20.796 INFO:tasks.rados.rados.0.trial043.stdout:590: seq_num 282 ranges {739574=725279,1968628=721093} 2026-02-20T11:18:20.802 INFO:tasks.rados.rados.0.trial043.stdout:590: writing trial04342147-9 from 739574 to 1464853 tid 1 2026-02-20T11:18:20.808 INFO:tasks.rados.rados.0.trial043.stdout:590: writing trial04342147-9 from 1968628 to 2689721 tid 2 2026-02-20T11:18:20.808 INFO:tasks.rados.rados.0.trial043.stdout:591: read oid 33 snap -1 2026-02-20T11:18:20.808 INFO:tasks.rados.rados.0.trial043.stdout:591: expect (ObjNum 201 snap 55 seq_num 201) 2026-02-20T11:18:20.808 INFO:tasks.rados.rados.0.trial043.stdout:592: write (excl) oid 25 current snap is 78 2026-02-20T11:18:20.808 INFO:tasks.rados.rados.0.trial043.stdout:592: seq_num 283 ranges {756381=400528,1910757=464751} 2026-02-20T11:18:20.812 INFO:tasks.rados.rados.0.trial043.stdout:592: writing trial04342147-25 from 756381 to 1156909 tid 1 2026-02-20T11:18:20.817 INFO:tasks.rados.rados.0.trial043.stdout:592: writing trial04342147-25 from 1910757 to 2375508 tid 2 2026-02-20T11:18:20.817 INFO:tasks.rados.rados.0.trial043.stdout:593: write oid 13 current snap is 78 2026-02-20T11:18:20.817 INFO:tasks.rados.rados.0.trial043.stdout:593: seq_num 284 ranges {773188=475777,1969239=724762} 2026-02-20T11:18:20.821 INFO:tasks.rados.rados.0.trial043.stdout:593: writing trial04342147-13 from 773188 to 1248965 tid 1 2026-02-20T11:18:20.827 INFO:tasks.rados.rados.0.trial043.stdout:593: writing trial04342147-13 from 1969239 to 2694001 tid 2 2026-02-20T11:18:20.827 INFO:tasks.rados.rados.0.trial043.stdout:594: snap_create 2026-02-20T11:18:20.827 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 4 2026-02-20T11:18:20.828 INFO:tasks.rados.rados.0.trial043.stdout:591: expect (ObjNum 201 snap 55 seq_num 201) 2026-02-20T11:18:20.851 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:18:20.856 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 336645 2026-02-20T11:18:20.893 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:20.927 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:20.925+0000 7f10ff97e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:20.930 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:20.929+0000 7f10ff97e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:20.932 INFO:tasks.rados.rados.0.trial043.stdout:591: done (3 left) 2026-02-20T11:18:20.932 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:18:20.933 INFO:tasks.rados.rados.0.trial043.stdout:593: finishing write tid 1 to trial04342147-13 2026-02-20T11:18:20.933 INFO:tasks.rados.rados.0.trial043.stdout:590: finishing write tid 1 to trial04342147-9 2026-02-20T11:18:20.933 INFO:tasks.rados.rados.0.trial043.stdout:593: finishing write tid 2 to trial04342147-13 2026-02-20T11:18:20.933 INFO:tasks.rados.rados.0.trial043.stdout:590: finishing write tid 2 to trial04342147-9 2026-02-20T11:18:20.933 INFO:tasks.rados.rados.0.trial043.stdout:593: finishing write tid 3 to trial04342147-13 2026-02-20T11:18:20.933 INFO:tasks.rados.rados.0.trial043.stdout:593: finishing write tid 4 to trial04342147-13 2026-02-20T11:18:20.933 INFO:tasks.rados.rados.0.trial043.stdout:593: oid 13 updating version 0 to 167 2026-02-20T11:18:20.933 INFO:tasks.rados.rados.0.trial043.stdout:593: oid 13 version 167 is already newer than 167 2026-02-20T11:18:20.933 INFO:tasks.rados.rados.0.trial043.stdout:593: oid 13 version 167 is already newer than 167 2026-02-20T11:18:20.933 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 13 v 167 (ObjNum 284 snap 78 seq_num 284) dirty exists 2026-02-20T11:18:20.933 INFO:tasks.rados.rados.0.trial043.stdout:593: left oid 13 (ObjNum 284 snap 78 seq_num 284) 2026-02-20T11:18:20.933 INFO:tasks.rados.rados.0.trial043.stdout:590: finishing write tid 3 to trial04342147-9 2026-02-20T11:18:20.934 INFO:tasks.rados.rados.0.trial043.stdout:593: done (2 left) 2026-02-20T11:18:20.934 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:18:20.934 INFO:tasks.rados.rados.0.trial043.stdout:590: finishing write tid 4 to trial04342147-9 2026-02-20T11:18:20.934 INFO:tasks.rados.rados.0.trial043.stdout:590: oid 9 updating version 0 to 139 2026-02-20T11:18:20.934 INFO:tasks.rados.rados.0.trial043.stdout:590: oid 9 version 139 is already newer than 139 2026-02-20T11:18:20.934 INFO:tasks.rados.rados.0.trial043.stdout:590: oid 9 version 139 is already newer than 139 2026-02-20T11:18:20.934 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 9 v 139 (ObjNum 282 snap 78 seq_num 282) dirty exists 2026-02-20T11:18:20.934 INFO:tasks.rados.rados.0.trial043.stdout:590: left oid 9 (ObjNum 282 snap 78 seq_num 282) 2026-02-20T11:18:20.934 INFO:tasks.rados.rados.0.trial043.stdout:592: finishing write tid 1 to trial04342147-25 2026-02-20T11:18:20.934 INFO:tasks.rados.rados.0.trial043.stdout:592: finishing write tid 2 to trial04342147-25 2026-02-20T11:18:20.934 INFO:tasks.rados.rados.0.trial043.stdout:592: finishing write tid 3 to trial04342147-25 2026-02-20T11:18:20.934 INFO:tasks.rados.rados.0.trial043.stdout:590: done (1 left) 2026-02-20T11:18:20.935 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:18:20.935 INFO:tasks.rados.rados.0.trial043.stdout:592: finishing write tid 4 to trial04342147-25 2026-02-20T11:18:20.935 INFO:tasks.rados.rados.0.trial043.stdout:592: oid 25 updating version 0 to 66 2026-02-20T11:18:20.936 INFO:tasks.rados.rados.0.trial043.stdout:592: oid 25 version 66 is already newer than 66 2026-02-20T11:18:20.936 INFO:tasks.rados.rados.0.trial043.stdout:592: oid 25 version 66 is already newer than 64 2026-02-20T11:18:20.936 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 25 v 66 (ObjNum 283 snap 78 seq_num 283) dirty exists 2026-02-20T11:18:20.936 INFO:tasks.rados.rados.0.trial043.stdout:592: left oid 25 (ObjNum 283 snap 78 seq_num 283) 2026-02-20T11:18:20.936 INFO:tasks.rados.rados.0.trial043.stdout:592: done (0 left) 2026-02-20T11:18:20.955 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:18:20.970 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:18:20.970 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:18:21.798 INFO:tasks.rados.rados.0.trial043.stdout:594: done (0 left) 2026-02-20T11:18:21.798 INFO:tasks.rados.rados.0.trial043.stdout:595: read oid 42 snap -1 2026-02-20T11:18:21.798 INFO:tasks.rados.rados.0.trial043.stdout:595: expect (ObjNum 274 snap 77 seq_num 274) 2026-02-20T11:18:21.798 INFO:tasks.rados.rados.0.trial043.stdout:596: snap_remove snap 74 2026-02-20T11:18:22.804 INFO:tasks.rados.rados.0.trial043.stdout:596: done (1 left) 2026-02-20T11:18:22.804 INFO:tasks.rados.rados.0.trial043.stdout:597: delete oid 35 current snap is 79 2026-02-20T11:18:22.805 INFO:tasks.rados.rados.0.trial043.stdout:595: expect (ObjNum 274 snap 77 seq_num 274) 2026-02-20T11:18:22.805 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:22.847 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:22.870 INFO:tasks.rados.rados.0.trial043.stdout:595: done (1 left) 2026-02-20T11:18:22.870 INFO:tasks.rados.rados.0.trial043.stdout:597: done (0 left) 2026-02-20T11:18:22.870 INFO:tasks.rados.rados.0.trial043.stdout:598: delete oid 37 current snap is 79 2026-02-20T11:18:22.896 INFO:tasks.rados.rados.0.trial043.stdout:598: done (0 left) 2026-02-20T11:18:22.896 INFO:tasks.rados.rados.0.trial043.stdout:599: delete oid 15 current snap is 79 2026-02-20T11:18:22.925 INFO:tasks.rados.rados.0.trial043.stdout:599: done (0 left) 2026-02-20T11:18:22.925 INFO:tasks.rados.rados.0.trial043.stdout:600: read oid 44 snap -1 2026-02-20T11:18:22.926 INFO:tasks.rados.rados.0.trial043.stdout:600: expect deleted 2026-02-20T11:18:22.926 INFO:tasks.rados.rados.0.trial043.stdout:601: snap_remove snap 61 2026-02-20T11:18:22.971 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:18:23.047 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:23.045+0000 7fea7910b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:23.050 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:23.050+0000 7fea7910b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:23.075 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:18:23.092 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:18:23.092 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:18:23.338 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:18:23.410 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:18:23.412 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:18:23.413 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:18:23.808 INFO:tasks.rados.rados.0.trial043.stdout:601: done (1 left) 2026-02-20T11:18:23.808 INFO:tasks.rados.rados.0.trial043.stdout:602: read oid 20 snap -1 2026-02-20T11:18:23.808 INFO:tasks.rados.rados.0.trial043.stdout:602: expect (ObjNum 128 snap 17 seq_num 128) 2026-02-20T11:18:23.809 INFO:tasks.rados.rados.0.trial043.stdout:603: snap_create 2026-02-20T11:18:23.809 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:18:23.809 INFO:tasks.rados.rados.0.trial043.stdout:600: done (1 left) 2026-02-20T11:18:23.809 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:18:23.857 INFO:tasks.rados.rados.0.trial043.stdout:602: expect (ObjNum 128 snap 17 seq_num 128) 2026-02-20T11:18:23.883 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:23.933 INFO:tasks.rados.rados.0.trial043.stdout:602: done (0 left) 2026-02-20T11:18:24.811 INFO:tasks.rados.rados.0.trial043.stdout:603: done (0 left) 2026-02-20T11:18:24.811 INFO:tasks.rados.rados.0.trial043.stdout:604: snap_remove snap 78 2026-02-20T11:18:24.950 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~160s 2026-02-20T11:18:25.093 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:18:25.171 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:25.169+0000 7f7b3c45a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:25.173 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:25.172+0000 7f7b3c45a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:25.199 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:18:25.214 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:18:25.215 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:18:25.812 INFO:tasks.rados.rados.0.trial043.stdout:604: done (0 left) 2026-02-20T11:18:25.812 INFO:tasks.rados.rados.0.trial043.stdout:605: write (excl) oid 28 current snap is 80 2026-02-20T11:18:25.812 INFO:tasks.rados.rados.0.trial043.stdout:605: seq_num 288 ranges {440416=776773,1686814=732100} 2026-02-20T11:18:25.816 INFO:tasks.rados.rados.0.trial043.stdout:605: writing trial04342147-28 from 440416 to 1217189 tid 1 2026-02-20T11:18:25.821 INFO:tasks.rados.rados.0.trial043.stdout:605: writing trial04342147-28 from 1686814 to 2418914 tid 2 2026-02-20T11:18:25.821 INFO:tasks.rados.rados.0.trial043.stdout:606: snap_create 2026-02-20T11:18:25.821 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:18:25.858 INFO:tasks.rados.rados.0.trial043.stdout:605: finishing write tid 1 to trial04342147-28 2026-02-20T11:18:25.875 INFO:tasks.rados.rados.0.trial043.stdout:605: finishing write tid 2 to trial04342147-28 2026-02-20T11:18:25.879 INFO:tasks.rados.rados.0.trial043.stdout:605: finishing write tid 3 to trial04342147-28 2026-02-20T11:18:25.881 INFO:tasks.rados.rados.0.trial043.stdout:605: finishing write tid 4 to trial04342147-28 2026-02-20T11:18:25.882 INFO:tasks.rados.rados.0.trial043.stdout:605: oid 28 updating version 0 to 68 2026-02-20T11:18:25.882 INFO:tasks.rados.rados.0.trial043.stdout:605: oid 28 updating version 68 to 70 2026-02-20T11:18:25.882 INFO:tasks.rados.rados.0.trial043.stdout:605: oid 28 version 70 is already newer than 70 2026-02-20T11:18:25.882 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 28 v 70 (ObjNum 288 snap 80 seq_num 288) dirty exists 2026-02-20T11:18:25.882 INFO:tasks.rados.rados.0.trial043.stdout:605: left oid 28 (ObjNum 288 snap 80 seq_num 288) 2026-02-20T11:18:25.882 INFO:tasks.rados.rados.0.trial043.stdout:605: done (0 left) 2026-02-20T11:18:26.818 INFO:tasks.rados.rados.0.trial043.stdout:606: done (0 left) 2026-02-20T11:18:26.818 INFO:tasks.rados.rados.0.trial043.stdout:607: delete oid 11 current snap is 81 2026-02-20T11:18:26.866 INFO:tasks.rados.rados.0.trial043.stdout:607: done (0 left) 2026-02-20T11:18:26.866 INFO:tasks.rados.rados.0.trial043.stdout:608: snap_remove snap 39 2026-02-20T11:18:27.216 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:18:27.292 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:27.290+0000 7f110b19c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:27.294 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:27.293+0000 7f110b19c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:27.320 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:18:27.336 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:18:27.336 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:18:27.821 INFO:tasks.rados.rados.0.trial043.stdout:608: done (0 left) 2026-02-20T11:18:27.822 INFO:tasks.rados.rados.0.trial043.stdout:609: read oid 44 snap -1 2026-02-20T11:18:27.822 INFO:tasks.rados.rados.0.trial043.stdout:609: expect deleted 2026-02-20T11:18:27.822 INFO:tasks.rados.rados.0.trial043.stdout:610: delete oid 45 current snap is 81 2026-02-20T11:18:27.835 INFO:tasks.rados.rados.0.trial043.stdout:610: done (1 left) 2026-02-20T11:18:27.836 INFO:tasks.rados.rados.0.trial043.stdout:611: write (excl) oid 43 current snap is 81 2026-02-20T11:18:27.836 INFO:tasks.rados.rados.0.trial043.stdout:611: seq_num 291 ranges {490837=718873,1862260=479427,2875032=15805} 2026-02-20T11:18:27.839 INFO:tasks.rados.rados.0.trial043.stdout:611: writing trial04342147-43 from 490837 to 1209710 tid 1 2026-02-20T11:18:27.844 INFO:tasks.rados.rados.0.trial043.stdout:611: writing trial04342147-43 from 1862260 to 2341687 tid 2 2026-02-20T11:18:27.846 INFO:tasks.rados.rados.0.trial043.stdout:611: writing trial04342147-43 from 2875032 to 2890837 tid 3 2026-02-20T11:18:27.846 INFO:tasks.rados.rados.0.trial043.stdout:612: write oid 4 current snap is 81 2026-02-20T11:18:27.847 INFO:tasks.rados.rados.0.trial043.stdout:612: seq_num 292 ranges {507644=794122,1920742=623085} 2026-02-20T11:18:27.850 INFO:tasks.rados.rados.0.trial043.stdout:612: writing trial04342147-4 from 507644 to 1301766 tid 1 2026-02-20T11:18:27.856 INFO:tasks.rados.rados.0.trial043.stdout:612: writing trial04342147-4 from 1920742 to 2543827 tid 2 2026-02-20T11:18:27.856 INFO:tasks.rados.rados.0.trial043.stdout:613: read oid 29 snap -1 2026-02-20T11:18:27.856 INFO:tasks.rados.rados.0.trial043.stdout:613: expect (ObjNum 248 snap 66 seq_num 248) 2026-02-20T11:18:27.856 INFO:tasks.rados.rados.0.trial043.stdout:614: read oid 49 snap -1 2026-02-20T11:18:27.856 INFO:tasks.rados.rados.0.trial043.stdout:614: expect (ObjNum 245 snap 66 seq_num 245) 2026-02-20T11:18:27.856 INFO:tasks.rados.rados.0.trial043.stdout:615: write oid 8 current snap is 81 2026-02-20T11:18:27.857 INFO:tasks.rados.rados.0.trial043.stdout:615: seq_num 293 ranges {524451=469371,1579224=483096,2529878=394573} 2026-02-20T11:18:27.859 INFO:tasks.rados.rados.0.trial043.stdout:615: writing trial04342147-8 from 524451 to 993822 tid 1 2026-02-20T11:18:27.863 INFO:tasks.rados.rados.0.trial043.stdout:615: writing trial04342147-8 from 1579224 to 2062320 tid 2 2026-02-20T11:18:27.867 INFO:tasks.rados.rados.0.trial043.stdout:615: writing trial04342147-8 from 2529878 to 2924451 tid 3 2026-02-20T11:18:27.867 INFO:tasks.rados.rados.0.trial043.stdout:616: read oid 36 snap -1 2026-02-20T11:18:27.867 INFO:tasks.rados.rados.0.trial043.stdout:616: expect (ObjNum 195 snap 52 seq_num 195) 2026-02-20T11:18:27.867 INFO:tasks.rados.rados.0.trial043.stdout:617: delete oid 2 current snap is 81 2026-02-20T11:18:27.875 INFO:tasks.rados.rados.0.trial043.stdout:609: done (7 left) 2026-02-20T11:18:27.875 INFO:tasks.rados.rados.0.trial043.stdout:617: done (6 left) 2026-02-20T11:18:27.875 INFO:tasks.rados.rados.0.trial043.stdout:618: delete oid 47 current snap is 81 2026-02-20T11:18:27.889 INFO:tasks.rados.rados.0.trial043.stdout:614: expect (ObjNum 245 snap 66 seq_num 245) 2026-02-20T11:18:27.889 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 189194 2026-02-20T11:18:28.001 INFO:tasks.rados.rados.0.trial043.stdout:616: expect (ObjNum 195 snap 52 seq_num 195) 2026-02-20T11:18:28.044 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:28.086 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:28.130 INFO:tasks.rados.rados.0.trial043.stdout:612: finishing write tid 1 to trial04342147-4 2026-02-20T11:18:28.130 INFO:tasks.rados.rados.0.trial043.stdout:614: done (6 left) 2026-02-20T11:18:28.130 INFO:tasks.rados.rados.0.trial043.stdout:616: done (5 left) 2026-02-20T11:18:28.130 INFO:tasks.rados.rados.0.trial043.stdout:618: done (4 left) 2026-02-20T11:18:28.130 INFO:tasks.rados.rados.0.trial043.stdout:619: delete oid 18 current snap is 81 2026-02-20T11:18:28.131 INFO:tasks.rados.rados.0.trial043.stdout:612: finishing write tid 2 to trial04342147-4 2026-02-20T11:18:28.131 INFO:tasks.rados.rados.0.trial043.stdout:612: finishing write tid 3 to trial04342147-4 2026-02-20T11:18:28.131 INFO:tasks.rados.rados.0.trial043.stdout:612: finishing write tid 4 to trial04342147-4 2026-02-20T11:18:28.131 INFO:tasks.rados.rados.0.trial043.stdout:612: oid 4 updating version 0 to 74 2026-02-20T11:18:28.131 INFO:tasks.rados.rados.0.trial043.stdout:612: oid 4 version 74 is already newer than 74 2026-02-20T11:18:28.131 INFO:tasks.rados.rados.0.trial043.stdout:612: oid 4 version 74 is already newer than 74 2026-02-20T11:18:28.131 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 4 v 74 (ObjNum 292 snap 81 seq_num 292) dirty exists 2026-02-20T11:18:28.131 INFO:tasks.rados.rados.0.trial043.stdout:612: left oid 4 (ObjNum 292 snap 81 seq_num 292) 2026-02-20T11:18:28.131 INFO:tasks.rados.rados.0.trial043.stdout:615: finishing write tid 1 to trial04342147-8 2026-02-20T11:18:28.131 INFO:tasks.rados.rados.0.trial043.stdout:615: finishing write tid 2 to trial04342147-8 2026-02-20T11:18:28.131 INFO:tasks.rados.rados.0.trial043.stdout:615: finishing write tid 3 to trial04342147-8 2026-02-20T11:18:28.131 INFO:tasks.rados.rados.0.trial043.stdout:615: finishing write tid 4 to trial04342147-8 2026-02-20T11:18:28.132 INFO:tasks.rados.rados.0.trial043.stdout:615: finishing write tid 5 to trial04342147-8 2026-02-20T11:18:28.132 INFO:tasks.rados.rados.0.trial043.stdout:615: oid 8 updating version 0 to 79 2026-02-20T11:18:28.132 INFO:tasks.rados.rados.0.trial043.stdout:615: oid 8 version 79 is already newer than 79 2026-02-20T11:18:28.132 INFO:tasks.rados.rados.0.trial043.stdout:615: oid 8 version 79 is already newer than 79 2026-02-20T11:18:28.132 INFO:tasks.rados.rados.0.trial043.stdout:615: oid 8 version 79 is already newer than 79 2026-02-20T11:18:28.132 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 8 v 79 (ObjNum 293 snap 81 seq_num 293) dirty exists 2026-02-20T11:18:28.132 INFO:tasks.rados.rados.0.trial043.stdout:615: left oid 8 (ObjNum 293 snap 81 seq_num 293) 2026-02-20T11:18:28.132 INFO:tasks.rados.rados.0.trial043.stdout:611: finishing write tid 1 to trial04342147-43 2026-02-20T11:18:28.132 INFO:tasks.rados.rados.0.trial043.stdout:611: finishing write tid 2 to trial04342147-43 2026-02-20T11:18:28.132 INFO:tasks.rados.rados.0.trial043.stdout:611: finishing write tid 3 to trial04342147-43 2026-02-20T11:18:28.132 INFO:tasks.rados.rados.0.trial043.stdout:611: finishing write tid 4 to trial04342147-43 2026-02-20T11:18:28.132 INFO:tasks.rados.rados.0.trial043.stdout:611: finishing write tid 5 to trial04342147-43 2026-02-20T11:18:28.132 INFO:tasks.rados.rados.0.trial043.stdout:611: oid 43 updating version 0 to 193 2026-02-20T11:18:28.133 INFO:tasks.rados.rados.0.trial043.stdout:611: oid 43 version 193 is already newer than 193 2026-02-20T11:18:28.133 INFO:tasks.rados.rados.0.trial043.stdout:611: oid 43 version 193 is already newer than 190 2026-02-20T11:18:28.133 INFO:tasks.rados.rados.0.trial043.stdout:611: oid 43 version 193 is already newer than 193 2026-02-20T11:18:28.133 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 43 v 193 (ObjNum 291 snap 81 seq_num 291) dirty exists 2026-02-20T11:18:28.133 INFO:tasks.rados.rados.0.trial043.stdout:611: left oid 43 (ObjNum 291 snap 81 seq_num 291) 2026-02-20T11:18:28.133 INFO:tasks.rados.rados.0.trial043.stdout:613: expect (ObjNum 248 snap 66 seq_num 248) 2026-02-20T11:18:28.161 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 483070 2026-02-20T11:18:28.194 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1278840 2026-02-20T11:18:28.213 INFO:tasks.rados.rados.0.trial043.stdout:611: done (4 left) 2026-02-20T11:18:28.214 INFO:tasks.rados.rados.0.trial043.stdout:612: done (3 left) 2026-02-20T11:18:28.214 INFO:tasks.rados.rados.0.trial043.stdout:613: done (2 left) 2026-02-20T11:18:28.214 INFO:tasks.rados.rados.0.trial043.stdout:615: done (1 left) 2026-02-20T11:18:28.214 INFO:tasks.rados.rados.0.trial043.stdout:619: done (0 left) 2026-02-20T11:18:28.214 INFO:tasks.rados.rados.0.trial043.stdout:620: delete oid 39 current snap is 81 2026-02-20T11:18:28.216 INFO:tasks.rados.rados.0.trial043.stdout:620: done (0 left) 2026-02-20T11:18:28.216 INFO:tasks.rados.rados.0.trial043.stdout:621: snap_remove snap 68 2026-02-20T11:18:28.414 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:18:28.485 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:18:28.488 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:18:28.488 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:18:28.763 INFO:tasks.rados.rados.0.trial043.stdout:621: done (0 left) 2026-02-20T11:18:28.763 INFO:tasks.rados.rados.0.trial043.stdout:622: read oid 6 snap -1 2026-02-20T11:18:28.763 INFO:tasks.rados.rados.0.trial043.stdout:622: expect (ObjNum 244 snap 66 seq_num 244) 2026-02-20T11:18:28.763 INFO:tasks.rados.rados.0.trial043.stdout:623: read oid 35 snap -1 2026-02-20T11:18:28.763 INFO:tasks.rados.rados.0.trial043.stdout:623: expect deleted 2026-02-20T11:18:28.763 INFO:tasks.rados.rados.0.trial043.stdout:624: write oid 48 current snap is 81 2026-02-20T11:18:28.763 INFO:tasks.rados.rados.0.trial043.stdout:624: seq_num 298 ranges {608486=561969,1871634=634092,2950640=57846} 2026-02-20T11:18:28.768 INFO:tasks.rados.rados.0.trial043.stdout:624: writing trial04342147-48 from 608486 to 1170455 tid 1 2026-02-20T11:18:28.773 INFO:tasks.rados.rados.0.trial043.stdout:624: writing trial04342147-48 from 1871634 to 2505726 tid 2 2026-02-20T11:18:28.776 INFO:tasks.rados.rados.0.trial043.stdout:624: writing trial04342147-48 from 2950640 to 3008486 tid 3 2026-02-20T11:18:28.776 INFO:tasks.rados.rados.0.trial043.stdout:625: read oid 46 snap -1 2026-02-20T11:18:28.776 INFO:tasks.rados.rados.0.trial043.stdout:625: expect (ObjNum 275 snap 77 seq_num 275) 2026-02-20T11:18:28.776 INFO:tasks.rados.rados.0.trial043.stdout:626: snap_create 2026-02-20T11:18:28.776 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 4 2026-02-20T11:18:28.787 INFO:tasks.rados.rados.0.trial043.stdout:623: done (3 left) 2026-02-20T11:18:28.787 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:18:28.795 INFO:tasks.rados.rados.0.trial043.stdout:624: finishing write tid 1 to trial04342147-48 2026-02-20T11:18:28.795 INFO:tasks.rados.rados.0.trial043.stdout:622: expect (ObjNum 244 snap 66 seq_num 244) 2026-02-20T11:18:28.796 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:28.845 INFO:tasks.rados.rados.0.trial043.stdout:624: finishing write tid 2 to trial04342147-48 2026-02-20T11:18:28.846 INFO:tasks.rados.rados.0.trial043.stdout:625: expect (ObjNum 275 snap 77 seq_num 275) 2026-02-20T11:18:28.875 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:28.937 INFO:tasks.rados.rados.0.trial043.stdout:624: finishing write tid 3 to trial04342147-48 2026-02-20T11:18:28.937 INFO:tasks.rados.rados.0.trial043.stdout:624: finishing write tid 4 to trial04342147-48 2026-02-20T11:18:28.937 INFO:tasks.rados.rados.0.trial043.stdout:624: finishing write tid 5 to trial04342147-48 2026-02-20T11:18:28.937 INFO:tasks.rados.rados.0.trial043.stdout:624: oid 48 updating version 0 to 54 2026-02-20T11:18:28.937 INFO:tasks.rados.rados.0.trial043.stdout:624: oid 48 version 54 is already newer than 54 2026-02-20T11:18:28.937 INFO:tasks.rados.rados.0.trial043.stdout:624: oid 48 version 54 is already newer than 54 2026-02-20T11:18:28.937 INFO:tasks.rados.rados.0.trial043.stdout:624: oid 48 version 54 is already newer than 54 2026-02-20T11:18:28.937 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 48 v 54 (ObjNum 298 snap 81 seq_num 298) dirty exists 2026-02-20T11:18:28.937 INFO:tasks.rados.rados.0.trial043.stdout:624: left oid 48 (ObjNum 298 snap 81 seq_num 298) 2026-02-20T11:18:28.938 INFO:tasks.rados.rados.0.trial043.stdout:622: done (2 left) 2026-02-20T11:18:28.938 INFO:tasks.rados.rados.0.trial043.stdout:624: done (1 left) 2026-02-20T11:18:28.938 INFO:tasks.rados.rados.0.trial043.stdout:625: done (0 left) 2026-02-20T11:18:29.337 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:18:29.412 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:29.410+0000 7f3a80d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:29.414 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:29.414+0000 7f3a80d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:29.439 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:18:29.456 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:18:29.457 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:18:29.832 INFO:tasks.rados.rados.0.trial043.stdout:626: done (0 left) 2026-02-20T11:18:29.833 INFO:tasks.rados.rados.0.trial043.stdout:627: snap_create 2026-02-20T11:18:30.454 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~165s 2026-02-20T11:18:30.838 INFO:tasks.rados.rados.0.trial043.stdout:627: done (0 left) 2026-02-20T11:18:30.838 INFO:tasks.rados.rados.0.trial043.stdout:628: read oid 48 snap 82 2026-02-20T11:18:30.838 INFO:tasks.rados.rados.0.trial043.stdout:628: expect (ObjNum 298 snap 81 seq_num 298) 2026-02-20T11:18:30.838 INFO:tasks.rados.rados.0.trial043.stdout:629: snap_remove snap 80 2026-02-20T11:18:31.457 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:18:31.535 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:31.533+0000 7fa070ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:31.537 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:31.536+0000 7fa070ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:31.562 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:18:31.581 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:18:31.581 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:18:31.841 INFO:tasks.rados.rados.0.trial043.stdout:629: done (1 left) 2026-02-20T11:18:31.841 INFO:tasks.rados.rados.0.trial043.stdout:630: write oid 37 current snap is 83 2026-02-20T11:18:31.841 INFO:tasks.rados.rados.0.trial043.stdout:630: seq_num 299 ranges {625293=637218,1813763=494103,2978063=47230} 2026-02-20T11:18:31.845 INFO:tasks.rados.rados.0.trial043.stdout:630: writing trial04342147-37 from 625293 to 1262511 tid 1 2026-02-20T11:18:31.849 INFO:tasks.rados.rados.0.trial043.stdout:630: writing trial04342147-37 from 1813763 to 2307866 tid 2 2026-02-20T11:18:31.852 INFO:tasks.rados.rados.0.trial043.stdout:630: writing trial04342147-37 from 2978063 to 3025293 tid 3 2026-02-20T11:18:31.852 INFO:tasks.rados.rados.0.trial043.stdout:631: write (excl) oid 1 current snap is 83 2026-02-20T11:18:31.853 INFO:tasks.rados.rados.0.trial043.stdout:631: seq_num 300 ranges {642100=712467,1872245=637761} 2026-02-20T11:18:31.857 INFO:tasks.rados.rados.0.trial043.stdout:631: writing trial04342147-1 from 642100 to 1354567 tid 1 2026-02-20T11:18:31.862 INFO:tasks.rados.rados.0.trial043.stdout:631: writing trial04342147-1 from 1872245 to 2510006 tid 2 2026-02-20T11:18:31.862 INFO:tasks.rados.rados.0.trial043.stdout:632: read oid 7 snap 77 2026-02-20T11:18:31.862 INFO:tasks.rados.rados.0.trial043.stdout:632: expect (ObjNum 63 snap 3 seq_num 63) 2026-02-20T11:18:31.862 INFO:tasks.rados.rados.0.trial043.stdout:633: write (excl) oid 46 current snap is 83 2026-02-20T11:18:31.863 INFO:tasks.rados.rados.0.trial043.stdout:628: expect (ObjNum 298 snap 81 seq_num 298) 2026-02-20T11:18:31.918 INFO:tasks.rados.rados.0.trial043.stdout:630: finishing write tid 1 to trial04342147-37 2026-02-20T11:18:31.918 INFO:tasks.rados.rados.0.trial043.stdout:630: finishing write tid 2 to trial04342147-37 2026-02-20T11:18:31.919 INFO:tasks.rados.rados.0.trial043.stdout:632: expect (ObjNum 63 snap 3 seq_num 63) 2026-02-20T11:18:31.956 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:32.014 INFO:tasks.rados.rados.0.trial043.stdout:630: finishing write tid 3 to trial04342147-37 2026-02-20T11:18:32.015 INFO:tasks.rados.rados.0.trial043.stdout:630: finishing write tid 4 to trial04342147-37 2026-02-20T11:18:32.015 INFO:tasks.rados.rados.0.trial043.stdout:633: seq_num 301 ranges {658907=787716,1930727=497772,3032909=25998} 2026-02-20T11:18:32.020 INFO:tasks.rados.rados.0.trial043.stdout:633: writing trial04342147-46 from 658907 to 1446623 tid 1 2026-02-20T11:18:32.024 INFO:tasks.rados.rados.0.trial043.stdout:633: writing trial04342147-46 from 1930727 to 2428499 tid 2 2026-02-20T11:18:32.027 INFO:tasks.rados.rados.0.trial043.stdout:633: writing trial04342147-46 from 3032909 to 3058907 tid 3 2026-02-20T11:18:32.028 INFO:tasks.rados.rados.0.trial043.stdout:628: done (4 left) 2026-02-20T11:18:32.028 INFO:tasks.rados.rados.0.trial043.stdout:632: done (3 left) 2026-02-20T11:18:32.028 INFO:tasks.rados.rados.0.trial043.stdout:634: read oid 34 snap 73 2026-02-20T11:18:32.028 INFO:tasks.rados.rados.0.trial043.stdout:634: expect (ObjNum 156 snap 34 seq_num 156) 2026-02-20T11:18:32.028 INFO:tasks.rados.rados.0.trial043.stdout:635: write (excl) oid 25 current snap is 83 2026-02-20T11:18:32.028 INFO:tasks.rados.rados.0.trial043.stdout:631: finishing write tid 1 to trial04342147-1 2026-02-20T11:18:32.028 INFO:tasks.rados.rados.0.trial043.stdout:630: finishing write tid 5 to trial04342147-37 2026-02-20T11:18:32.028 INFO:tasks.rados.rados.0.trial043.stdout:630: oid 37 updating version 0 to 175 2026-02-20T11:18:32.028 INFO:tasks.rados.rados.0.trial043.stdout:630: oid 37 version 175 is already newer than 175 2026-02-20T11:18:32.028 INFO:tasks.rados.rados.0.trial043.stdout:630: oid 37 version 175 is already newer than 175 2026-02-20T11:18:32.029 INFO:tasks.rados.rados.0.trial043.stdout:630: oid 37 version 175 is already newer than 175 2026-02-20T11:18:32.029 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 37 v 175 (ObjNum 299 snap 83 seq_num 299) dirty exists 2026-02-20T11:18:32.029 INFO:tasks.rados.rados.0.trial043.stdout:630: left oid 37 (ObjNum 299 snap 83 seq_num 299) 2026-02-20T11:18:32.029 INFO:tasks.rados.rados.0.trial043.stdout:631: finishing write tid 2 to trial04342147-1 2026-02-20T11:18:32.029 INFO:tasks.rados.rados.0.trial043.stdout:631: finishing write tid 3 to trial04342147-1 2026-02-20T11:18:32.029 INFO:tasks.rados.rados.0.trial043.stdout:635: seq_num 302 ranges {675714=462965,1589209=641430,2943979=131735} 2026-02-20T11:18:32.030 INFO:tasks.rados.rados.0.trial043.stdout:635: writing trial04342147-25 from 675714 to 1138679 tid 1 2026-02-20T11:18:32.035 INFO:tasks.rados.rados.0.trial043.stdout:635: writing trial04342147-25 from 1589209 to 2230639 tid 2 2026-02-20T11:18:32.038 INFO:tasks.rados.rados.0.trial043.stdout:635: writing trial04342147-25 from 2943979 to 3075714 tid 3 2026-02-20T11:18:32.038 INFO:tasks.rados.rados.0.trial043.stdout:630: done (4 left) 2026-02-20T11:18:32.038 INFO:tasks.rados.rados.0.trial043.stdout:636: snap_create 2026-02-20T11:18:32.038 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 4 2026-02-20T11:18:32.039 INFO:tasks.rados.rados.0.trial043.stdout:631: finishing write tid 4 to trial04342147-1 2026-02-20T11:18:32.039 INFO:tasks.rados.rados.0.trial043.stdout:631: oid 1 updating version 0 to 72 2026-02-20T11:18:32.039 INFO:tasks.rados.rados.0.trial043.stdout:631: oid 1 updating version 72 to 74 2026-02-20T11:18:32.039 INFO:tasks.rados.rados.0.trial043.stdout:631: oid 1 version 74 is already newer than 74 2026-02-20T11:18:32.039 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 1 v 74 (ObjNum 300 snap 83 seq_num 300) dirty exists 2026-02-20T11:18:32.039 INFO:tasks.rados.rados.0.trial043.stdout:631: left oid 1 (ObjNum 300 snap 83 seq_num 300) 2026-02-20T11:18:32.039 INFO:tasks.rados.rados.0.trial043.stdout:631: done (3 left) 2026-02-20T11:18:32.039 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:18:32.056 INFO:tasks.rados.rados.0.trial043.stdout:633: finishing write tid 1 to trial04342147-46 2026-02-20T11:18:32.061 INFO:tasks.rados.rados.0.trial043.stdout:635: finishing write tid 1 to trial04342147-25 2026-02-20T11:18:32.065 INFO:tasks.rados.rados.0.trial043.stdout:634: expect (ObjNum 156 snap 34 seq_num 156) 2026-02-20T11:18:32.175 INFO:tasks.rados.rados.0.trial043.stdout:633: finishing write tid 2 to trial04342147-46 2026-02-20T11:18:32.175 INFO:tasks.rados.rados.0.trial043.stdout:633: finishing write tid 3 to trial04342147-46 2026-02-20T11:18:32.175 INFO:tasks.rados.rados.0.trial043.stdout:634: done (2 left) 2026-02-20T11:18:32.175 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:18:32.176 INFO:tasks.rados.rados.0.trial043.stdout:635: finishing write tid 2 to trial04342147-25 2026-02-20T11:18:32.176 INFO:tasks.rados.rados.0.trial043.stdout:633: finishing write tid 4 to trial04342147-46 2026-02-20T11:18:32.176 INFO:tasks.rados.rados.0.trial043.stdout:635: finishing write tid 3 to trial04342147-25 2026-02-20T11:18:32.176 INFO:tasks.rados.rados.0.trial043.stdout:633: finishing write tid 5 to trial04342147-46 2026-02-20T11:18:32.176 INFO:tasks.rados.rados.0.trial043.stdout:633: oid 46 updating version 0 to 19 2026-02-20T11:18:32.176 INFO:tasks.rados.rados.0.trial043.stdout:633: oid 46 updating version 19 to 22 2026-02-20T11:18:32.176 INFO:tasks.rados.rados.0.trial043.stdout:633: oid 46 version 22 is already newer than 22 2026-02-20T11:18:32.176 INFO:tasks.rados.rados.0.trial043.stdout:633: oid 46 version 22 is already newer than 22 2026-02-20T11:18:32.176 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 46 v 22 (ObjNum 301 snap 83 seq_num 301) dirty exists 2026-02-20T11:18:32.176 INFO:tasks.rados.rados.0.trial043.stdout:633: left oid 46 (ObjNum 301 snap 83 seq_num 301) 2026-02-20T11:18:32.176 INFO:tasks.rados.rados.0.trial043.stdout:635: finishing write tid 4 to trial04342147-25 2026-02-20T11:18:32.176 INFO:tasks.rados.rados.0.trial043.stdout:635: finishing write tid 5 to trial04342147-25 2026-02-20T11:18:32.176 INFO:tasks.rados.rados.0.trial043.stdout:635: oid 25 updating version 0 to 76 2026-02-20T11:18:32.176 INFO:tasks.rados.rados.0.trial043.stdout:635: oid 25 updating version 76 to 79 2026-02-20T11:18:32.176 INFO:tasks.rados.rados.0.trial043.stdout:635: oid 25 version 79 is already newer than 79 2026-02-20T11:18:32.176 INFO:tasks.rados.rados.0.trial043.stdout:635: oid 25 version 79 is already newer than 79 2026-02-20T11:18:32.176 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 25 v 79 (ObjNum 302 snap 83 seq_num 302) dirty exists 2026-02-20T11:18:32.176 INFO:tasks.rados.rados.0.trial043.stdout:635: left oid 25 (ObjNum 302 snap 83 seq_num 302) 2026-02-20T11:18:32.176 INFO:tasks.rados.rados.0.trial043.stdout:633: done (1 left) 2026-02-20T11:18:32.176 INFO:tasks.rados.rados.0.trial043.stdout:635: done (0 left) 2026-02-20T11:18:32.847 INFO:tasks.rados.rados.0.trial043.stdout:636: done (0 left) 2026-02-20T11:18:32.848 INFO:tasks.rados.rados.0.trial043.stdout:637: write oid 3 current snap is 84 2026-02-20T11:18:32.848 INFO:tasks.rados.rados.0.trial043.stdout:637: seq_num 303 ranges {692521=538214,1931338=785088} 2026-02-20T11:18:32.852 INFO:tasks.rados.rados.0.trial043.stdout:637: writing trial04342147-3 from 692521 to 1230735 tid 1 2026-02-20T11:18:32.859 INFO:tasks.rados.rados.0.trial043.stdout:637: writing trial04342147-3 from 1931338 to 2716426 tid 2 2026-02-20T11:18:32.859 INFO:tasks.rados.rados.0.trial043.stdout:638: read oid 50 snap 76 2026-02-20T11:18:32.860 INFO:tasks.rados.rados.0.trial043.stdout:638: expect (ObjNum 267 snap 75 seq_num 267) 2026-02-20T11:18:32.860 INFO:tasks.rados.rados.0.trial043.stdout:639: delete oid 17 current snap is 84 2026-02-20T11:18:32.886 INFO:tasks.rados.rados.0.trial043.stdout:637: finishing write tid 1 to trial04342147-3 2026-02-20T11:18:32.897 INFO:tasks.rados.rados.0.trial043.stdout:637: finishing write tid 2 to trial04342147-3 2026-02-20T11:18:32.907 INFO:tasks.rados.rados.0.trial043.stdout:637: finishing write tid 3 to trial04342147-3 2026-02-20T11:18:32.911 INFO:tasks.rados.rados.0.trial043.stdout:638: expect (ObjNum 267 snap 75 seq_num 267) 2026-02-20T11:18:32.948 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:33.022 INFO:tasks.rados.rados.0.trial043.stdout:637: finishing write tid 4 to trial04342147-3 2026-02-20T11:18:33.022 INFO:tasks.rados.rados.0.trial043.stdout:637: oid 3 updating version 0 to 178 2026-02-20T11:18:33.022 INFO:tasks.rados.rados.0.trial043.stdout:637: oid 3 version 178 is already newer than 178 2026-02-20T11:18:33.022 INFO:tasks.rados.rados.0.trial043.stdout:637: oid 3 version 178 is already newer than 178 2026-02-20T11:18:33.022 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 3 v 178 (ObjNum 303 snap 84 seq_num 303) dirty exists 2026-02-20T11:18:33.022 INFO:tasks.rados.rados.0.trial043.stdout:637: left oid 3 (ObjNum 303 snap 84 seq_num 303) 2026-02-20T11:18:33.022 INFO:tasks.rados.rados.0.trial043.stdout:637: done (2 left) 2026-02-20T11:18:33.023 INFO:tasks.rados.rados.0.trial043.stdout:638: done (1 left) 2026-02-20T11:18:33.023 INFO:tasks.rados.rados.0.trial043.stdout:639: done (0 left) 2026-02-20T11:18:33.023 INFO:tasks.rados.rados.0.trial043.stdout:640: read oid 11 snap -1 2026-02-20T11:18:33.023 INFO:tasks.rados.rados.0.trial043.stdout:640: expect deleted 2026-02-20T11:18:33.023 INFO:tasks.rados.rados.0.trial043.stdout:641: snap_create 2026-02-20T11:18:33.023 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:18:33.026 INFO:tasks.rados.rados.0.trial043.stdout:640: done (0 left) 2026-02-20T11:18:33.490 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:18:33.561 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:18:33.563 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:18:33.564 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:18:33.583 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:18:33.660 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:33.657+0000 7f9cf30c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:33.662 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:33.662+0000 7f9cf30c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:33.689 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:18:33.706 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:18:33.707 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:18:33.847 INFO:tasks.rados.rados.0.trial043.stdout:641: done (0 left) 2026-02-20T11:18:33.848 INFO:tasks.rados.rados.0.trial043.stdout:642: snap_remove snap 54 2026-02-20T11:18:34.854 INFO:tasks.rados.rados.0.trial043.stdout:642: done (0 left) 2026-02-20T11:18:34.854 INFO:tasks.rados.rados.0.trial043.stdout:643: read oid 17 snap -1 2026-02-20T11:18:34.855 INFO:tasks.rados.rados.0.trial043.stdout:643: expect deleted 2026-02-20T11:18:34.855 INFO:tasks.rados.rados.0.trial043.stdout:644: read oid 26 snap -1 2026-02-20T11:18:34.855 INFO:tasks.rados.rados.0.trial043.stdout:644: expect (ObjNum 271 snap 77 seq_num 271) 2026-02-20T11:18:34.855 INFO:tasks.rados.rados.0.trial043.stdout:645: snap_remove snap 83 2026-02-20T11:18:35.709 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:18:35.785 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:35.783+0000 7f2a45d5b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:35.787 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:35.787+0000 7f2a45d5b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:35.813 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:18:35.829 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:18:35.830 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:18:35.853 INFO:tasks.rados.rados.0.trial043.stdout:645: done (2 left) 2026-02-20T11:18:35.854 INFO:tasks.rados.rados.0.trial043.stdout:646: write (excl) oid 42 current snap is 85 2026-02-20T11:18:35.854 INFO:tasks.rados.rados.0.trial043.stdout:646: seq_num 305 ranges {726135=405065,1764655=788757,3026248=99887} 2026-02-20T11:18:35.858 INFO:tasks.rados.rados.0.trial043.stdout:646: writing trial04342147-42 from 726135 to 1131200 tid 1 2026-02-20T11:18:35.863 INFO:tasks.rados.rados.0.trial043.stdout:646: writing trial04342147-42 from 1764655 to 2553412 tid 2 2026-02-20T11:18:35.866 INFO:tasks.rados.rados.0.trial043.stdout:646: writing trial04342147-42 from 3026248 to 3126135 tid 3 2026-02-20T11:18:35.866 INFO:tasks.rados.rados.0.trial043.stdout:647: read oid 34 snap 82 2026-02-20T11:18:35.866 INFO:tasks.rados.rados.0.trial043.stdout:647: expect (ObjNum 156 snap 34 seq_num 156) 2026-02-20T11:18:35.866 INFO:tasks.rados.rados.0.trial043.stdout:648: write oid 29 current snap is 85 2026-02-20T11:18:35.866 INFO:tasks.rados.rados.0.trial043.stdout:648: seq_num 306 ranges {742942=480314,1823137=648768} 2026-02-20T11:18:35.870 INFO:tasks.rados.rados.0.trial043.stdout:648: writing trial04342147-29 from 742942 to 1223256 tid 1 2026-02-20T11:18:35.875 INFO:tasks.rados.rados.0.trial043.stdout:648: writing trial04342147-29 from 1823137 to 2471905 tid 2 2026-02-20T11:18:35.875 INFO:tasks.rados.rados.0.trial043.stdout:649: write oid 16 current snap is 85 2026-02-20T11:18:35.875 INFO:tasks.rados.rados.0.trial043.stdout:649: seq_num 307 ranges {759749=555563,1765266=792426,2964741=195008} 2026-02-20T11:18:35.880 INFO:tasks.rados.rados.0.trial043.stdout:649: writing trial04342147-16 from 759749 to 1315312 tid 1 2026-02-20T11:18:35.885 INFO:tasks.rados.rados.0.trial043.stdout:649: writing trial04342147-16 from 1765266 to 2557692 tid 2 2026-02-20T11:18:35.888 INFO:tasks.rados.rados.0.trial043.stdout:649: writing trial04342147-16 from 2964741 to 3159749 tid 3 2026-02-20T11:18:35.888 INFO:tasks.rados.rados.0.trial043.stdout:643: done (5 left) 2026-02-20T11:18:35.888 INFO:tasks.rados.rados.0.trial043.stdout:650: snap_remove snap 73 2026-02-20T11:18:35.958 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~171s 2026-02-20T11:18:36.859 INFO:tasks.rados.rados.0.trial043.stdout:650: done (5 left) 2026-02-20T11:18:36.859 INFO:tasks.rados.rados.0.trial043.stdout:651: snap_create 2026-02-20T11:18:36.859 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 5 2026-02-20T11:18:36.859 INFO:tasks.rados.rados.0.trial043.stdout:644: expect (ObjNum 271 snap 77 seq_num 271) 2026-02-20T11:18:36.859 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:36.893 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:36.915 INFO:tasks.rados.rados.0.trial043.stdout:644: done (4 left) 2026-02-20T11:18:36.915 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 4 2026-02-20T11:18:36.915 INFO:tasks.rados.rados.0.trial043.stdout:647: expect (ObjNum 156 snap 34 seq_num 156) 2026-02-20T11:18:36.920 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 985494 2026-02-20T11:18:36.967 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1719225 2026-02-20T11:18:36.985 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 981507 2026-02-20T11:18:37.014 INFO:tasks.rados.rados.0.trial043.stdout:646: finishing write tid 1 to trial04342147-42 2026-02-20T11:18:37.014 INFO:tasks.rados.rados.0.trial043.stdout:646: finishing write tid 2 to trial04342147-42 2026-02-20T11:18:37.014 INFO:tasks.rados.rados.0.trial043.stdout:648: finishing write tid 1 to trial04342147-29 2026-02-20T11:18:37.014 INFO:tasks.rados.rados.0.trial043.stdout:646: finishing write tid 3 to trial04342147-42 2026-02-20T11:18:37.014 INFO:tasks.rados.rados.0.trial043.stdout:647: done (3 left) 2026-02-20T11:18:37.015 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:18:37.015 INFO:tasks.rados.rados.0.trial043.stdout:648: finishing write tid 2 to trial04342147-29 2026-02-20T11:18:37.015 INFO:tasks.rados.rados.0.trial043.stdout:646: finishing write tid 4 to trial04342147-42 2026-02-20T11:18:37.015 INFO:tasks.rados.rados.0.trial043.stdout:646: finishing write tid 5 to trial04342147-42 2026-02-20T11:18:37.015 INFO:tasks.rados.rados.0.trial043.stdout:646: oid 42 updating version 0 to 93 2026-02-20T11:18:37.015 INFO:tasks.rados.rados.0.trial043.stdout:646: oid 42 version 93 is already newer than 93 2026-02-20T11:18:37.015 INFO:tasks.rados.rados.0.trial043.stdout:646: oid 42 version 93 is already newer than 90 2026-02-20T11:18:37.015 INFO:tasks.rados.rados.0.trial043.stdout:646: oid 42 version 93 is already newer than 93 2026-02-20T11:18:37.015 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 42 v 93 (ObjNum 305 snap 85 seq_num 305) dirty exists 2026-02-20T11:18:37.015 INFO:tasks.rados.rados.0.trial043.stdout:646: left oid 42 (ObjNum 305 snap 85 seq_num 305) 2026-02-20T11:18:37.015 INFO:tasks.rados.rados.0.trial043.stdout:648: finishing write tid 3 to trial04342147-29 2026-02-20T11:18:37.015 INFO:tasks.rados.rados.0.trial043.stdout:648: finishing write tid 4 to trial04342147-29 2026-02-20T11:18:37.016 INFO:tasks.rados.rados.0.trial043.stdout:648: oid 29 updating version 0 to 211 2026-02-20T11:18:37.016 INFO:tasks.rados.rados.0.trial043.stdout:648: oid 29 version 211 is already newer than 211 2026-02-20T11:18:37.016 INFO:tasks.rados.rados.0.trial043.stdout:648: oid 29 version 211 is already newer than 211 2026-02-20T11:18:37.016 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 29 v 211 (ObjNum 306 snap 85 seq_num 306) dirty exists 2026-02-20T11:18:37.016 INFO:tasks.rados.rados.0.trial043.stdout:648: left oid 29 (ObjNum 306 snap 85 seq_num 306) 2026-02-20T11:18:37.016 INFO:tasks.rados.rados.0.trial043.stdout:649: finishing write tid 1 to trial04342147-16 2026-02-20T11:18:37.016 INFO:tasks.rados.rados.0.trial043.stdout:649: finishing write tid 2 to trial04342147-16 2026-02-20T11:18:37.016 INFO:tasks.rados.rados.0.trial043.stdout:649: finishing write tid 3 to trial04342147-16 2026-02-20T11:18:37.016 INFO:tasks.rados.rados.0.trial043.stdout:649: finishing write tid 4 to trial04342147-16 2026-02-20T11:18:37.016 INFO:tasks.rados.rados.0.trial043.stdout:649: finishing write tid 5 to trial04342147-16 2026-02-20T11:18:37.016 INFO:tasks.rados.rados.0.trial043.stdout:649: oid 16 updating version 0 to 215 2026-02-20T11:18:37.016 INFO:tasks.rados.rados.0.trial043.stdout:649: oid 16 version 215 is already newer than 215 2026-02-20T11:18:37.017 INFO:tasks.rados.rados.0.trial043.stdout:649: oid 16 version 215 is already newer than 215 2026-02-20T11:18:37.017 INFO:tasks.rados.rados.0.trial043.stdout:649: oid 16 version 215 is already newer than 215 2026-02-20T11:18:37.017 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 16 v 215 (ObjNum 307 snap 85 seq_num 307) dirty exists 2026-02-20T11:18:37.017 INFO:tasks.rados.rados.0.trial043.stdout:649: left oid 16 (ObjNum 307 snap 85 seq_num 307) 2026-02-20T11:18:37.017 INFO:tasks.rados.rados.0.trial043.stdout:646: done (2 left) 2026-02-20T11:18:37.017 INFO:tasks.rados.rados.0.trial043.stdout:648: done (1 left) 2026-02-20T11:18:37.017 INFO:tasks.rados.rados.0.trial043.stdout:649: done (0 left) 2026-02-20T11:18:37.831 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:18:37.860 INFO:tasks.rados.rados.0.trial043.stdout:651: done (0 left) 2026-02-20T11:18:37.860 INFO:tasks.rados.rados.0.trial043.stdout:652: read oid 7 snap -1 2026-02-20T11:18:37.860 INFO:tasks.rados.rados.0.trial043.stdout:652: expect (ObjNum 63 snap 3 seq_num 63) 2026-02-20T11:18:37.860 INFO:tasks.rados.rados.0.trial043.stdout:653: read oid 42 snap -1 2026-02-20T11:18:37.860 INFO:tasks.rados.rados.0.trial043.stdout:653: expect (ObjNum 305 snap 85 seq_num 305) 2026-02-20T11:18:37.860 INFO:tasks.rados.rados.0.trial043.stdout:654: write oid 10 current snap is 86 2026-02-20T11:18:37.860 INFO:tasks.rados.rados.0.trial043.stdout:654: seq_num 308 ranges {776556=630812,1823748=652437,3108517=68039} 2026-02-20T11:18:37.865 INFO:tasks.rados.rados.0.trial043.stdout:654: writing trial04342147-10 from 776556 to 1407368 tid 1 2026-02-20T11:18:37.869 INFO:tasks.rados.rados.0.trial043.stdout:654: writing trial04342147-10 from 1823748 to 2476185 tid 2 2026-02-20T11:18:37.872 INFO:tasks.rados.rados.0.trial043.stdout:654: writing trial04342147-10 from 3108517 to 3176556 tid 3 2026-02-20T11:18:37.872 INFO:tasks.rados.rados.0.trial043.stdout:655: read oid 29 snap -1 2026-02-20T11:18:37.872 INFO:tasks.rados.rados.0.trial043.stdout:655: expect (ObjNum 306 snap 85 seq_num 306) 2026-02-20T11:18:37.872 INFO:tasks.rados.rados.0.trial043.stdout:656: snap_remove snap 76 2026-02-20T11:18:37.910 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:37.908+0000 7f5d3541a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:37.912 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:37.912+0000 7f5d3541a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:37.940 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:18:37.957 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:18:37.957 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:18:38.565 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:18:38.635 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:18:38.638 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:18:38.638 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:18:38.765 INFO:tasks.rados.rados.0.trial043.stdout:656: done (4 left) 2026-02-20T11:18:38.765 INFO:tasks.rados.rados.0.trial043.stdout:657: write oid 27 current snap is 86 2026-02-20T11:18:38.765 INFO:tasks.rados.rados.0.trial043.stdout:657: seq_num 309 ranges {793363=706061,2282230=796095} 2026-02-20T11:18:38.770 INFO:tasks.rados.rados.0.trial043.stdout:657: writing trial04342147-27 from 793363 to 1499424 tid 1 2026-02-20T11:18:38.777 INFO:tasks.rados.rados.0.trial043.stdout:657: writing trial04342147-27 from 2282230 to 3078325 tid 2 2026-02-20T11:18:38.777 INFO:tasks.rados.rados.0.trial043.stdout:658: snap_create 2026-02-20T11:18:38.777 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 5 2026-02-20T11:18:38.777 INFO:tasks.rados.rados.0.trial043.stdout:653: expect (ObjNum 305 snap 85 seq_num 305) 2026-02-20T11:18:38.777 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:38.813 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:38.854 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 967705 2026-02-20T11:18:38.879 INFO:tasks.rados.rados.0.trial043.stdout:655: expect (ObjNum 306 snap 85 seq_num 306) 2026-02-20T11:18:38.970 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:39.018 INFO:tasks.rados.rados.0.trial043.stdout:652: expect (ObjNum 63 snap 3 seq_num 63) 2026-02-20T11:18:39.073 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:39.108 INFO:tasks.rados.rados.0.trial043.stdout:654: finishing write tid 1 to trial04342147-10 2026-02-20T11:18:39.108 INFO:tasks.rados.rados.0.trial043.stdout:654: finishing write tid 2 to trial04342147-10 2026-02-20T11:18:39.108 INFO:tasks.rados.rados.0.trial043.stdout:654: finishing write tid 3 to trial04342147-10 2026-02-20T11:18:39.108 INFO:tasks.rados.rados.0.trial043.stdout:652: done (4 left) 2026-02-20T11:18:39.108 INFO:tasks.rados.rados.0.trial043.stdout:653: done (3 left) 2026-02-20T11:18:39.108 INFO:tasks.rados.rados.0.trial043.stdout:655: done (2 left) 2026-02-20T11:18:39.108 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:18:39.109 INFO:tasks.rados.rados.0.trial043.stdout:654: finishing write tid 4 to trial04342147-10 2026-02-20T11:18:39.109 INFO:tasks.rados.rados.0.trial043.stdout:654: finishing write tid 5 to trial04342147-10 2026-02-20T11:18:39.109 INFO:tasks.rados.rados.0.trial043.stdout:654: oid 10 updating version 0 to 73 2026-02-20T11:18:39.109 INFO:tasks.rados.rados.0.trial043.stdout:654: oid 10 version 73 is already newer than 73 2026-02-20T11:18:39.109 INFO:tasks.rados.rados.0.trial043.stdout:654: oid 10 version 73 is already newer than 73 2026-02-20T11:18:39.109 INFO:tasks.rados.rados.0.trial043.stdout:654: oid 10 version 73 is already newer than 73 2026-02-20T11:18:39.109 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 10 v 73 (ObjNum 308 snap 86 seq_num 308) dirty exists 2026-02-20T11:18:39.109 INFO:tasks.rados.rados.0.trial043.stdout:654: left oid 10 (ObjNum 308 snap 86 seq_num 308) 2026-02-20T11:18:39.109 INFO:tasks.rados.rados.0.trial043.stdout:654: done (1 left) 2026-02-20T11:18:39.109 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:18:39.109 INFO:tasks.rados.rados.0.trial043.stdout:657: finishing write tid 1 to trial04342147-27 2026-02-20T11:18:39.109 INFO:tasks.rados.rados.0.trial043.stdout:657: finishing write tid 2 to trial04342147-27 2026-02-20T11:18:39.109 INFO:tasks.rados.rados.0.trial043.stdout:657: finishing write tid 3 to trial04342147-27 2026-02-20T11:18:39.109 INFO:tasks.rados.rados.0.trial043.stdout:657: finishing write tid 4 to trial04342147-27 2026-02-20T11:18:39.109 INFO:tasks.rados.rados.0.trial043.stdout:657: oid 27 updating version 0 to 198 2026-02-20T11:18:39.109 INFO:tasks.rados.rados.0.trial043.stdout:657: oid 27 version 198 is already newer than 198 2026-02-20T11:18:39.109 INFO:tasks.rados.rados.0.trial043.stdout:657: oid 27 version 198 is already newer than 198 2026-02-20T11:18:39.110 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 27 v 198 (ObjNum 309 snap 86 seq_num 309) dirty exists 2026-02-20T11:18:39.110 INFO:tasks.rados.rados.0.trial043.stdout:657: left oid 27 (ObjNum 309 snap 86 seq_num 309) 2026-02-20T11:18:39.110 INFO:tasks.rados.rados.0.trial043.stdout:657: done (0 left) 2026-02-20T11:18:39.867 INFO:tasks.rados.rados.0.trial043.stdout:658: done (0 left) 2026-02-20T11:18:39.867 INFO:tasks.rados.rados.0.trial043.stdout:659: read oid 11 snap -1 2026-02-20T11:18:39.868 INFO:tasks.rados.rados.0.trial043.stdout:659: expect deleted 2026-02-20T11:18:39.868 INFO:tasks.rados.rados.0.trial043.stdout:660: write (excl) oid 4 current snap is 87 2026-02-20T11:18:39.868 INFO:tasks.rados.rados.0.trial043.stdout:660: seq_num 310 ranges {410170=781310,1940712=656106,3163363=46807} 2026-02-20T11:18:39.871 INFO:tasks.rados.rados.0.trial043.stdout:660: writing trial04342147-4 from 410170 to 1191480 tid 1 2026-02-20T11:18:39.876 INFO:tasks.rados.rados.0.trial043.stdout:660: writing trial04342147-4 from 1940712 to 2596818 tid 2 2026-02-20T11:18:39.879 INFO:tasks.rados.rados.0.trial043.stdout:660: writing trial04342147-4 from 3163363 to 3210170 tid 3 2026-02-20T11:18:39.879 INFO:tasks.rados.rados.0.trial043.stdout:661: delete oid 48 current snap is 87 2026-02-20T11:18:39.904 INFO:tasks.rados.rados.0.trial043.stdout:659: done (2 left) 2026-02-20T11:18:39.904 INFO:tasks.rados.rados.0.trial043.stdout:661: done (1 left) 2026-02-20T11:18:39.904 INFO:tasks.rados.rados.0.trial043.stdout:662: snap_remove snap 71 2026-02-20T11:18:39.958 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:18:40.033 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:40.031+0000 7f284990b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:40.035 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:40.034+0000 7f284990b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:40.061 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:18:40.079 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:18:40.079 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:18:40.869 INFO:tasks.rados.rados.0.trial043.stdout:662: done (1 left) 2026-02-20T11:18:40.869 INFO:tasks.rados.rados.0.trial043.stdout:663: write (excl) oid 43 current snap is 87 2026-02-20T11:18:40.869 INFO:tasks.rados.rados.0.trial043.stdout:660: finishing write tid 1 to trial04342147-4 2026-02-20T11:18:40.869 INFO:tasks.rados.rados.0.trial043.stdout:663: seq_num 312 ranges {443784=648161,1657676=659775,2818209=425575} 2026-02-20T11:18:40.873 INFO:tasks.rados.rados.0.trial043.stdout:663: writing trial04342147-43 from 443784 to 1091945 tid 1 2026-02-20T11:18:40.878 INFO:tasks.rados.rados.0.trial043.stdout:663: writing trial04342147-43 from 1657676 to 2317451 tid 2 2026-02-20T11:18:40.882 INFO:tasks.rados.rados.0.trial043.stdout:663: writing trial04342147-43 from 2818209 to 3243784 tid 3 2026-02-20T11:18:40.882 INFO:tasks.rados.rados.0.trial043.stdout:664: snap_remove snap 86 2026-02-20T11:18:41.463 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~176s 2026-02-20T11:18:41.870 INFO:tasks.rados.rados.0.trial043.stdout:664: done (2 left) 2026-02-20T11:18:41.870 INFO:tasks.rados.rados.0.trial043.stdout:665: snap_remove snap 77 2026-02-20T11:18:42.081 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:18:42.159 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:42.157+0000 7fdd6f4cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:42.161 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:42.160+0000 7fdd6f4cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:42.189 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:18:42.204 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:18:42.205 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:18:42.877 INFO:tasks.rados.rados.0.trial043.stdout:665: done (2 left) 2026-02-20T11:18:42.877 INFO:tasks.rados.rados.0.trial043.stdout:666: read oid 42 snap 79 2026-02-20T11:18:42.877 INFO:tasks.rados.rados.0.trial043.stdout:666: expect (ObjNum 274 snap 77 seq_num 274) 2026-02-20T11:18:42.877 INFO:tasks.rados.rados.0.trial043.stdout:667: delete oid 25 current snap is 87 2026-02-20T11:18:42.877 INFO:tasks.rados.rados.0.trial043.stdout:660: finishing write tid 2 to trial04342147-4 2026-02-20T11:18:42.878 INFO:tasks.rados.rados.0.trial043.stdout:660: finishing write tid 3 to trial04342147-4 2026-02-20T11:18:42.878 INFO:tasks.rados.rados.0.trial043.stdout:660: finishing write tid 4 to trial04342147-4 2026-02-20T11:18:42.878 INFO:tasks.rados.rados.0.trial043.stdout:660: finishing write tid 5 to trial04342147-4 2026-02-20T11:18:42.878 INFO:tasks.rados.rados.0.trial043.stdout:660: oid 4 updating version 0 to 81 2026-02-20T11:18:42.878 INFO:tasks.rados.rados.0.trial043.stdout:660: oid 4 updating version 81 to 84 2026-02-20T11:18:42.878 INFO:tasks.rados.rados.0.trial043.stdout:660: oid 4 version 84 is already newer than 84 2026-02-20T11:18:42.878 INFO:tasks.rados.rados.0.trial043.stdout:660: oid 4 version 84 is already newer than 84 2026-02-20T11:18:42.878 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 4 v 84 (ObjNum 310 snap 87 seq_num 310) dirty exists 2026-02-20T11:18:42.878 INFO:tasks.rados.rados.0.trial043.stdout:660: left oid 4 (ObjNum 310 snap 87 seq_num 310) 2026-02-20T11:18:42.878 INFO:tasks.rados.rados.0.trial043.stdout:663: finishing write tid 1 to trial04342147-43 2026-02-20T11:18:42.878 INFO:tasks.rados.rados.0.trial043.stdout:663: finishing write tid 2 to trial04342147-43 2026-02-20T11:18:42.878 INFO:tasks.rados.rados.0.trial043.stdout:663: finishing write tid 3 to trial04342147-43 2026-02-20T11:18:42.878 INFO:tasks.rados.rados.0.trial043.stdout:663: finishing write tid 4 to trial04342147-43 2026-02-20T11:18:42.879 INFO:tasks.rados.rados.0.trial043.stdout:663: finishing write tid 5 to trial04342147-43 2026-02-20T11:18:42.879 INFO:tasks.rados.rados.0.trial043.stdout:663: oid 43 updating version 0 to 230 2026-02-20T11:18:42.879 INFO:tasks.rados.rados.0.trial043.stdout:663: oid 43 version 230 is already newer than 230 2026-02-20T11:18:42.879 INFO:tasks.rados.rados.0.trial043.stdout:663: oid 43 version 230 is already newer than 230 2026-02-20T11:18:42.879 INFO:tasks.rados.rados.0.trial043.stdout:663: oid 43 version 230 is already newer than 227 2026-02-20T11:18:42.879 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 43 v 230 (ObjNum 312 snap 87 seq_num 312) dirty exists 2026-02-20T11:18:42.879 INFO:tasks.rados.rados.0.trial043.stdout:663: left oid 43 (ObjNum 312 snap 87 seq_num 312) 2026-02-20T11:18:42.903 INFO:tasks.rados.rados.0.trial043.stdout:666: expect (ObjNum 274 snap 77 seq_num 274) 2026-02-20T11:18:42.952 INFO:tasks.rados.rados.0.trial043.stdout:660: done (3 left) 2026-02-20T11:18:42.952 INFO:tasks.rados.rados.0.trial043.stdout:663: done (2 left) 2026-02-20T11:18:42.952 INFO:tasks.rados.rados.0.trial043.stdout:666: done (1 left) 2026-02-20T11:18:42.952 INFO:tasks.rados.rados.0.trial043.stdout:667: done (0 left) 2026-02-20T11:18:42.952 INFO:tasks.rados.rados.0.trial043.stdout:668: read oid 19 snap -1 2026-02-20T11:18:42.952 INFO:tasks.rados.rados.0.trial043.stdout:668: expect (ObjNum 261 snap 74 seq_num 261) 2026-02-20T11:18:42.952 INFO:tasks.rados.rados.0.trial043.stdout:669: snap_remove snap 82 2026-02-20T11:18:43.639 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:18:43.710 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:18:43.713 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:18:43.713 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:18:43.878 INFO:tasks.rados.rados.0.trial043.stdout:669: done (1 left) 2026-02-20T11:18:43.878 INFO:tasks.rados.rados.0.trial043.stdout:670: write oid 36 current snap is 87 2026-02-20T11:18:43.878 INFO:tasks.rados.rados.0.trial043.stdout:670: seq_num 314 ranges {477398=798659,1774640=663444,2873055=404343} 2026-02-20T11:18:43.883 INFO:tasks.rados.rados.0.trial043.stdout:670: writing trial04342147-36 from 477398 to 1276057 tid 1 2026-02-20T11:18:43.887 INFO:tasks.rados.rados.0.trial043.stdout:670: writing trial04342147-36 from 1774640 to 2438084 tid 2 2026-02-20T11:18:43.892 INFO:tasks.rados.rados.0.trial043.stdout:670: writing trial04342147-36 from 2873055 to 3277398 tid 3 2026-02-20T11:18:43.892 INFO:tasks.rados.rados.0.trial043.stdout:671: read oid 49 snap -1 2026-02-20T11:18:43.892 INFO:tasks.rados.rados.0.trial043.stdout:671: expect (ObjNum 245 snap 66 seq_num 245) 2026-02-20T11:18:43.892 INFO:tasks.rados.rados.0.trial043.stdout:672: write oid 10 current snap is 87 2026-02-20T11:18:43.892 INFO:tasks.rados.rados.0.trial043.stdout:672: seq_num 315 ranges {494205=473908,1433122=407102,2384125=462684} 2026-02-20T11:18:43.895 INFO:tasks.rados.rados.0.trial043.stdout:672: writing trial04342147-10 from 494205 to 968113 tid 1 2026-02-20T11:18:43.898 INFO:tasks.rados.rados.0.trial043.stdout:672: writing trial04342147-10 from 1433122 to 1840224 tid 2 2026-02-20T11:18:43.902 INFO:tasks.rados.rados.0.trial043.stdout:672: writing trial04342147-10 from 2384125 to 2846809 tid 3 2026-02-20T11:18:43.902 INFO:tasks.rados.rados.0.trial043.stdout:673: write (excl) oid 41 current snap is 87 2026-02-20T11:18:43.902 INFO:tasks.rados.rados.0.trial043.stdout:668: expect (ObjNum 261 snap 74 seq_num 261) 2026-02-20T11:18:43.915 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1571406 2026-02-20T11:18:44.037 INFO:tasks.rados.rados.0.trial043.stdout:671: expect (ObjNum 245 snap 66 seq_num 245) 2026-02-20T11:18:44.037 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:44.143 INFO:tasks.rados.rados.0.trial043.stdout:672: finishing write tid 1 to trial04342147-10 2026-02-20T11:18:44.143 INFO:tasks.rados.rados.0.trial043.stdout:673: seq_num 316 ranges {511012=549157,1775251=550760,3095195=215817} 2026-02-20T11:18:44.147 INFO:tasks.rados.rados.0.trial043.stdout:673: writing trial04342147-41 from 511012 to 1060169 tid 1 2026-02-20T11:18:44.152 INFO:tasks.rados.rados.0.trial043.stdout:673: writing trial04342147-41 from 1775251 to 2326011 tid 2 2026-02-20T11:18:44.156 INFO:tasks.rados.rados.0.trial043.stdout:673: writing trial04342147-41 from 3095195 to 3311012 tid 3 2026-02-20T11:18:44.157 INFO:tasks.rados.rados.0.trial043.stdout:668: done (4 left) 2026-02-20T11:18:44.157 INFO:tasks.rados.rados.0.trial043.stdout:671: done (3 left) 2026-02-20T11:18:44.157 INFO:tasks.rados.rados.0.trial043.stdout:674: delete oid 25 current snap is 87 2026-02-20T11:18:44.157 INFO:tasks.rados.rados.0.trial043.stdout:670: finishing write tid 1 to trial04342147-36 2026-02-20T11:18:44.157 INFO:tasks.rados.rados.0.trial043.stdout:672: finishing write tid 2 to trial04342147-10 2026-02-20T11:18:44.157 INFO:tasks.rados.rados.0.trial043.stdout:670: finishing write tid 2 to trial04342147-36 2026-02-20T11:18:44.157 INFO:tasks.rados.rados.0.trial043.stdout:672: finishing write tid 3 to trial04342147-10 2026-02-20T11:18:44.157 INFO:tasks.rados.rados.0.trial043.stdout:670: finishing write tid 3 to trial04342147-36 2026-02-20T11:18:44.157 INFO:tasks.rados.rados.0.trial043.stdout:670: finishing write tid 4 to trial04342147-36 2026-02-20T11:18:44.157 INFO:tasks.rados.rados.0.trial043.stdout:672: finishing write tid 4 to trial04342147-10 2026-02-20T11:18:44.157 INFO:tasks.rados.rados.0.trial043.stdout:670: finishing write tid 5 to trial04342147-36 2026-02-20T11:18:44.157 INFO:tasks.rados.rados.0.trial043.stdout:670: oid 36 updating version 0 to 36 2026-02-20T11:18:44.158 INFO:tasks.rados.rados.0.trial043.stdout:670: oid 36 version 36 is already newer than 36 2026-02-20T11:18:44.158 INFO:tasks.rados.rados.0.trial043.stdout:670: oid 36 version 36 is already newer than 36 2026-02-20T11:18:44.158 INFO:tasks.rados.rados.0.trial043.stdout:670: oid 36 version 36 is already newer than 36 2026-02-20T11:18:44.158 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 36 v 36 (ObjNum 314 snap 87 seq_num 314) dirty exists 2026-02-20T11:18:44.158 INFO:tasks.rados.rados.0.trial043.stdout:670: left oid 36 (ObjNum 314 snap 87 seq_num 314) 2026-02-20T11:18:44.158 INFO:tasks.rados.rados.0.trial043.stdout:672: finishing write tid 5 to trial04342147-10 2026-02-20T11:18:44.158 INFO:tasks.rados.rados.0.trial043.stdout:672: oid 10 updating version 0 to 77 2026-02-20T11:18:44.158 INFO:tasks.rados.rados.0.trial043.stdout:672: oid 10 version 77 is already newer than 77 2026-02-20T11:18:44.158 INFO:tasks.rados.rados.0.trial043.stdout:672: oid 10 version 77 is already newer than 77 2026-02-20T11:18:44.158 INFO:tasks.rados.rados.0.trial043.stdout:672: oid 10 version 77 is already newer than 77 2026-02-20T11:18:44.158 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 10 v 77 (ObjNum 315 snap 87 seq_num 315) dirty exists 2026-02-20T11:18:44.158 INFO:tasks.rados.rados.0.trial043.stdout:672: left oid 10 (ObjNum 315 snap 87 seq_num 315) 2026-02-20T11:18:44.160 INFO:tasks.rados.rados.0.trial043.stdout:670: done (3 left) 2026-02-20T11:18:44.160 INFO:tasks.rados.rados.0.trial043.stdout:672: done (2 left) 2026-02-20T11:18:44.160 INFO:tasks.rados.rados.0.trial043.stdout:674: done (1 left) 2026-02-20T11:18:44.161 INFO:tasks.rados.rados.0.trial043.stdout:675: read oid 8 snap -1 2026-02-20T11:18:44.161 INFO:tasks.rados.rados.0.trial043.stdout:675: expect (ObjNum 293 snap 81 seq_num 293) 2026-02-20T11:18:44.161 INFO:tasks.rados.rados.0.trial043.stdout:676: read oid 43 snap -1 2026-02-20T11:18:44.161 INFO:tasks.rados.rados.0.trial043.stdout:676: expect (ObjNum 312 snap 87 seq_num 312) 2026-02-20T11:18:44.161 INFO:tasks.rados.rados.0.trial043.stdout:677: read oid 1 snap 75 2026-02-20T11:18:44.161 INFO:tasks.rados.rados.0.trial043.stdout:677: expect (ObjNum 207 snap 58 seq_num 207) 2026-02-20T11:18:44.161 INFO:tasks.rados.rados.0.trial043.stdout:678: snap_create 2026-02-20T11:18:44.161 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 4 2026-02-20T11:18:44.187 INFO:tasks.rados.rados.0.trial043.stdout:677: expect (ObjNum 207 snap 58 seq_num 207) 2026-02-20T11:18:44.187 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:44.205 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:18:44.281 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:44.279+0000 7f7543635640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:44.284 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:44.283+0000 7f7543635640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:44.310 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:18:44.327 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:18:44.327 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:18:44.366 INFO:tasks.rados.rados.0.trial043.stdout:677: done (3 left) 2026-02-20T11:18:44.366 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:18:44.366 INFO:tasks.rados.rados.0.trial043.stdout:673: finishing write tid 1 to trial04342147-41 2026-02-20T11:18:44.367 INFO:tasks.rados.rados.0.trial043.stdout:676: expect (ObjNum 312 snap 87 seq_num 312) 2026-02-20T11:18:44.413 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:44.466 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 370884 2026-02-20T11:18:44.518 INFO:tasks.rados.rados.0.trial043.stdout:675: expect (ObjNum 293 snap 81 seq_num 293) 2026-02-20T11:18:44.636 INFO:tasks.rados.rados.0.trial043.stdout:673: finishing write tid 2 to trial04342147-41 2026-02-20T11:18:44.636 INFO:tasks.rados.rados.0.trial043.stdout:673: finishing write tid 3 to trial04342147-41 2026-02-20T11:18:44.636 INFO:tasks.rados.rados.0.trial043.stdout:673: finishing write tid 4 to trial04342147-41 2026-02-20T11:18:44.637 INFO:tasks.rados.rados.0.trial043.stdout:673: finishing write tid 5 to trial04342147-41 2026-02-20T11:18:44.637 INFO:tasks.rados.rados.0.trial043.stdout:673: oid 41 updating version 0 to 176 2026-02-20T11:18:44.637 INFO:tasks.rados.rados.0.trial043.stdout:673: oid 41 version 176 is already newer than 173 2026-02-20T11:18:44.637 INFO:tasks.rados.rados.0.trial043.stdout:673: oid 41 version 176 is already newer than 176 2026-02-20T11:18:44.637 INFO:tasks.rados.rados.0.trial043.stdout:673: oid 41 version 176 is already newer than 176 2026-02-20T11:18:44.638 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 41 v 176 (ObjNum 316 snap 87 seq_num 316) dirty exists 2026-02-20T11:18:44.638 INFO:tasks.rados.rados.0.trial043.stdout:673: left oid 41 (ObjNum 316 snap 87 seq_num 316) 2026-02-20T11:18:44.638 INFO:tasks.rados.rados.0.trial043.stdout:673: done (2 left) 2026-02-20T11:18:44.638 INFO:tasks.rados.rados.0.trial043.stdout:675: done (1 left) 2026-02-20T11:18:44.638 INFO:tasks.rados.rados.0.trial043.stdout:676: done (0 left) 2026-02-20T11:18:44.884 INFO:tasks.rados.rados.0.trial043.stdout:678: done (0 left) 2026-02-20T11:18:44.884 INFO:tasks.rados.rados.0.trial043.stdout:679: read oid 15 snap -1 2026-02-20T11:18:44.884 INFO:tasks.rados.rados.0.trial043.stdout:679: expect deleted 2026-02-20T11:18:44.884 INFO:tasks.rados.rados.0.trial043.stdout:680: write (excl) oid 31 current snap is 88 2026-02-20T11:18:44.884 INFO:tasks.rados.rados.0.trial043.stdout:680: seq_num 318 ranges {544626=699655,1892215=554429,3150041=194585} 2026-02-20T11:18:44.889 INFO:tasks.rados.rados.0.trial043.stdout:680: writing trial04342147-31 from 544626 to 1244281 tid 1 2026-02-20T11:18:44.894 INFO:tasks.rados.rados.0.trial043.stdout:680: writing trial04342147-31 from 1892215 to 2446644 tid 2 2026-02-20T11:18:44.897 INFO:tasks.rados.rados.0.trial043.stdout:680: writing trial04342147-31 from 3150041 to 3344626 tid 3 2026-02-20T11:18:44.898 INFO:tasks.rados.rados.0.trial043.stdout:681: snap_remove snap 85 2026-02-20T11:18:45.887 INFO:tasks.rados.rados.0.trial043.stdout:681: done (2 left) 2026-02-20T11:18:45.887 INFO:tasks.rados.rados.0.trial043.stdout:682: write (excl) oid 7 current snap is 88 2026-02-20T11:18:45.887 INFO:tasks.rados.rados.0.trial043.stdout:682: seq_num 319 ranges {561433=774904,1950697=414440,2777464=583969} 2026-02-20T11:18:45.893 INFO:tasks.rados.rados.0.trial043.stdout:682: writing trial04342147-7 from 561433 to 1336337 tid 1 2026-02-20T11:18:45.897 INFO:tasks.rados.rados.0.trial043.stdout:682: writing trial04342147-7 from 1950697 to 2365137 tid 2 2026-02-20T11:18:45.901 INFO:tasks.rados.rados.0.trial043.stdout:682: writing trial04342147-7 from 2777464 to 3361433 tid 3 2026-02-20T11:18:45.901 INFO:tasks.rados.rados.0.trial043.stdout:683: write (excl) oid 28 current snap is 88 2026-02-20T11:18:45.901 INFO:tasks.rados.rados.0.trial043.stdout:683: seq_num 320 ranges {578240=566506,1609179=558098,2804887=551750} 2026-02-20T11:18:45.905 INFO:tasks.rados.rados.0.trial043.stdout:683: writing trial04342147-28 from 578240 to 1144746 tid 1 2026-02-20T11:18:45.909 INFO:tasks.rados.rados.0.trial043.stdout:683: writing trial04342147-28 from 1609179 to 2167277 tid 2 2026-02-20T11:18:45.914 INFO:tasks.rados.rados.0.trial043.stdout:683: writing trial04342147-28 from 2804887 to 3356637 tid 3 2026-02-20T11:18:45.914 INFO:tasks.rados.rados.0.trial043.stdout:684: read oid 32 snap -1 2026-02-20T11:18:45.914 INFO:tasks.rados.rados.0.trial043.stdout:684: expect deleted 2026-02-20T11:18:45.914 INFO:tasks.rados.rados.0.trial043.stdout:685: read oid 42 snap -1 2026-02-20T11:18:45.914 INFO:tasks.rados.rados.0.trial043.stdout:685: expect (ObjNum 305 snap 85 seq_num 305) 2026-02-20T11:18:45.914 INFO:tasks.rados.rados.0.trial043.stdout:686: write (excl) oid 10 current snap is 88 2026-02-20T11:18:45.914 INFO:tasks.rados.rados.0.trial043.stdout:686: seq_num 321 ranges {595047=641755,1667661=418109,2548663=763022} 2026-02-20T11:18:45.919 INFO:tasks.rados.rados.0.trial043.stdout:686: writing trial04342147-10 from 595047 to 1236802 tid 1 2026-02-20T11:18:45.922 INFO:tasks.rados.rados.0.trial043.stdout:686: writing trial04342147-10 from 1667661 to 2085770 tid 2 2026-02-20T11:18:45.927 INFO:tasks.rados.rados.0.trial043.stdout:686: writing trial04342147-10 from 2548663 to 3311685 tid 3 2026-02-20T11:18:45.927 INFO:tasks.rados.rados.0.trial043.stdout:679: done (6 left) 2026-02-20T11:18:45.927 INFO:tasks.rados.rados.0.trial043.stdout:687: snap_remove snap 75 2026-02-20T11:18:46.328 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:18:46.406 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:46.404+0000 7fa4bc811640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:46.408 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:46.407+0000 7fa4bc811640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:46.434 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:18:46.451 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:18:46.452 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:18:46.889 INFO:tasks.rados.rados.0.trial043.stdout:687: done (6 left) 2026-02-20T11:18:46.890 INFO:tasks.rados.rados.0.trial043.stdout:688: delete oid 35 current snap is 88 2026-02-20T11:18:46.890 INFO:tasks.rados.rados.0.trial043.stdout:680: finishing write tid 1 to trial04342147-31 2026-02-20T11:18:46.890 INFO:tasks.rados.rados.0.trial043.stdout:680: finishing write tid 2 to trial04342147-31 2026-02-20T11:18:46.890 INFO:tasks.rados.rados.0.trial043.stdout:680: finishing write tid 3 to trial04342147-31 2026-02-20T11:18:46.890 INFO:tasks.rados.rados.0.trial043.stdout:680: finishing write tid 4 to trial04342147-31 2026-02-20T11:18:46.890 INFO:tasks.rados.rados.0.trial043.stdout:680: finishing write tid 5 to trial04342147-31 2026-02-20T11:18:46.890 INFO:tasks.rados.rados.0.trial043.stdout:680: oid 31 updating version 0 to 61 2026-02-20T11:18:46.890 INFO:tasks.rados.rados.0.trial043.stdout:680: oid 31 version 61 is already newer than 61 2026-02-20T11:18:46.890 INFO:tasks.rados.rados.0.trial043.stdout:680: oid 31 version 61 is already newer than 58 2026-02-20T11:18:46.890 INFO:tasks.rados.rados.0.trial043.stdout:680: oid 31 version 61 is already newer than 61 2026-02-20T11:18:46.890 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 31 v 61 (ObjNum 318 snap 88 seq_num 318) dirty exists 2026-02-20T11:18:46.890 INFO:tasks.rados.rados.0.trial043.stdout:680: left oid 31 (ObjNum 318 snap 88 seq_num 318) 2026-02-20T11:18:46.890 INFO:tasks.rados.rados.0.trial043.stdout:685: expect (ObjNum 305 snap 85 seq_num 305) 2026-02-20T11:18:46.938 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2995909 2026-02-20T11:18:46.968 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~182s 2026-02-20T11:18:46.983 INFO:tasks.rados.rados.0.trial043.stdout:682: finishing write tid 1 to trial04342147-7 2026-02-20T11:18:46.984 INFO:tasks.rados.rados.0.trial043.stdout:682: finishing write tid 2 to trial04342147-7 2026-02-20T11:18:46.984 INFO:tasks.rados.rados.0.trial043.stdout:680: done (6 left) 2026-02-20T11:18:46.984 INFO:tasks.rados.rados.0.trial043.stdout:684: done (5 left) 2026-02-20T11:18:46.984 INFO:tasks.rados.rados.0.trial043.stdout:685: done (4 left) 2026-02-20T11:18:46.984 INFO:tasks.rados.rados.0.trial043.stdout:688: done (3 left) 2026-02-20T11:18:46.984 INFO:tasks.rados.rados.0.trial043.stdout:689: write (excl) oid 26 current snap is 88 2026-02-20T11:18:46.984 INFO:tasks.rados.rados.0.trial043.stdout:682: finishing write tid 3 to trial04342147-7 2026-02-20T11:18:46.984 INFO:tasks.rados.rados.0.trial043.stdout:682: finishing write tid 4 to trial04342147-7 2026-02-20T11:18:46.984 INFO:tasks.rados.rados.0.trial043.stdout:682: finishing write tid 5 to trial04342147-7 2026-02-20T11:18:46.985 INFO:tasks.rados.rados.0.trial043.stdout:682: oid 7 updating version 0 to 79 2026-02-20T11:18:46.985 INFO:tasks.rados.rados.0.trial043.stdout:682: oid 7 updating version 79 to 82 2026-02-20T11:18:46.985 INFO:tasks.rados.rados.0.trial043.stdout:682: oid 7 version 82 is already newer than 82 2026-02-20T11:18:46.985 INFO:tasks.rados.rados.0.trial043.stdout:682: oid 7 version 82 is already newer than 82 2026-02-20T11:18:46.985 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 7 v 82 (ObjNum 319 snap 88 seq_num 319) dirty exists 2026-02-20T11:18:46.985 INFO:tasks.rados.rados.0.trial043.stdout:682: left oid 7 (ObjNum 319 snap 88 seq_num 319) 2026-02-20T11:18:46.985 INFO:tasks.rados.rados.0.trial043.stdout:683: finishing write tid 1 to trial04342147-28 2026-02-20T11:18:46.985 INFO:tasks.rados.rados.0.trial043.stdout:683: finishing write tid 2 to trial04342147-28 2026-02-20T11:18:46.985 INFO:tasks.rados.rados.0.trial043.stdout:689: seq_num 323 ranges {628661=792253,2184625=421778,3403509=25152} 2026-02-20T11:18:46.989 INFO:tasks.rados.rados.0.trial043.stdout:689: writing trial04342147-26 from 628661 to 1420914 tid 1 2026-02-20T11:18:46.993 INFO:tasks.rados.rados.0.trial043.stdout:689: writing trial04342147-26 from 2184625 to 2606403 tid 2 2026-02-20T11:18:46.996 INFO:tasks.rados.rados.0.trial043.stdout:689: writing trial04342147-26 from 3403509 to 3428661 tid 3 2026-02-20T11:18:46.997 INFO:tasks.rados.rados.0.trial043.stdout:682: done (3 left) 2026-02-20T11:18:46.997 INFO:tasks.rados.rados.0.trial043.stdout:690: write (excl) oid 40 current snap is 88 2026-02-20T11:18:46.997 INFO:tasks.rados.rados.0.trial043.stdout:690: seq_num 324 ranges {645468=467502,1726754=565436,2798226=596838} 2026-02-20T11:18:47.000 INFO:tasks.rados.rados.0.trial043.stdout:690: writing trial04342147-40 from 645468 to 1112970 tid 1 2026-02-20T11:18:47.005 INFO:tasks.rados.rados.0.trial043.stdout:690: writing trial04342147-40 from 1726754 to 2292190 tid 2 2026-02-20T11:18:47.010 INFO:tasks.rados.rados.0.trial043.stdout:690: writing trial04342147-40 from 2798226 to 3395064 tid 3 2026-02-20T11:18:47.011 INFO:tasks.rados.rados.0.trial043.stdout:691: snap_create 2026-02-20T11:18:47.011 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 4 2026-02-20T11:18:47.011 INFO:tasks.rados.rados.0.trial043.stdout:683: finishing write tid 3 to trial04342147-28 2026-02-20T11:18:47.011 INFO:tasks.rados.rados.0.trial043.stdout:683: finishing write tid 4 to trial04342147-28 2026-02-20T11:18:47.011 INFO:tasks.rados.rados.0.trial043.stdout:683: finishing write tid 5 to trial04342147-28 2026-02-20T11:18:47.011 INFO:tasks.rados.rados.0.trial043.stdout:683: oid 28 updating version 0 to 86 2026-02-20T11:18:47.011 INFO:tasks.rados.rados.0.trial043.stdout:683: oid 28 version 86 is already newer than 86 2026-02-20T11:18:47.011 INFO:tasks.rados.rados.0.trial043.stdout:683: oid 28 version 86 is already newer than 83 2026-02-20T11:18:47.011 INFO:tasks.rados.rados.0.trial043.stdout:683: oid 28 version 86 is already newer than 86 2026-02-20T11:18:47.011 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 28 v 86 (ObjNum 320 snap 88 seq_num 320) dirty exists 2026-02-20T11:18:47.011 INFO:tasks.rados.rados.0.trial043.stdout:683: left oid 28 (ObjNum 320 snap 88 seq_num 320) 2026-02-20T11:18:47.012 INFO:tasks.rados.rados.0.trial043.stdout:686: finishing write tid 1 to trial04342147-10 2026-02-20T11:18:47.012 INFO:tasks.rados.rados.0.trial043.stdout:686: finishing write tid 2 to trial04342147-10 2026-02-20T11:18:47.012 INFO:tasks.rados.rados.0.trial043.stdout:686: finishing write tid 3 to trial04342147-10 2026-02-20T11:18:47.012 INFO:tasks.rados.rados.0.trial043.stdout:683: done (3 left) 2026-02-20T11:18:47.012 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:18:47.012 INFO:tasks.rados.rados.0.trial043.stdout:686: finishing write tid 4 to trial04342147-10 2026-02-20T11:18:47.012 INFO:tasks.rados.rados.0.trial043.stdout:686: finishing write tid 5 to trial04342147-10 2026-02-20T11:18:47.012 INFO:tasks.rados.rados.0.trial043.stdout:686: oid 10 updating version 0 to 84 2026-02-20T11:18:47.012 INFO:tasks.rados.rados.0.trial043.stdout:686: oid 10 updating version 84 to 87 2026-02-20T11:18:47.012 INFO:tasks.rados.rados.0.trial043.stdout:686: oid 10 version 87 is already newer than 87 2026-02-20T11:18:47.012 INFO:tasks.rados.rados.0.trial043.stdout:686: oid 10 version 87 is already newer than 87 2026-02-20T11:18:47.012 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 10 v 87 (ObjNum 321 snap 88 seq_num 321) dirty exists 2026-02-20T11:18:47.012 INFO:tasks.rados.rados.0.trial043.stdout:686: left oid 10 (ObjNum 321 snap 88 seq_num 321) 2026-02-20T11:18:47.013 INFO:tasks.rados.rados.0.trial043.stdout:686: done (2 left) 2026-02-20T11:18:47.013 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:18:47.032 INFO:tasks.rados.rados.0.trial043.stdout:690: finishing write tid 1 to trial04342147-40 2026-02-20T11:18:47.033 INFO:tasks.rados.rados.0.trial043.stdout:689: finishing write tid 1 to trial04342147-26 2026-02-20T11:18:47.050 INFO:tasks.rados.rados.0.trial043.stdout:689: finishing write tid 2 to trial04342147-26 2026-02-20T11:18:47.050 INFO:tasks.rados.rados.0.trial043.stdout:690: finishing write tid 2 to trial04342147-40 2026-02-20T11:18:47.057 INFO:tasks.rados.rados.0.trial043.stdout:689: finishing write tid 3 to trial04342147-26 2026-02-20T11:18:47.057 INFO:tasks.rados.rados.0.trial043.stdout:690: finishing write tid 3 to trial04342147-40 2026-02-20T11:18:47.061 INFO:tasks.rados.rados.0.trial043.stdout:689: finishing write tid 4 to trial04342147-26 2026-02-20T11:18:47.063 INFO:tasks.rados.rados.0.trial043.stdout:689: finishing write tid 5 to trial04342147-26 2026-02-20T11:18:47.064 INFO:tasks.rados.rados.0.trial043.stdout:689: oid 26 updating version 0 to 212 2026-02-20T11:18:47.064 INFO:tasks.rados.rados.0.trial043.stdout:689: oid 26 updating version 212 to 215 2026-02-20T11:18:47.064 INFO:tasks.rados.rados.0.trial043.stdout:689: oid 26 version 215 is already newer than 215 2026-02-20T11:18:47.064 INFO:tasks.rados.rados.0.trial043.stdout:689: oid 26 version 215 is already newer than 215 2026-02-20T11:18:47.064 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 26 v 215 (ObjNum 323 snap 88 seq_num 323) dirty exists 2026-02-20T11:18:47.064 INFO:tasks.rados.rados.0.trial043.stdout:689: left oid 26 (ObjNum 323 snap 88 seq_num 323) 2026-02-20T11:18:47.064 INFO:tasks.rados.rados.0.trial043.stdout:689: done (1 left) 2026-02-20T11:18:47.064 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:18:47.065 INFO:tasks.rados.rados.0.trial043.stdout:690: finishing write tid 4 to trial04342147-40 2026-02-20T11:18:47.066 INFO:tasks.rados.rados.0.trial043.stdout:690: finishing write tid 5 to trial04342147-40 2026-02-20T11:18:47.066 INFO:tasks.rados.rados.0.trial043.stdout:690: oid 40 updating version 0 to 183 2026-02-20T11:18:47.066 INFO:tasks.rados.rados.0.trial043.stdout:690: oid 40 version 183 is already newer than 183 2026-02-20T11:18:47.066 INFO:tasks.rados.rados.0.trial043.stdout:690: oid 40 version 183 is already newer than 180 2026-02-20T11:18:47.066 INFO:tasks.rados.rados.0.trial043.stdout:690: oid 40 version 183 is already newer than 183 2026-02-20T11:18:47.066 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 40 v 183 (ObjNum 324 snap 88 seq_num 324) dirty exists 2026-02-20T11:18:47.066 INFO:tasks.rados.rados.0.trial043.stdout:690: left oid 40 (ObjNum 324 snap 88 seq_num 324) 2026-02-20T11:18:47.066 INFO:tasks.rados.rados.0.trial043.stdout:690: done (0 left) 2026-02-20T11:18:47.893 INFO:tasks.rados.rados.0.trial043.stdout:691: done (0 left) 2026-02-20T11:18:47.893 INFO:tasks.rados.rados.0.trial043.stdout:692: read oid 48 snap -1 2026-02-20T11:18:47.894 INFO:tasks.rados.rados.0.trial043.stdout:692: expect deleted 2026-02-20T11:18:47.894 INFO:tasks.rados.rados.0.trial043.stdout:693: write oid 30 current snap is 89 2026-02-20T11:18:47.894 INFO:tasks.rados.rados.0.trial043.stdout:693: seq_num 325 ranges {662275=542751,1785236=425447,2942002=520273} 2026-02-20T11:18:47.898 INFO:tasks.rados.rados.0.trial043.stdout:693: writing trial04342147-30 from 662275 to 1205026 tid 1 2026-02-20T11:18:47.901 INFO:tasks.rados.rados.0.trial043.stdout:693: writing trial04342147-30 from 1785236 to 2210683 tid 2 2026-02-20T11:18:47.907 INFO:tasks.rados.rados.0.trial043.stdout:693: writing trial04342147-30 from 2942002 to 3462275 tid 3 2026-02-20T11:18:47.907 INFO:tasks.rados.rados.0.trial043.stdout:694: write oid 46 current snap is 89 2026-02-20T11:18:47.907 INFO:tasks.rados.rados.0.trial043.stdout:694: seq_num 326 ranges {679082=618000,1843718=569105,2853072=626010} 2026-02-20T11:18:47.912 INFO:tasks.rados.rados.0.trial043.stdout:694: writing trial04342147-46 from 679082 to 1297082 tid 1 2026-02-20T11:18:47.916 INFO:tasks.rados.rados.0.trial043.stdout:694: writing trial04342147-46 from 1843718 to 2412823 tid 2 2026-02-20T11:18:47.921 INFO:tasks.rados.rados.0.trial043.stdout:694: writing trial04342147-46 from 2853072 to 3479082 tid 3 2026-02-20T11:18:47.921 INFO:tasks.rados.rados.0.trial043.stdout:695: write (excl) oid 37 current snap is 89 2026-02-20T11:18:47.921 INFO:tasks.rados.rados.0.trial043.stdout:695: seq_num 327 ranges {695889=409602,1618553=712763,2996848=499041} 2026-02-20T11:18:47.925 INFO:tasks.rados.rados.0.trial043.stdout:695: writing trial04342147-37 from 695889 to 1105491 tid 1 2026-02-20T11:18:47.929 INFO:tasks.rados.rados.0.trial043.stdout:695: writing trial04342147-37 from 1618553 to 2331316 tid 2 2026-02-20T11:18:47.934 INFO:tasks.rados.rados.0.trial043.stdout:695: writing trial04342147-37 from 2996848 to 3495889 tid 3 2026-02-20T11:18:47.934 INFO:tasks.rados.rados.0.trial043.stdout:696: write (excl) oid 25 current snap is 89 2026-02-20T11:18:47.934 INFO:tasks.rados.rados.0.trial043.stdout:696: seq_num 328 ranges {712696=484851,1960682=572774,3307918=204778} 2026-02-20T11:18:47.938 INFO:tasks.rados.rados.0.trial043.stdout:696: writing trial04342147-25 from 712696 to 1197547 tid 1 2026-02-20T11:18:47.945 INFO:tasks.rados.rados.0.trial043.stdout:696: writing trial04342147-25 from 1960682 to 2533456 tid 2 2026-02-20T11:18:47.949 INFO:tasks.rados.rados.0.trial043.stdout:696: writing trial04342147-25 from 3307918 to 3512696 tid 3 2026-02-20T11:18:47.950 INFO:tasks.rados.rados.0.trial043.stdout:697: delete oid 29 current snap is 89 2026-02-20T11:18:47.950 INFO:tasks.rados.rados.0.trial043.stdout:693: finishing write tid 1 to trial04342147-30 2026-02-20T11:18:47.953 INFO:tasks.rados.rados.0.trial043.stdout:694: finishing write tid 1 to trial04342147-46 2026-02-20T11:18:47.953 INFO:tasks.rados.rados.0.trial043.stdout:693: finishing write tid 2 to trial04342147-30 2026-02-20T11:18:47.963 INFO:tasks.rados.rados.0.trial043.stdout:694: finishing write tid 2 to trial04342147-46 2026-02-20T11:18:47.973 INFO:tasks.rados.rados.0.trial043.stdout:696: finishing write tid 1 to trial04342147-25 2026-02-20T11:18:47.979 INFO:tasks.rados.rados.0.trial043.stdout:694: finishing write tid 3 to trial04342147-46 2026-02-20T11:18:47.980 INFO:tasks.rados.rados.0.trial043.stdout:693: finishing write tid 3 to trial04342147-30 2026-02-20T11:18:47.985 INFO:tasks.rados.rados.0.trial043.stdout:694: finishing write tid 4 to trial04342147-46 2026-02-20T11:18:47.988 INFO:tasks.rados.rados.0.trial043.stdout:694: finishing write tid 5 to trial04342147-46 2026-02-20T11:18:47.988 INFO:tasks.rados.rados.0.trial043.stdout:694: oid 46 updating version 0 to 27 2026-02-20T11:18:47.988 INFO:tasks.rados.rados.0.trial043.stdout:694: oid 46 version 27 is already newer than 27 2026-02-20T11:18:47.988 INFO:tasks.rados.rados.0.trial043.stdout:694: oid 46 version 27 is already newer than 27 2026-02-20T11:18:47.988 INFO:tasks.rados.rados.0.trial043.stdout:694: oid 46 version 27 is already newer than 27 2026-02-20T11:18:47.988 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 46 v 27 (ObjNum 326 snap 89 seq_num 326) dirty exists 2026-02-20T11:18:47.988 INFO:tasks.rados.rados.0.trial043.stdout:694: left oid 46 (ObjNum 326 snap 89 seq_num 326) 2026-02-20T11:18:47.989 INFO:tasks.rados.rados.0.trial043.stdout:693: finishing write tid 4 to trial04342147-30 2026-02-20T11:18:47.991 INFO:tasks.rados.rados.0.trial043.stdout:693: finishing write tid 5 to trial04342147-30 2026-02-20T11:18:47.991 INFO:tasks.rados.rados.0.trial043.stdout:693: oid 30 updating version 0 to 41 2026-02-20T11:18:47.991 INFO:tasks.rados.rados.0.trial043.stdout:693: oid 30 version 41 is already newer than 41 2026-02-20T11:18:47.991 INFO:tasks.rados.rados.0.trial043.stdout:693: oid 30 version 41 is already newer than 41 2026-02-20T11:18:47.991 INFO:tasks.rados.rados.0.trial043.stdout:693: oid 30 version 41 is already newer than 41 2026-02-20T11:18:47.991 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 30 v 41 (ObjNum 325 snap 89 seq_num 325) dirty exists 2026-02-20T11:18:47.991 INFO:tasks.rados.rados.0.trial043.stdout:693: left oid 30 (ObjNum 325 snap 89 seq_num 325) 2026-02-20T11:18:47.993 INFO:tasks.rados.rados.0.trial043.stdout:696: finishing write tid 2 to trial04342147-25 2026-02-20T11:18:48.005 INFO:tasks.rados.rados.0.trial043.stdout:696: finishing write tid 3 to trial04342147-25 2026-02-20T11:18:48.006 INFO:tasks.rados.rados.0.trial043.stdout:692: done (5 left) 2026-02-20T11:18:48.006 INFO:tasks.rados.rados.0.trial043.stdout:693: done (4 left) 2026-02-20T11:18:48.006 INFO:tasks.rados.rados.0.trial043.stdout:694: done (3 left) 2026-02-20T11:18:48.006 INFO:tasks.rados.rados.0.trial043.stdout:697: done (2 left) 2026-02-20T11:18:48.006 INFO:tasks.rados.rados.0.trial043.stdout:698: delete oid 16 current snap is 89 2026-02-20T11:18:48.014 INFO:tasks.rados.rados.0.trial043.stdout:696: finishing write tid 4 to trial04342147-25 2026-02-20T11:18:48.017 INFO:tasks.rados.rados.0.trial043.stdout:695: finishing write tid 1 to trial04342147-37 2026-02-20T11:18:48.017 INFO:tasks.rados.rados.0.trial043.stdout:696: finishing write tid 5 to trial04342147-25 2026-02-20T11:18:48.017 INFO:tasks.rados.rados.0.trial043.stdout:696: oid 25 updating version 0 to 90 2026-02-20T11:18:48.017 INFO:tasks.rados.rados.0.trial043.stdout:696: oid 25 version 90 is already newer than 90 2026-02-20T11:18:48.017 INFO:tasks.rados.rados.0.trial043.stdout:696: oid 25 version 90 is already newer than 90 2026-02-20T11:18:48.017 INFO:tasks.rados.rados.0.trial043.stdout:696: oid 25 version 90 is already newer than 87 2026-02-20T11:18:48.017 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 25 v 90 (ObjNum 328 snap 89 seq_num 328) dirty exists 2026-02-20T11:18:48.017 INFO:tasks.rados.rados.0.trial043.stdout:696: left oid 25 (ObjNum 328 snap 89 seq_num 328) 2026-02-20T11:18:48.031 INFO:tasks.rados.rados.0.trial043.stdout:696: done (2 left) 2026-02-20T11:18:48.031 INFO:tasks.rados.rados.0.trial043.stdout:698: done (1 left) 2026-02-20T11:18:48.032 INFO:tasks.rados.rados.0.trial043.stdout:699: snap_create 2026-02-20T11:18:48.032 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:18:48.033 INFO:tasks.rados.rados.0.trial043.stdout:695: finishing write tid 2 to trial04342147-37 2026-02-20T11:18:48.041 INFO:tasks.rados.rados.0.trial043.stdout:695: finishing write tid 3 to trial04342147-37 2026-02-20T11:18:48.045 INFO:tasks.rados.rados.0.trial043.stdout:695: finishing write tid 4 to trial04342147-37 2026-02-20T11:18:48.047 INFO:tasks.rados.rados.0.trial043.stdout:695: finishing write tid 5 to trial04342147-37 2026-02-20T11:18:48.047 INFO:tasks.rados.rados.0.trial043.stdout:695: oid 37 updating version 0 to 183 2026-02-20T11:18:48.048 INFO:tasks.rados.rados.0.trial043.stdout:695: oid 37 version 183 is already newer than 183 2026-02-20T11:18:48.048 INFO:tasks.rados.rados.0.trial043.stdout:695: oid 37 version 183 is already newer than 180 2026-02-20T11:18:48.048 INFO:tasks.rados.rados.0.trial043.stdout:695: oid 37 version 183 is already newer than 183 2026-02-20T11:18:48.048 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 37 v 183 (ObjNum 327 snap 89 seq_num 327) dirty exists 2026-02-20T11:18:48.048 INFO:tasks.rados.rados.0.trial043.stdout:695: left oid 37 (ObjNum 327 snap 89 seq_num 327) 2026-02-20T11:18:48.048 INFO:tasks.rados.rados.0.trial043.stdout:695: done (0 left) 2026-02-20T11:18:48.453 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:18:48.531 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:48.530+0000 7f7765fe4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:48.533 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:48.534+0000 7f7765fe4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:48.559 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:18:48.575 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:18:48.575 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:18:48.714 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:18:48.765 INFO:tasks.rados.rados.0.trial043.stdout:699: done (0 left) 2026-02-20T11:18:48.765 INFO:tasks.rados.rados.0.trial043.stdout:700: read oid 42 snap -1 2026-02-20T11:18:48.766 INFO:tasks.rados.rados.0.trial043.stdout:700: expect (ObjNum 305 snap 85 seq_num 305) 2026-02-20T11:18:48.766 INFO:tasks.rados.rados.0.trial043.stdout:701: delete oid 31 current snap is 90 2026-02-20T11:18:48.787 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:18:48.789 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:18:48.790 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:18:48.795 INFO:tasks.rados.rados.0.trial043.stdout:700: expect (ObjNum 305 snap 85 seq_num 305) 2026-02-20T11:18:48.887 INFO:tasks.rados.rados.0.trial043.stdout:700: done (1 left) 2026-02-20T11:18:48.887 INFO:tasks.rados.rados.0.trial043.stdout:701: done (0 left) 2026-02-20T11:18:48.887 INFO:tasks.rados.rados.0.trial043.stdout:702: snap_remove snap 84 2026-02-20T11:18:49.767 INFO:tasks.rados.rados.0.trial043.stdout:702: done (0 left) 2026-02-20T11:18:49.768 INFO:tasks.rados.rados.0.trial043.stdout:703: write (excl) oid 40 current snap is 90 2026-02-20T11:18:49.768 INFO:tasks.rados.rados.0.trial043.stdout:703: seq_num 332 ranges {779924=785847,2078257=580112,3301257=278667} 2026-02-20T11:18:49.773 INFO:tasks.rados.rados.0.trial043.stdout:703: writing trial04342147-40 from 779924 to 1565771 tid 1 2026-02-20T11:18:49.778 INFO:tasks.rados.rados.0.trial043.stdout:703: writing trial04342147-40 from 2078257 to 2658369 tid 2 2026-02-20T11:18:49.781 INFO:tasks.rados.rados.0.trial043.stdout:703: writing trial04342147-40 from 3301257 to 3579924 tid 3 2026-02-20T11:18:49.782 INFO:tasks.rados.rados.0.trial043.stdout:704: snap_create 2026-02-20T11:18:49.782 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:18:49.816 INFO:tasks.rados.rados.0.trial043.stdout:703: finishing write tid 1 to trial04342147-40 2026-02-20T11:18:49.833 INFO:tasks.rados.rados.0.trial043.stdout:703: finishing write tid 2 to trial04342147-40 2026-02-20T11:18:49.841 INFO:tasks.rados.rados.0.trial043.stdout:703: finishing write tid 3 to trial04342147-40 2026-02-20T11:18:49.847 INFO:tasks.rados.rados.0.trial043.stdout:703: finishing write tid 4 to trial04342147-40 2026-02-20T11:18:49.849 INFO:tasks.rados.rados.0.trial043.stdout:703: finishing write tid 5 to trial04342147-40 2026-02-20T11:18:49.849 INFO:tasks.rados.rados.0.trial043.stdout:703: oid 40 updating version 0 to 192 2026-02-20T11:18:49.849 INFO:tasks.rados.rados.0.trial043.stdout:703: oid 40 version 192 is already newer than 192 2026-02-20T11:18:49.850 INFO:tasks.rados.rados.0.trial043.stdout:703: oid 40 version 192 is already newer than 192 2026-02-20T11:18:49.850 INFO:tasks.rados.rados.0.trial043.stdout:703: oid 40 version 192 is already newer than 189 2026-02-20T11:18:49.850 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 40 v 192 (ObjNum 332 snap 90 seq_num 332) dirty exists 2026-02-20T11:18:49.850 INFO:tasks.rados.rados.0.trial043.stdout:703: left oid 40 (ObjNum 332 snap 90 seq_num 332) 2026-02-20T11:18:49.850 INFO:tasks.rados.rados.0.trial043.stdout:703: done (0 left) 2026-02-20T11:18:50.576 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:18:50.655 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:50.654+0000 7f1d8445a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:50.658 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:50.658+0000 7f1d8445a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:50.683 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:18:50.699 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:18:50.700 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:18:50.770 INFO:tasks.rados.rados.0.trial043.stdout:704: done (0 left) 2026-02-20T11:18:50.770 INFO:tasks.rados.rados.0.trial043.stdout:705: snap_remove snap 88 2026-02-20T11:18:51.773 INFO:tasks.rados.rados.0.trial043.stdout:705: done (0 left) 2026-02-20T11:18:51.773 INFO:tasks.rados.rados.0.trial043.stdout:706: read oid 12 snap -1 2026-02-20T11:18:51.774 INFO:tasks.rados.rados.0.trial043.stdout:706: expect (ObjNum 225 snap 63 seq_num 225) 2026-02-20T11:18:51.774 INFO:tasks.rados.rados.0.trial043.stdout:707: write (excl) oid 27 current snap is 91 2026-02-20T11:18:51.774 INFO:tasks.rados.rados.0.trial043.stdout:707: seq_num 333 ranges {796731=461096,1736739=723770,2928680=668051} 2026-02-20T11:18:51.778 INFO:tasks.rados.rados.0.trial043.stdout:707: writing trial04342147-27 from 796731 to 1257827 tid 1 2026-02-20T11:18:51.783 INFO:tasks.rados.rados.0.trial043.stdout:707: writing trial04342147-27 from 1736739 to 2460509 tid 2 2026-02-20T11:18:51.788 INFO:tasks.rados.rados.0.trial043.stdout:707: writing trial04342147-27 from 2928680 to 3596731 tid 3 2026-02-20T11:18:51.788 INFO:tasks.rados.rados.0.trial043.stdout:708: read oid 20 snap -1 2026-02-20T11:18:51.788 INFO:tasks.rados.rados.0.trial043.stdout:708: expect (ObjNum 128 snap 17 seq_num 128) 2026-02-20T11:18:51.788 INFO:tasks.rados.rados.0.trial043.stdout:709: write (excl) oid 18 current snap is 91 2026-02-20T11:18:51.788 INFO:tasks.rados.rados.0.trial043.stdout:709: seq_num 334 ranges {413538=536345,1395221=583781,2556103=658617} 2026-02-20T11:18:51.791 INFO:tasks.rados.rados.0.trial043.stdout:709: writing trial04342147-18 from 413538 to 949883 tid 1 2026-02-20T11:18:51.795 INFO:tasks.rados.rados.0.trial043.stdout:709: writing trial04342147-18 from 1395221 to 1979002 tid 2 2026-02-20T11:18:51.800 INFO:tasks.rados.rados.0.trial043.stdout:709: writing trial04342147-18 from 2556103 to 3214720 tid 3 2026-02-20T11:18:51.800 INFO:tasks.rados.rados.0.trial043.stdout:710: write oid 33 current snap is 91 2026-02-20T11:18:51.800 INFO:tasks.rados.rados.0.trial043.stdout:710: seq_num 335 ranges {430345=727947,1570056=727439,2699879=469889} 2026-02-20T11:18:51.805 INFO:tasks.rados.rados.0.trial043.stdout:710: writing trial04342147-33 from 430345 to 1158292 tid 1 2026-02-20T11:18:51.810 INFO:tasks.rados.rados.0.trial043.stdout:710: writing trial04342147-33 from 1570056 to 2297495 tid 2 2026-02-20T11:18:51.813 INFO:tasks.rados.rados.0.trial043.stdout:710: writing trial04342147-33 from 2699879 to 3169768 tid 3 2026-02-20T11:18:51.813 INFO:tasks.rados.rados.0.trial043.stdout:711: snap_remove snap 64 2026-02-20T11:18:52.473 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~187s 2026-02-20T11:18:52.701 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:18:52.777 INFO:tasks.rados.rados.0.trial043.stdout:711: done (5 left) 2026-02-20T11:18:52.777 INFO:tasks.rados.rados.0.trial043.stdout:712: snap_create 2026-02-20T11:18:52.777 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 5 2026-02-20T11:18:52.778 INFO:tasks.rados.rados.0.trial043.stdout:709: finishing write tid 1 to trial04342147-18 2026-02-20T11:18:52.778 INFO:tasks.rados.rados.0.trial043.stdout:708: expect (ObjNum 128 snap 17 seq_num 128) 2026-02-20T11:18:52.778 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:52.777+0000 7f08aadb8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:52.781 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:52.781+0000 7f08aadb8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:52.786 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1805983 2026-02-20T11:18:52.812 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:18:52.820 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:52.830 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:18:52.830 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:18:52.845 INFO:tasks.rados.rados.0.trial043.stdout:706: expect (ObjNum 225 snap 63 seq_num 225) 2026-02-20T11:18:52.867 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:18:52.931 INFO:tasks.rados.rados.0.trial043.stdout:709: finishing write tid 2 to trial04342147-18 2026-02-20T11:18:52.932 INFO:tasks.rados.rados.0.trial043.stdout:706: done (4 left) 2026-02-20T11:18:52.932 INFO:tasks.rados.rados.0.trial043.stdout:708: done (3 left) 2026-02-20T11:18:52.932 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:18:52.932 INFO:tasks.rados.rados.0.trial043.stdout:707: finishing write tid 1 to trial04342147-27 2026-02-20T11:18:52.932 INFO:tasks.rados.rados.0.trial043.stdout:709: finishing write tid 3 to trial04342147-18 2026-02-20T11:18:52.932 INFO:tasks.rados.rados.0.trial043.stdout:709: finishing write tid 4 to trial04342147-18 2026-02-20T11:18:52.932 INFO:tasks.rados.rados.0.trial043.stdout:709: finishing write tid 5 to trial04342147-18 2026-02-20T11:18:52.932 INFO:tasks.rados.rados.0.trial043.stdout:709: oid 18 updating version 0 to 45 2026-02-20T11:18:52.932 INFO:tasks.rados.rados.0.trial043.stdout:709: oid 18 version 45 is already newer than 45 2026-02-20T11:18:52.932 INFO:tasks.rados.rados.0.trial043.stdout:709: oid 18 version 45 is already newer than 45 2026-02-20T11:18:52.933 INFO:tasks.rados.rados.0.trial043.stdout:709: oid 18 version 45 is already newer than 42 2026-02-20T11:18:52.933 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 18 v 45 (ObjNum 334 snap 91 seq_num 334) dirty exists 2026-02-20T11:18:52.933 INFO:tasks.rados.rados.0.trial043.stdout:709: left oid 18 (ObjNum 334 snap 91 seq_num 334) 2026-02-20T11:18:52.933 INFO:tasks.rados.rados.0.trial043.stdout:707: finishing write tid 2 to trial04342147-27 2026-02-20T11:18:52.933 INFO:tasks.rados.rados.0.trial043.stdout:707: finishing write tid 3 to trial04342147-27 2026-02-20T11:18:52.933 INFO:tasks.rados.rados.0.trial043.stdout:709: done (2 left) 2026-02-20T11:18:52.933 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:18:52.933 INFO:tasks.rados.rados.0.trial043.stdout:707: finishing write tid 4 to trial04342147-27 2026-02-20T11:18:52.933 INFO:tasks.rados.rados.0.trial043.stdout:707: finishing write tid 5 to trial04342147-27 2026-02-20T11:18:52.933 INFO:tasks.rados.rados.0.trial043.stdout:707: oid 27 updating version 0 to 221 2026-02-20T11:18:52.933 INFO:tasks.rados.rados.0.trial043.stdout:707: oid 27 updating version 221 to 224 2026-02-20T11:18:52.933 INFO:tasks.rados.rados.0.trial043.stdout:707: oid 27 version 224 is already newer than 224 2026-02-20T11:18:52.934 INFO:tasks.rados.rados.0.trial043.stdout:707: oid 27 version 224 is already newer than 224 2026-02-20T11:18:52.934 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 27 v 224 (ObjNum 333 snap 91 seq_num 333) dirty exists 2026-02-20T11:18:52.934 INFO:tasks.rados.rados.0.trial043.stdout:707: left oid 27 (ObjNum 333 snap 91 seq_num 333) 2026-02-20T11:18:52.934 INFO:tasks.rados.rados.0.trial043.stdout:710: finishing write tid 1 to trial04342147-33 2026-02-20T11:18:52.934 INFO:tasks.rados.rados.0.trial043.stdout:710: finishing write tid 2 to trial04342147-33 2026-02-20T11:18:52.934 INFO:tasks.rados.rados.0.trial043.stdout:710: finishing write tid 3 to trial04342147-33 2026-02-20T11:18:52.934 INFO:tasks.rados.rados.0.trial043.stdout:707: done (1 left) 2026-02-20T11:18:52.934 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:18:52.934 INFO:tasks.rados.rados.0.trial043.stdout:710: finishing write tid 4 to trial04342147-33 2026-02-20T11:18:52.934 INFO:tasks.rados.rados.0.trial043.stdout:710: finishing write tid 5 to trial04342147-33 2026-02-20T11:18:52.934 INFO:tasks.rados.rados.0.trial043.stdout:710: oid 33 updating version 0 to 255 2026-02-20T11:18:52.934 INFO:tasks.rados.rados.0.trial043.stdout:710: oid 33 version 255 is already newer than 255 2026-02-20T11:18:52.935 INFO:tasks.rados.rados.0.trial043.stdout:710: oid 33 version 255 is already newer than 255 2026-02-20T11:18:52.935 INFO:tasks.rados.rados.0.trial043.stdout:710: oid 33 version 255 is already newer than 255 2026-02-20T11:18:52.935 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 33 v 255 (ObjNum 335 snap 91 seq_num 335) dirty exists 2026-02-20T11:18:52.935 INFO:tasks.rados.rados.0.trial043.stdout:710: left oid 33 (ObjNum 335 snap 91 seq_num 335) 2026-02-20T11:18:52.935 INFO:tasks.rados.rados.0.trial043.stdout:710: done (0 left) 2026-02-20T11:18:53.778 INFO:tasks.rados.rados.0.trial043.stdout:712: done (0 left) 2026-02-20T11:18:53.778 INFO:tasks.rados.rados.0.trial043.stdout:713: snap_remove snap 87 2026-02-20T11:18:53.791 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:18:53.860 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:18:53.862 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:18:53.863 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:18:54.781 INFO:tasks.rados.rados.0.trial043.stdout:713: done (0 left) 2026-02-20T11:18:54.781 INFO:tasks.rados.rados.0.trial043.stdout:714: delete oid 19 current snap is 92 2026-02-20T11:18:54.826 INFO:tasks.rados.rados.0.trial043.stdout:714: done (0 left) 2026-02-20T11:18:54.826 INFO:tasks.rados.rados.0.trial043.stdout:715: write oid 48 current snap is 92 2026-02-20T11:18:54.826 INFO:tasks.rados.rados.0.trial043.stdout:715: seq_num 337 ranges {463959=478445,1570667=731108,3038372=492433} 2026-02-20T11:18:54.829 INFO:tasks.rados.rados.0.trial043.stdout:715: writing trial04342147-48 from 463959 to 942404 tid 1 2026-02-20T11:18:54.831 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:18:54.835 INFO:tasks.rados.rados.0.trial043.stdout:715: writing trial04342147-48 from 1570667 to 2301775 tid 2 2026-02-20T11:18:54.839 INFO:tasks.rados.rados.0.trial043.stdout:715: writing trial04342147-48 from 3038372 to 3530805 tid 3 2026-02-20T11:18:54.839 INFO:tasks.rados.rados.0.trial043.stdout:716: delete oid 50 current snap is 92 2026-02-20T11:18:54.849 INFO:tasks.rados.rados.0.trial043.stdout:715: finishing write tid 1 to trial04342147-48 2026-02-20T11:18:54.856 INFO:tasks.rados.rados.0.trial043.stdout:715: finishing write tid 2 to trial04342147-48 2026-02-20T11:18:54.866 INFO:tasks.rados.rados.0.trial043.stdout:715: finishing write tid 3 to trial04342147-48 2026-02-20T11:18:54.871 INFO:tasks.rados.rados.0.trial043.stdout:715: finishing write tid 4 to trial04342147-48 2026-02-20T11:18:54.872 INFO:tasks.rados.rados.0.trial043.stdout:715: finishing write tid 5 to trial04342147-48 2026-02-20T11:18:54.872 INFO:tasks.rados.rados.0.trial043.stdout:715: oid 48 updating version 0 to 67 2026-02-20T11:18:54.872 INFO:tasks.rados.rados.0.trial043.stdout:715: oid 48 version 67 is already newer than 67 2026-02-20T11:18:54.872 INFO:tasks.rados.rados.0.trial043.stdout:715: oid 48 version 67 is already newer than 67 2026-02-20T11:18:54.872 INFO:tasks.rados.rados.0.trial043.stdout:715: oid 48 version 67 is already newer than 67 2026-02-20T11:18:54.872 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 48 v 67 (ObjNum 337 snap 92 seq_num 337) dirty exists 2026-02-20T11:18:54.872 INFO:tasks.rados.rados.0.trial043.stdout:715: left oid 48 (ObjNum 337 snap 92 seq_num 337) 2026-02-20T11:18:54.909 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:54.908+0000 7f1edc993640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:54.912 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:54.912+0000 7f1edc993640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:54.928 INFO:tasks.rados.rados.0.trial043.stdout:715: done (1 left) 2026-02-20T11:18:54.928 INFO:tasks.rados.rados.0.trial043.stdout:716: done (0 left) 2026-02-20T11:18:54.928 INFO:tasks.rados.rados.0.trial043.stdout:717: snap_remove snap 89 2026-02-20T11:18:54.938 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:18:54.954 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:18:54.954 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:18:55.787 INFO:tasks.rados.rados.0.trial043.stdout:717: done (0 left) 2026-02-20T11:18:55.787 INFO:tasks.rados.rados.0.trial043.stdout:718: write oid 13 current snap is 92 2026-02-20T11:18:55.787 INFO:tasks.rados.rados.0.trial043.stdout:718: seq_num 339 ranges {497573=628943,1687631=734777,3093218=604355} 2026-02-20T11:18:55.791 INFO:tasks.rados.rados.0.trial043.stdout:718: writing trial04342147-13 from 497573 to 1126516 tid 1 2026-02-20T11:18:55.796 INFO:tasks.rados.rados.0.trial043.stdout:718: writing trial04342147-13 from 1687631 to 2422408 tid 2 2026-02-20T11:18:55.801 INFO:tasks.rados.rados.0.trial043.stdout:718: writing trial04342147-13 from 3093218 to 3697573 tid 3 2026-02-20T11:18:55.801 INFO:tasks.rados.rados.0.trial043.stdout:719: snap_remove snap 90 2026-02-20T11:18:56.791 INFO:tasks.rados.rados.0.trial043.stdout:719: done (1 left) 2026-02-20T11:18:56.792 INFO:tasks.rados.rados.0.trial043.stdout:720: snap_create 2026-02-20T11:18:56.792 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:18:56.792 INFO:tasks.rados.rados.0.trial043.stdout:718: finishing write tid 1 to trial04342147-13 2026-02-20T11:18:56.792 INFO:tasks.rados.rados.0.trial043.stdout:718: finishing write tid 2 to trial04342147-13 2026-02-20T11:18:56.792 INFO:tasks.rados.rados.0.trial043.stdout:718: finishing write tid 3 to trial04342147-13 2026-02-20T11:18:56.792 INFO:tasks.rados.rados.0.trial043.stdout:718: finishing write tid 4 to trial04342147-13 2026-02-20T11:18:56.792 INFO:tasks.rados.rados.0.trial043.stdout:718: finishing write tid 5 to trial04342147-13 2026-02-20T11:18:56.792 INFO:tasks.rados.rados.0.trial043.stdout:718: oid 13 updating version 0 to 188 2026-02-20T11:18:56.792 INFO:tasks.rados.rados.0.trial043.stdout:718: oid 13 version 188 is already newer than 188 2026-02-20T11:18:56.792 INFO:tasks.rados.rados.0.trial043.stdout:718: oid 13 version 188 is already newer than 188 2026-02-20T11:18:56.792 INFO:tasks.rados.rados.0.trial043.stdout:718: oid 13 version 188 is already newer than 188 2026-02-20T11:18:56.793 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 13 v 188 (ObjNum 339 snap 92 seq_num 339) dirty exists 2026-02-20T11:18:56.793 INFO:tasks.rados.rados.0.trial043.stdout:718: left oid 13 (ObjNum 339 snap 92 seq_num 339) 2026-02-20T11:18:56.793 INFO:tasks.rados.rados.0.trial043.stdout:718: done (0 left) 2026-02-20T11:18:56.955 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:18:57.031 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:57.030+0000 7f297f7dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:57.033 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:57.033+0000 7f297f7dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:57.058 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:18:57.073 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:18:57.074 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:18:57.792 INFO:tasks.rados.rados.0.trial043.stdout:720: done (0 left) 2026-02-20T11:18:57.793 INFO:tasks.rados.rados.0.trial043.stdout:721: read oid 44 snap 79 2026-02-20T11:18:57.793 INFO:tasks.rados.rados.0.trial043.stdout:721: expect deleted 2026-02-20T11:18:57.793 INFO:tasks.rados.rados.0.trial043.stdout:722: read oid 47 snap -1 2026-02-20T11:18:57.793 INFO:tasks.rados.rados.0.trial043.stdout:722: expect deleted 2026-02-20T11:18:57.793 INFO:tasks.rados.rados.0.trial043.stdout:723: write oid 15 current snap is 93 2026-02-20T11:18:57.793 INFO:tasks.rados.rados.0.trial043.stdout:723: seq_num 340 ranges {514380=704192,1629760=478435,2604288=442602} 2026-02-20T11:18:57.797 INFO:tasks.rados.rados.0.trial043.stdout:723: writing trial04342147-15 from 514380 to 1218572 tid 1 2026-02-20T11:18:57.800 INFO:tasks.rados.rados.0.trial043.stdout:723: writing trial04342147-15 from 1629760 to 2108195 tid 2 2026-02-20T11:18:57.804 INFO:tasks.rados.rados.0.trial043.stdout:723: writing trial04342147-15 from 2604288 to 3046890 tid 3 2026-02-20T11:18:57.804 INFO:tasks.rados.rados.0.trial043.stdout:724: snap_create 2026-02-20T11:18:57.804 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:18:57.814 INFO:tasks.rados.rados.0.trial043.stdout:721: done (2 left) 2026-02-20T11:18:57.814 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:18:57.816 INFO:tasks.rados.rados.0.trial043.stdout:722: done (1 left) 2026-02-20T11:18:57.816 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:18:57.833 INFO:tasks.rados.rados.0.trial043.stdout:723: finishing write tid 1 to trial04342147-15 2026-02-20T11:18:57.843 INFO:tasks.rados.rados.0.trial043.stdout:723: finishing write tid 2 to trial04342147-15 2026-02-20T11:18:57.850 INFO:tasks.rados.rados.0.trial043.stdout:723: finishing write tid 3 to trial04342147-15 2026-02-20T11:18:57.856 INFO:tasks.rados.rados.0.trial043.stdout:723: finishing write tid 4 to trial04342147-15 2026-02-20T11:18:57.858 INFO:tasks.rados.rados.0.trial043.stdout:723: finishing write tid 5 to trial04342147-15 2026-02-20T11:18:57.858 INFO:tasks.rados.rados.0.trial043.stdout:723: oid 15 updating version 0 to 56 2026-02-20T11:18:57.858 INFO:tasks.rados.rados.0.trial043.stdout:723: oid 15 version 56 is already newer than 56 2026-02-20T11:18:57.858 INFO:tasks.rados.rados.0.trial043.stdout:723: oid 15 version 56 is already newer than 56 2026-02-20T11:18:57.858 INFO:tasks.rados.rados.0.trial043.stdout:723: oid 15 version 56 is already newer than 56 2026-02-20T11:18:57.858 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 15 v 56 (ObjNum 340 snap 93 seq_num 340) dirty exists 2026-02-20T11:18:57.859 INFO:tasks.rados.rados.0.trial043.stdout:723: left oid 15 (ObjNum 340 snap 93 seq_num 340) 2026-02-20T11:18:57.859 INFO:tasks.rados.rados.0.trial043.stdout:723: done (0 left) 2026-02-20T11:18:57.978 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~193s 2026-02-20T11:18:58.768 INFO:tasks.rados.rados.0.trial043.stdout:724: done (0 left) 2026-02-20T11:18:58.769 INFO:tasks.rados.rados.0.trial043.stdout:725: snap_create 2026-02-20T11:18:58.865 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:18:58.936 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:18:58.938 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:18:58.939 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:18:59.075 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:18:59.150 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:59.149+0000 7fe7c1a11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:59.152 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:18:59.152+0000 7fe7c1a11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:18:59.178 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:18:59.193 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:18:59.194 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:18:59.770 INFO:tasks.rados.rados.0.trial043.stdout:725: done (0 left) 2026-02-20T11:18:59.770 INFO:tasks.rados.rados.0.trial043.stdout:726: write (excl) oid 44 current snap is 95 2026-02-20T11:18:59.770 INFO:tasks.rados.rados.0.trial043.stdout:726: seq_num 341 ranges {531187=779441,2088242=738446,3431711=299476} 2026-02-20T11:18:59.774 INFO:tasks.rados.rados.0.trial043.stdout:726: writing trial04342147-44 from 531187 to 1310628 tid 1 2026-02-20T11:18:59.780 INFO:tasks.rados.rados.0.trial043.stdout:726: writing trial04342147-44 from 2088242 to 2826688 tid 2 2026-02-20T11:18:59.784 INFO:tasks.rados.rados.0.trial043.stdout:726: writing trial04342147-44 from 3431711 to 3731187 tid 3 2026-02-20T11:18:59.784 INFO:tasks.rados.rados.0.trial043.stdout:727: delete oid 38 current snap is 95 2026-02-20T11:18:59.807 INFO:tasks.rados.rados.0.trial043.stdout:726: finishing write tid 1 to trial04342147-44 2026-02-20T11:18:59.824 INFO:tasks.rados.rados.0.trial043.stdout:727: done (1 left) 2026-02-20T11:18:59.824 INFO:tasks.rados.rados.0.trial043.stdout:728: write (excl) oid 13 current snap is 95 2026-02-20T11:18:59.824 INFO:tasks.rados.rados.0.trial043.stdout:728: seq_num 343 ranges {564801=646292,1921559=742115,3202910=561891} 2026-02-20T11:18:59.829 INFO:tasks.rados.rados.0.trial043.stdout:728: writing trial04342147-13 from 564801 to 1211093 tid 1 2026-02-20T11:18:59.834 INFO:tasks.rados.rados.0.trial043.stdout:728: writing trial04342147-13 from 1921559 to 2663674 tid 2 2026-02-20T11:18:59.840 INFO:tasks.rados.rados.0.trial043.stdout:728: writing trial04342147-13 from 3202910 to 3764801 tid 3 2026-02-20T11:18:59.840 INFO:tasks.rados.rados.0.trial043.stdout:729: delete oid 17 current snap is 95 2026-02-20T11:18:59.840 INFO:tasks.rados.rados.0.trial043.stdout:726: finishing write tid 2 to trial04342147-44 2026-02-20T11:18:59.840 INFO:tasks.rados.rados.0.trial043.stdout:726: finishing write tid 3 to trial04342147-44 2026-02-20T11:18:59.840 INFO:tasks.rados.rados.0.trial043.stdout:726: finishing write tid 4 to trial04342147-44 2026-02-20T11:18:59.840 INFO:tasks.rados.rados.0.trial043.stdout:726: finishing write tid 5 to trial04342147-44 2026-02-20T11:18:59.840 INFO:tasks.rados.rados.0.trial043.stdout:726: oid 44 updating version 0 to 97 2026-02-20T11:18:59.840 INFO:tasks.rados.rados.0.trial043.stdout:726: oid 44 version 97 is already newer than 94 2026-02-20T11:18:59.840 INFO:tasks.rados.rados.0.trial043.stdout:726: oid 44 version 97 is already newer than 97 2026-02-20T11:18:59.840 INFO:tasks.rados.rados.0.trial043.stdout:726: oid 44 version 97 is already newer than 97 2026-02-20T11:18:59.841 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 44 v 97 (ObjNum 341 snap 95 seq_num 341) dirty exists 2026-02-20T11:18:59.841 INFO:tasks.rados.rados.0.trial043.stdout:726: left oid 44 (ObjNum 341 snap 95 seq_num 341) 2026-02-20T11:18:59.842 INFO:tasks.rados.rados.0.trial043.stdout:726: done (2 left) 2026-02-20T11:18:59.842 INFO:tasks.rados.rados.0.trial043.stdout:729: done (1 left) 2026-02-20T11:18:59.842 INFO:tasks.rados.rados.0.trial043.stdout:730: write oid 3 current snap is 95 2026-02-20T11:18:59.842 INFO:tasks.rados.rados.0.trial043.stdout:730: seq_num 345 ranges {598415=796790,1922170=745784,3141403=415315} 2026-02-20T11:18:59.847 INFO:tasks.rados.rados.0.trial043.stdout:730: writing trial04342147-3 from 598415 to 1395205 tid 1 2026-02-20T11:18:59.852 INFO:tasks.rados.rados.0.trial043.stdout:730: writing trial04342147-3 from 1922170 to 2667954 tid 2 2026-02-20T11:18:59.856 INFO:tasks.rados.rados.0.trial043.stdout:730: writing trial04342147-3 from 3141403 to 3556718 tid 3 2026-02-20T11:18:59.856 INFO:tasks.rados.rados.0.trial043.stdout:731: read oid 9 snap -1 2026-02-20T11:18:59.857 INFO:tasks.rados.rados.0.trial043.stdout:731: expect (ObjNum 282 snap 78 seq_num 282) 2026-02-20T11:18:59.857 INFO:tasks.rados.rados.0.trial043.stdout:732: write oid 16 current snap is 95 2026-02-20T11:18:59.857 INFO:tasks.rados.rados.0.trial043.stdout:732: seq_num 346 ranges {615222=472039,1580652=489442,2768826=626587} 2026-02-20T11:18:59.860 INFO:tasks.rados.rados.0.trial043.stdout:732: writing trial04342147-16 from 615222 to 1087261 tid 1 2026-02-20T11:18:59.864 INFO:tasks.rados.rados.0.trial043.stdout:732: writing trial04342147-16 from 1580652 to 2070094 tid 2 2026-02-20T11:18:59.869 INFO:tasks.rados.rados.0.trial043.stdout:732: writing trial04342147-16 from 2768826 to 3395413 tid 3 2026-02-20T11:18:59.869 INFO:tasks.rados.rados.0.trial043.stdout:733: read oid 22 snap 93 2026-02-20T11:18:59.870 INFO:tasks.rados.rados.0.trial043.stdout:733: expect (ObjNum 281 snap 77 seq_num 281) 2026-02-20T11:18:59.870 INFO:tasks.rados.rados.0.trial043.stdout:734: snap_remove snap 94 2026-02-20T11:19:00.802 INFO:tasks.rados.rados.0.trial043.stdout:734: done (5 left) 2026-02-20T11:19:00.802 INFO:tasks.rados.rados.0.trial043.stdout:735: write oid 27 current snap is 95 2026-02-20T11:19:00.802 INFO:tasks.rados.rados.0.trial043.stdout:728: finishing write tid 1 to trial04342147-13 2026-02-20T11:19:00.803 INFO:tasks.rados.rados.0.trial043.stdout:735: seq_num 347 ranges {632029=547288,1639134=749453,2796249=437859,3653524=178505} 2026-02-20T11:19:00.806 INFO:tasks.rados.rados.0.trial043.stdout:735: writing trial04342147-27 from 632029 to 1179317 tid 1 2026-02-20T11:19:00.811 INFO:tasks.rados.rados.0.trial043.stdout:735: writing trial04342147-27 from 1639134 to 2388587 tid 2 2026-02-20T11:19:00.815 INFO:tasks.rados.rados.0.trial043.stdout:735: writing trial04342147-27 from 2796249 to 3234108 tid 3 2026-02-20T11:19:00.817 INFO:tasks.rados.rados.0.trial043.stdout:735: writing trial04342147-27 from 3653524 to 3832029 tid 4 2026-02-20T11:19:00.817 INFO:tasks.rados.rados.0.trial043.stdout:736: snap_remove snap 92 2026-02-20T11:19:01.195 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:19:01.271 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:01.270+0000 7fae6205a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:01.273 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:01.273+0000 7fae6205a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:01.299 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:19:01.316 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:19:01.316 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:19:01.804 INFO:tasks.rados.rados.0.trial043.stdout:736: done (6 left) 2026-02-20T11:19:01.804 INFO:tasks.rados.rados.0.trial043.stdout:737: delete oid 1 current snap is 95 2026-02-20T11:19:01.804 INFO:tasks.rados.rados.0.trial043.stdout:731: expect (ObjNum 282 snap 78 seq_num 282) 2026-02-20T11:19:01.836 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2103026 2026-02-20T11:19:01.881 INFO:tasks.rados.rados.0.trial043.stdout:728: finishing write tid 2 to trial04342147-13 2026-02-20T11:19:01.881 INFO:tasks.rados.rados.0.trial043.stdout:728: finishing write tid 3 to trial04342147-13 2026-02-20T11:19:01.881 INFO:tasks.rados.rados.0.trial043.stdout:731: done (6 left) 2026-02-20T11:19:01.882 INFO:tasks.rados.rados.0.trial043.stdout:737: done (5 left) 2026-02-20T11:19:01.882 INFO:tasks.rados.rados.0.trial043.stdout:738: snap_remove snap 81 2026-02-20T11:19:02.810 INFO:tasks.rados.rados.0.trial043.stdout:738: done (5 left) 2026-02-20T11:19:02.810 INFO:tasks.rados.rados.0.trial043.stdout:739: delete oid 15 current snap is 95 2026-02-20T11:19:02.810 INFO:tasks.rados.rados.0.trial043.stdout:732: finishing write tid 1 to trial04342147-16 2026-02-20T11:19:02.810 INFO:tasks.rados.rados.0.trial043.stdout:728: finishing write tid 4 to trial04342147-13 2026-02-20T11:19:02.811 INFO:tasks.rados.rados.0.trial043.stdout:728: finishing write tid 5 to trial04342147-13 2026-02-20T11:19:02.811 INFO:tasks.rados.rados.0.trial043.stdout:728: oid 13 updating version 0 to 193 2026-02-20T11:19:02.811 INFO:tasks.rados.rados.0.trial043.stdout:728: oid 13 version 193 is already newer than 193 2026-02-20T11:19:02.811 INFO:tasks.rados.rados.0.trial043.stdout:728: oid 13 version 193 is already newer than 193 2026-02-20T11:19:02.811 INFO:tasks.rados.rados.0.trial043.stdout:728: oid 13 version 193 is already newer than 190 2026-02-20T11:19:02.811 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 13 v 193 (ObjNum 343 snap 95 seq_num 343) dirty exists 2026-02-20T11:19:02.811 INFO:tasks.rados.rados.0.trial043.stdout:728: left oid 13 (ObjNum 343 snap 95 seq_num 343) 2026-02-20T11:19:02.811 INFO:tasks.rados.rados.0.trial043.stdout:732: finishing write tid 2 to trial04342147-16 2026-02-20T11:19:02.811 INFO:tasks.rados.rados.0.trial043.stdout:730: finishing write tid 1 to trial04342147-3 2026-02-20T11:19:02.812 INFO:tasks.rados.rados.0.trial043.stdout:732: finishing write tid 3 to trial04342147-16 2026-02-20T11:19:02.812 INFO:tasks.rados.rados.0.trial043.stdout:730: finishing write tid 2 to trial04342147-3 2026-02-20T11:19:02.812 INFO:tasks.rados.rados.0.trial043.stdout:733: expect (ObjNum 281 snap 77 seq_num 281) 2026-02-20T11:19:02.863 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:02.953 INFO:tasks.rados.rados.0.trial043.stdout:732: finishing write tid 4 to trial04342147-16 2026-02-20T11:19:02.953 INFO:tasks.rados.rados.0.trial043.stdout:730: finishing write tid 3 to trial04342147-3 2026-02-20T11:19:02.953 INFO:tasks.rados.rados.0.trial043.stdout:728: done (5 left) 2026-02-20T11:19:02.953 INFO:tasks.rados.rados.0.trial043.stdout:733: done (4 left) 2026-02-20T11:19:02.953 INFO:tasks.rados.rados.0.trial043.stdout:739: done (3 left) 2026-02-20T11:19:02.953 INFO:tasks.rados.rados.0.trial043.stdout:740: snap_remove snap 91 2026-02-20T11:19:03.318 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:19:03.396 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:03.396+0000 7fe06c5ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:03.399 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:03.399+0000 7fe06c5ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:03.424 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:19:03.440 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:19:03.440 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:19:03.482 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~198s 2026-02-20T11:19:03.810 INFO:tasks.rados.rados.0.trial043.stdout:740: done (3 left) 2026-02-20T11:19:03.811 INFO:tasks.rados.rados.0.trial043.stdout:741: snap_remove snap 79 2026-02-20T11:19:03.940 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:19:04.011 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:19:04.014 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:19:04.015 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:19:04.815 INFO:tasks.rados.rados.0.trial043.stdout:741: done (3 left) 2026-02-20T11:19:04.815 INFO:tasks.rados.rados.0.trial043.stdout:742: delete oid 34 current snap is 95 2026-02-20T11:19:04.815 INFO:tasks.rados.rados.0.trial043.stdout:732: finishing write tid 5 to trial04342147-16 2026-02-20T11:19:04.815 INFO:tasks.rados.rados.0.trial043.stdout:732: oid 16 updating version 0 to 271 2026-02-20T11:19:04.815 INFO:tasks.rados.rados.0.trial043.stdout:732: oid 16 version 271 is already newer than 271 2026-02-20T11:19:04.815 INFO:tasks.rados.rados.0.trial043.stdout:732: oid 16 version 271 is already newer than 271 2026-02-20T11:19:04.815 INFO:tasks.rados.rados.0.trial043.stdout:732: oid 16 version 271 is already newer than 271 2026-02-20T11:19:04.815 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 16 v 271 (ObjNum 346 snap 95 seq_num 346) dirty exists 2026-02-20T11:19:04.815 INFO:tasks.rados.rados.0.trial043.stdout:732: left oid 16 (ObjNum 346 snap 95 seq_num 346) 2026-02-20T11:19:04.815 INFO:tasks.rados.rados.0.trial043.stdout:730: finishing write tid 4 to trial04342147-3 2026-02-20T11:19:04.815 INFO:tasks.rados.rados.0.trial043.stdout:730: finishing write tid 5 to trial04342147-3 2026-02-20T11:19:04.815 INFO:tasks.rados.rados.0.trial043.stdout:730: oid 3 updating version 0 to 198 2026-02-20T11:19:04.815 INFO:tasks.rados.rados.0.trial043.stdout:730: oid 3 version 198 is already newer than 198 2026-02-20T11:19:04.815 INFO:tasks.rados.rados.0.trial043.stdout:730: oid 3 version 198 is already newer than 198 2026-02-20T11:19:04.815 INFO:tasks.rados.rados.0.trial043.stdout:730: oid 3 version 198 is already newer than 198 2026-02-20T11:19:04.815 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 3 v 198 (ObjNum 345 snap 95 seq_num 345) dirty exists 2026-02-20T11:19:04.815 INFO:tasks.rados.rados.0.trial043.stdout:730: left oid 3 (ObjNum 345 snap 95 seq_num 345) 2026-02-20T11:19:04.816 INFO:tasks.rados.rados.0.trial043.stdout:735: finishing write tid 1 to trial04342147-27 2026-02-20T11:19:04.816 INFO:tasks.rados.rados.0.trial043.stdout:735: finishing write tid 2 to trial04342147-27 2026-02-20T11:19:04.816 INFO:tasks.rados.rados.0.trial043.stdout:735: finishing write tid 3 to trial04342147-27 2026-02-20T11:19:04.816 INFO:tasks.rados.rados.0.trial043.stdout:735: finishing write tid 4 to trial04342147-27 2026-02-20T11:19:04.816 INFO:tasks.rados.rados.0.trial043.stdout:735: finishing write tid 5 to trial04342147-27 2026-02-20T11:19:04.816 INFO:tasks.rados.rados.0.trial043.stdout:735: finishing write tid 6 to trial04342147-27 2026-02-20T11:19:04.816 INFO:tasks.rados.rados.0.trial043.stdout:735: oid 27 updating version 0 to 240 2026-02-20T11:19:04.816 INFO:tasks.rados.rados.0.trial043.stdout:735: oid 27 version 240 is already newer than 240 2026-02-20T11:19:04.816 INFO:tasks.rados.rados.0.trial043.stdout:735: oid 27 version 240 is already newer than 240 2026-02-20T11:19:04.816 INFO:tasks.rados.rados.0.trial043.stdout:735: oid 27 version 240 is already newer than 240 2026-02-20T11:19:04.816 INFO:tasks.rados.rados.0.trial043.stdout:735: oid 27 version 240 is already newer than 240 2026-02-20T11:19:04.816 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 27 v 240 (ObjNum 347 snap 95 seq_num 347) dirty exists 2026-02-20T11:19:04.816 INFO:tasks.rados.rados.0.trial043.stdout:735: left oid 27 (ObjNum 347 snap 95 seq_num 347) 2026-02-20T11:19:04.856 INFO:tasks.rados.rados.0.trial043.stdout:730: done (3 left) 2026-02-20T11:19:04.856 INFO:tasks.rados.rados.0.trial043.stdout:732: done (2 left) 2026-02-20T11:19:04.856 INFO:tasks.rados.rados.0.trial043.stdout:735: done (1 left) 2026-02-20T11:19:04.856 INFO:tasks.rados.rados.0.trial043.stdout:742: done (0 left) 2026-02-20T11:19:04.856 INFO:tasks.rados.rados.0.trial043.stdout:743: write (excl) oid 37 current snap is 95 2026-02-20T11:19:04.856 INFO:tasks.rados.rados.0.trial043.stdout:743: seq_num 351 ranges {699257=564637,1873062=640438,3305941=593316} 2026-02-20T11:19:04.860 INFO:tasks.rados.rados.0.trial043.stdout:743: writing trial04342147-37 from 699257 to 1263894 tid 1 2026-02-20T11:19:04.866 INFO:tasks.rados.rados.0.trial043.stdout:743: writing trial04342147-37 from 1873062 to 2513500 tid 2 2026-02-20T11:19:04.871 INFO:tasks.rados.rados.0.trial043.stdout:743: writing trial04342147-37 from 3305941 to 3899257 tid 3 2026-02-20T11:19:04.871 INFO:tasks.rados.rados.0.trial043.stdout:744: snap_remove snap 93 2026-02-20T11:19:05.441 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:19:05.518 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:05.516+0000 7fbcaa96f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:05.520 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:05.520+0000 7fbcaa96f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:05.546 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:19:05.561 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:19:05.562 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:19:05.820 INFO:tasks.rados.rados.0.trial043.stdout:744: done (1 left) 2026-02-20T11:19:05.820 INFO:tasks.rados.rados.0.trial043.stdout:745: read oid 16 snap -1 2026-02-20T11:19:05.821 INFO:tasks.rados.rados.0.trial043.stdout:745: expect (ObjNum 346 snap 95 seq_num 346) 2026-02-20T11:19:05.821 INFO:tasks.rados.rados.0.trial043.stdout:746: read oid 12 snap -1 2026-02-20T11:19:05.821 INFO:tasks.rados.rados.0.trial043.stdout:746: expect (ObjNum 225 snap 63 seq_num 225) 2026-02-20T11:19:05.821 INFO:tasks.rados.rados.0.trial043.stdout:747: delete oid 42 current snap is 95 2026-02-20T11:19:05.821 INFO:tasks.rados.rados.0.trial043.stdout:743: finishing write tid 1 to trial04342147-37 2026-02-20T11:19:05.821 INFO:tasks.rados.rados.0.trial043.stdout:743: finishing write tid 2 to trial04342147-37 2026-02-20T11:19:05.821 INFO:tasks.rados.rados.0.trial043.stdout:743: finishing write tid 3 to trial04342147-37 2026-02-20T11:19:05.821 INFO:tasks.rados.rados.0.trial043.stdout:743: finishing write tid 4 to trial04342147-37 2026-02-20T11:19:05.821 INFO:tasks.rados.rados.0.trial043.stdout:743: finishing write tid 5 to trial04342147-37 2026-02-20T11:19:05.821 INFO:tasks.rados.rados.0.trial043.stdout:743: oid 37 updating version 0 to 203 2026-02-20T11:19:05.821 INFO:tasks.rados.rados.0.trial043.stdout:743: oid 37 version 203 is already newer than 203 2026-02-20T11:19:05.822 INFO:tasks.rados.rados.0.trial043.stdout:743: oid 37 version 203 is already newer than 203 2026-02-20T11:19:05.822 INFO:tasks.rados.rados.0.trial043.stdout:743: oid 37 version 203 is already newer than 200 2026-02-20T11:19:05.822 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 37 v 203 (ObjNum 351 snap 95 seq_num 351) dirty exists 2026-02-20T11:19:05.822 INFO:tasks.rados.rados.0.trial043.stdout:743: left oid 37 (ObjNum 351 snap 95 seq_num 351) 2026-02-20T11:19:05.857 INFO:tasks.rados.rados.0.trial043.stdout:745: expect (ObjNum 346 snap 95 seq_num 346) 2026-02-20T11:19:05.857 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:05.928 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2624038 2026-02-20T11:19:05.942 INFO:tasks.rados.rados.0.trial043.stdout:743: done (3 left) 2026-02-20T11:19:05.943 INFO:tasks.rados.rados.0.trial043.stdout:745: done (2 left) 2026-02-20T11:19:05.943 INFO:tasks.rados.rados.0.trial043.stdout:747: done (1 left) 2026-02-20T11:19:05.943 INFO:tasks.rados.rados.0.trial043.stdout:748: read oid 1 snap -1 2026-02-20T11:19:05.943 INFO:tasks.rados.rados.0.trial043.stdout:748: expect deleted 2026-02-20T11:19:05.943 INFO:tasks.rados.rados.0.trial043.stdout:749: write oid 36 current snap is 95 2026-02-20T11:19:05.943 INFO:tasks.rados.rados.0.trial043.stdout:746: expect (ObjNum 225 snap 63 seq_num 225) 2026-02-20T11:19:05.954 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 3438737 2026-02-20T11:19:06.021 INFO:tasks.rados.rados.0.trial043.stdout:749: seq_num 353 ranges {732871=715135,1873673=644107,3244434=688437} 2026-02-20T11:19:06.027 INFO:tasks.rados.rados.0.trial043.stdout:749: writing trial04342147-36 from 732871 to 1448006 tid 1 2026-02-20T11:19:06.031 INFO:tasks.rados.rados.0.trial043.stdout:749: writing trial04342147-36 from 1873673 to 2517780 tid 2 2026-02-20T11:19:06.037 INFO:tasks.rados.rados.0.trial043.stdout:749: writing trial04342147-36 from 3244434 to 3932871 tid 3 2026-02-20T11:19:06.037 INFO:tasks.rados.rados.0.trial043.stdout:746: done (2 left) 2026-02-20T11:19:06.038 INFO:tasks.rados.rados.0.trial043.stdout:748: done (1 left) 2026-02-20T11:19:06.038 INFO:tasks.rados.rados.0.trial043.stdout:750: delete oid 27 current snap is 95 2026-02-20T11:19:06.048 INFO:tasks.rados.rados.0.trial043.stdout:749: finishing write tid 1 to trial04342147-36 2026-02-20T11:19:06.057 INFO:tasks.rados.rados.0.trial043.stdout:749: finishing write tid 2 to trial04342147-36 2026-02-20T11:19:06.062 INFO:tasks.rados.rados.0.trial043.stdout:750: done (1 left) 2026-02-20T11:19:06.062 INFO:tasks.rados.rados.0.trial043.stdout:751: write oid 21 current snap is 95 2026-02-20T11:19:06.063 INFO:tasks.rados.rados.0.trial043.stdout:751: seq_num 355 ranges {766485=465633,1990637=647776,3299280=667205} 2026-02-20T11:19:06.067 INFO:tasks.rados.rados.0.trial043.stdout:751: writing trial04342147-21 from 766485 to 1232118 tid 1 2026-02-20T11:19:06.073 INFO:tasks.rados.rados.0.trial043.stdout:751: writing trial04342147-21 from 1990637 to 2638413 tid 2 2026-02-20T11:19:06.080 INFO:tasks.rados.rados.0.trial043.stdout:751: writing trial04342147-21 from 3299280 to 3966485 tid 3 2026-02-20T11:19:06.080 INFO:tasks.rados.rados.0.trial043.stdout:752: delete oid 37 current snap is 95 2026-02-20T11:19:06.081 INFO:tasks.rados.rados.0.trial043.stdout:749: finishing write tid 3 to trial04342147-36 2026-02-20T11:19:06.081 INFO:tasks.rados.rados.0.trial043.stdout:749: finishing write tid 4 to trial04342147-36 2026-02-20T11:19:06.081 INFO:tasks.rados.rados.0.trial043.stdout:749: finishing write tid 5 to trial04342147-36 2026-02-20T11:19:06.081 INFO:tasks.rados.rados.0.trial043.stdout:749: oid 36 updating version 0 to 40 2026-02-20T11:19:06.081 INFO:tasks.rados.rados.0.trial043.stdout:749: oid 36 version 40 is already newer than 40 2026-02-20T11:19:06.082 INFO:tasks.rados.rados.0.trial043.stdout:749: oid 36 version 40 is already newer than 40 2026-02-20T11:19:06.082 INFO:tasks.rados.rados.0.trial043.stdout:749: oid 36 version 40 is already newer than 40 2026-02-20T11:19:06.082 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 36 v 40 (ObjNum 353 snap 95 seq_num 353) dirty exists 2026-02-20T11:19:06.082 INFO:tasks.rados.rados.0.trial043.stdout:749: left oid 36 (ObjNum 353 snap 95 seq_num 353) 2026-02-20T11:19:06.087 INFO:tasks.rados.rados.0.trial043.stdout:751: finishing write tid 1 to trial04342147-21 2026-02-20T11:19:06.094 INFO:tasks.rados.rados.0.trial043.stdout:751: finishing write tid 2 to trial04342147-21 2026-02-20T11:19:06.101 INFO:tasks.rados.rados.0.trial043.stdout:751: finishing write tid 3 to trial04342147-21 2026-02-20T11:19:06.106 INFO:tasks.rados.rados.0.trial043.stdout:751: finishing write tid 4 to trial04342147-21 2026-02-20T11:19:06.107 INFO:tasks.rados.rados.0.trial043.stdout:749: done (2 left) 2026-02-20T11:19:06.107 INFO:tasks.rados.rados.0.trial043.stdout:752: done (1 left) 2026-02-20T11:19:06.107 INFO:tasks.rados.rados.0.trial043.stdout:753: snap_create 2026-02-20T11:19:06.107 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:19:06.108 INFO:tasks.rados.rados.0.trial043.stdout:751: finishing write tid 5 to trial04342147-21 2026-02-20T11:19:06.108 INFO:tasks.rados.rados.0.trial043.stdout:751: oid 21 updating version 0 to 51 2026-02-20T11:19:06.109 INFO:tasks.rados.rados.0.trial043.stdout:751: oid 21 version 51 is already newer than 51 2026-02-20T11:19:06.109 INFO:tasks.rados.rados.0.trial043.stdout:751: oid 21 version 51 is already newer than 51 2026-02-20T11:19:06.109 INFO:tasks.rados.rados.0.trial043.stdout:751: oid 21 version 51 is already newer than 51 2026-02-20T11:19:06.109 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 21 v 51 (ObjNum 355 snap 95 seq_num 355) dirty exists 2026-02-20T11:19:06.109 INFO:tasks.rados.rados.0.trial043.stdout:751: left oid 21 (ObjNum 355 snap 95 seq_num 355) 2026-02-20T11:19:06.109 INFO:tasks.rados.rados.0.trial043.stdout:751: done (0 left) 2026-02-20T11:19:06.824 INFO:tasks.rados.rados.0.trial043.stdout:753: done (0 left) 2026-02-20T11:19:06.825 INFO:tasks.rados.rados.0.trial043.stdout:754: write (excl) oid 41 current snap is 96 2026-02-20T11:19:06.825 INFO:tasks.rados.rados.0.trial043.stdout:754: seq_num 357 ranges {400099=616131,1591248=408851} 2026-02-20T11:19:06.828 INFO:tasks.rados.rados.0.trial043.stdout:754: writing trial04342147-41 from 400099 to 1016230 tid 1 2026-02-20T11:19:06.832 INFO:tasks.rados.rados.0.trial043.stdout:754: writing trial04342147-41 from 1591248 to 2000099 tid 2 2026-02-20T11:19:06.832 INFO:tasks.rados.rados.0.trial043.stdout:755: read oid 17 snap -1 2026-02-20T11:19:06.832 INFO:tasks.rados.rados.0.trial043.stdout:755: expect deleted 2026-02-20T11:19:06.833 INFO:tasks.rados.rados.0.trial043.stdout:756: snap_remove snap 95 2026-02-20T11:19:07.563 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:19:07.639 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:07.638+0000 7f625ad93640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:07.642 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:07.642+0000 7f625ad93640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:07.668 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:19:07.684 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:19:07.685 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:19:07.826 INFO:tasks.rados.rados.0.trial043.stdout:756: done (2 left) 2026-02-20T11:19:07.827 INFO:tasks.rados.rados.0.trial043.stdout:757: write oid 7 current snap is 96 2026-02-20T11:19:07.827 INFO:tasks.rados.rados.0.trial043.stdout:757: seq_num 358 ranges {416906=407733,1366083=511456} 2026-02-20T11:19:07.830 INFO:tasks.rados.rados.0.trial043.stdout:757: writing trial04342147-7 from 416906 to 824639 tid 1 2026-02-20T11:19:07.834 INFO:tasks.rados.rados.0.trial043.stdout:757: writing trial04342147-7 from 1366083 to 1877539 tid 2 2026-02-20T11:19:07.834 INFO:tasks.rados.rados.0.trial043.stdout:758: write oid 15 current snap is 96 2026-02-20T11:19:07.834 INFO:tasks.rados.rados.0.trial043.stdout:758: seq_num 359 ranges {433713=482982,1424565=609148} 2026-02-20T11:19:07.838 INFO:tasks.rados.rados.0.trial043.stdout:758: writing trial04342147-15 from 433713 to 916695 tid 1 2026-02-20T11:19:07.842 INFO:tasks.rados.rados.0.trial043.stdout:758: writing trial04342147-15 from 1424565 to 2033713 tid 2 2026-02-20T11:19:07.842 INFO:tasks.rados.rados.0.trial043.stdout:759: write oid 11 current snap is 96 2026-02-20T11:19:07.843 INFO:tasks.rados.rados.0.trial043.stdout:759: seq_num 360 ranges {450520=558231,1483047=515125} 2026-02-20T11:19:07.846 INFO:tasks.rados.rados.0.trial043.stdout:759: writing trial04342147-11 from 450520 to 1008751 tid 1 2026-02-20T11:19:07.851 INFO:tasks.rados.rados.0.trial043.stdout:759: writing trial04342147-11 from 1483047 to 1998172 tid 2 2026-02-20T11:19:07.851 INFO:tasks.rados.rados.0.trial043.stdout:755: done (4 left) 2026-02-20T11:19:07.851 INFO:tasks.rados.rados.0.trial043.stdout:760: read oid 50 snap -1 2026-02-20T11:19:07.851 INFO:tasks.rados.rados.0.trial043.stdout:760: expect deleted 2026-02-20T11:19:07.851 INFO:tasks.rados.rados.0.trial043.stdout:761: snap_create 2026-02-20T11:19:07.851 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 5 2026-02-20T11:19:07.851 INFO:tasks.rados.rados.0.trial043.stdout:754: finishing write tid 1 to trial04342147-41 2026-02-20T11:19:07.851 INFO:tasks.rados.rados.0.trial043.stdout:754: finishing write tid 2 to trial04342147-41 2026-02-20T11:19:07.851 INFO:tasks.rados.rados.0.trial043.stdout:754: finishing write tid 3 to trial04342147-41 2026-02-20T11:19:07.852 INFO:tasks.rados.rados.0.trial043.stdout:754: finishing write tid 4 to trial04342147-41 2026-02-20T11:19:07.852 INFO:tasks.rados.rados.0.trial043.stdout:754: oid 41 updating version 0 to 218 2026-02-20T11:19:07.852 INFO:tasks.rados.rados.0.trial043.stdout:754: oid 41 updating version 218 to 220 2026-02-20T11:19:07.852 INFO:tasks.rados.rados.0.trial043.stdout:754: oid 41 version 220 is already newer than 220 2026-02-20T11:19:07.852 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 41 v 220 (ObjNum 357 snap 96 seq_num 357) dirty exists 2026-02-20T11:19:07.852 INFO:tasks.rados.rados.0.trial043.stdout:754: left oid 41 (ObjNum 357 snap 96 seq_num 357) 2026-02-20T11:19:07.852 INFO:tasks.rados.rados.0.trial043.stdout:754: done (4 left) 2026-02-20T11:19:07.852 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 4 2026-02-20T11:19:07.865 INFO:tasks.rados.rados.0.trial043.stdout:757: finishing write tid 1 to trial04342147-7 2026-02-20T11:19:07.865 INFO:tasks.rados.rados.0.trial043.stdout:760: done (3 left) 2026-02-20T11:19:07.865 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:19:07.872 INFO:tasks.rados.rados.0.trial043.stdout:757: finishing write tid 2 to trial04342147-7 2026-02-20T11:19:07.879 INFO:tasks.rados.rados.0.trial043.stdout:758: finishing write tid 1 to trial04342147-15 2026-02-20T11:19:07.879 INFO:tasks.rados.rados.0.trial043.stdout:757: finishing write tid 3 to trial04342147-7 2026-02-20T11:19:07.881 INFO:tasks.rados.rados.0.trial043.stdout:757: finishing write tid 4 to trial04342147-7 2026-02-20T11:19:07.881 INFO:tasks.rados.rados.0.trial043.stdout:757: oid 7 updating version 0 to 90 2026-02-20T11:19:07.881 INFO:tasks.rados.rados.0.trial043.stdout:757: oid 7 version 90 is already newer than 90 2026-02-20T11:19:07.881 INFO:tasks.rados.rados.0.trial043.stdout:757: oid 7 version 90 is already newer than 90 2026-02-20T11:19:07.881 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 7 v 90 (ObjNum 358 snap 96 seq_num 358) dirty exists 2026-02-20T11:19:07.881 INFO:tasks.rados.rados.0.trial043.stdout:757: left oid 7 (ObjNum 358 snap 96 seq_num 358) 2026-02-20T11:19:07.881 INFO:tasks.rados.rados.0.trial043.stdout:757: done (2 left) 2026-02-20T11:19:07.882 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:19:07.895 INFO:tasks.rados.rados.0.trial043.stdout:758: finishing write tid 2 to trial04342147-15 2026-02-20T11:19:07.911 INFO:tasks.rados.rados.0.trial043.stdout:758: finishing write tid 3 to trial04342147-15 2026-02-20T11:19:07.915 INFO:tasks.rados.rados.0.trial043.stdout:758: finishing write tid 4 to trial04342147-15 2026-02-20T11:19:07.915 INFO:tasks.rados.rados.0.trial043.stdout:758: oid 15 updating version 0 to 61 2026-02-20T11:19:07.915 INFO:tasks.rados.rados.0.trial043.stdout:758: oid 15 version 61 is already newer than 61 2026-02-20T11:19:07.915 INFO:tasks.rados.rados.0.trial043.stdout:758: oid 15 version 61 is already newer than 61 2026-02-20T11:19:07.915 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 15 v 61 (ObjNum 359 snap 96 seq_num 359) dirty exists 2026-02-20T11:19:07.916 INFO:tasks.rados.rados.0.trial043.stdout:758: left oid 15 (ObjNum 359 snap 96 seq_num 359) 2026-02-20T11:19:07.916 INFO:tasks.rados.rados.0.trial043.stdout:758: done (1 left) 2026-02-20T11:19:07.916 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:19:07.962 INFO:tasks.rados.rados.0.trial043.stdout:759: finishing write tid 1 to trial04342147-11 2026-02-20T11:19:07.967 INFO:tasks.rados.rados.0.trial043.stdout:759: finishing write tid 2 to trial04342147-11 2026-02-20T11:19:07.973 INFO:tasks.rados.rados.0.trial043.stdout:759: finishing write tid 3 to trial04342147-11 2026-02-20T11:19:07.975 INFO:tasks.rados.rados.0.trial043.stdout:759: finishing write tid 4 to trial04342147-11 2026-02-20T11:19:07.975 INFO:tasks.rados.rados.0.trial043.stdout:759: oid 11 updating version 0 to 288 2026-02-20T11:19:07.975 INFO:tasks.rados.rados.0.trial043.stdout:759: oid 11 version 288 is already newer than 288 2026-02-20T11:19:07.975 INFO:tasks.rados.rados.0.trial043.stdout:759: oid 11 version 288 is already newer than 288 2026-02-20T11:19:07.975 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 11 v 288 (ObjNum 360 snap 96 seq_num 360) dirty exists 2026-02-20T11:19:07.975 INFO:tasks.rados.rados.0.trial043.stdout:759: left oid 11 (ObjNum 360 snap 96 seq_num 360) 2026-02-20T11:19:07.975 INFO:tasks.rados.rados.0.trial043.stdout:759: done (0 left) 2026-02-20T11:19:08.771 INFO:tasks.rados.rados.0.trial043.stdout:761: done (0 left) 2026-02-20T11:19:08.771 INFO:tasks.rados.rados.0.trial043.stdout:762: write oid 35 current snap is 97 2026-02-20T11:19:08.771 INFO:tasks.rados.rados.0.trial043.stdout:762: seq_num 361 ranges {467327=633480,1825176=242151} 2026-02-20T11:19:08.775 INFO:tasks.rados.rados.0.trial043.stdout:762: writing trial04342147-35 from 467327 to 1100807 tid 1 2026-02-20T11:19:08.779 INFO:tasks.rados.rados.0.trial043.stdout:762: writing trial04342147-35 from 1825176 to 2067327 tid 2 2026-02-20T11:19:08.779 INFO:tasks.rados.rados.0.trial043.stdout:763: snap_create 2026-02-20T11:19:08.779 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:19:08.802 INFO:tasks.rados.rados.0.trial043.stdout:762: finishing write tid 1 to trial04342147-35 2026-02-20T11:19:08.807 INFO:tasks.rados.rados.0.trial043.stdout:762: finishing write tid 2 to trial04342147-35 2026-02-20T11:19:08.810 INFO:tasks.rados.rados.0.trial043.stdout:762: finishing write tid 3 to trial04342147-35 2026-02-20T11:19:08.812 INFO:tasks.rados.rados.0.trial043.stdout:762: finishing write tid 4 to trial04342147-35 2026-02-20T11:19:08.812 INFO:tasks.rados.rados.0.trial043.stdout:762: oid 35 updating version 0 to 207 2026-02-20T11:19:08.812 INFO:tasks.rados.rados.0.trial043.stdout:762: oid 35 version 207 is already newer than 207 2026-02-20T11:19:08.812 INFO:tasks.rados.rados.0.trial043.stdout:762: oid 35 version 207 is already newer than 207 2026-02-20T11:19:08.813 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 35 v 207 (ObjNum 361 snap 97 seq_num 361) dirty exists 2026-02-20T11:19:08.813 INFO:tasks.rados.rados.0.trial043.stdout:762: left oid 35 (ObjNum 361 snap 97 seq_num 361) 2026-02-20T11:19:08.813 INFO:tasks.rados.rados.0.trial043.stdout:762: done (0 left) 2026-02-20T11:19:08.986 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~204s 2026-02-20T11:19:09.015 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:19:09.087 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:19:09.090 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:19:09.091 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:19:09.686 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:19:09.762 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:09.761+0000 7f2f0950b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:09.764 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:09.764+0000 7f2f0950b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:09.770 INFO:tasks.rados.rados.0.trial043.stdout:763: done (0 left) 2026-02-20T11:19:09.770 INFO:tasks.rados.rados.0.trial043.stdout:764: write (excl) oid 7 current snap is 98 2026-02-20T11:19:09.771 INFO:tasks.rados.rados.0.trial043.stdout:764: seq_num 362 ranges {484134=708729,1883658=200476} 2026-02-20T11:19:09.774 INFO:tasks.rados.rados.0.trial043.stdout:764: writing trial04342147-7 from 484134 to 1192863 tid 1 2026-02-20T11:19:09.778 INFO:tasks.rados.rados.0.trial043.stdout:764: writing trial04342147-7 from 1883658 to 2084134 tid 2 2026-02-20T11:19:09.778 INFO:tasks.rados.rados.0.trial043.stdout:765: snap_remove snap 96 2026-02-20T11:19:09.802 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:19:09.821 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:19:09.822 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:19:10.772 INFO:tasks.rados.rados.0.trial043.stdout:765: done (1 left) 2026-02-20T11:19:10.773 INFO:tasks.rados.rados.0.trial043.stdout:766: snap_remove snap 97 2026-02-20T11:19:11.774 INFO:tasks.rados.rados.0.trial043.stdout:766: done (1 left) 2026-02-20T11:19:11.774 INFO:tasks.rados.rados.0.trial043.stdout:767: snap_create 2026-02-20T11:19:11.774 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:19:11.774 INFO:tasks.rados.rados.0.trial043.stdout:764: finishing write tid 1 to trial04342147-7 2026-02-20T11:19:11.774 INFO:tasks.rados.rados.0.trial043.stdout:764: finishing write tid 2 to trial04342147-7 2026-02-20T11:19:11.774 INFO:tasks.rados.rados.0.trial043.stdout:764: finishing write tid 3 to trial04342147-7 2026-02-20T11:19:11.774 INFO:tasks.rados.rados.0.trial043.stdout:764: finishing write tid 4 to trial04342147-7 2026-02-20T11:19:11.774 INFO:tasks.rados.rados.0.trial043.stdout:764: oid 7 updating version 0 to 94 2026-02-20T11:19:11.775 INFO:tasks.rados.rados.0.trial043.stdout:764: oid 7 version 94 is already newer than 94 2026-02-20T11:19:11.775 INFO:tasks.rados.rados.0.trial043.stdout:764: oid 7 version 94 is already newer than 92 2026-02-20T11:19:11.775 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 7 v 94 (ObjNum 362 snap 98 seq_num 362) dirty exists 2026-02-20T11:19:11.775 INFO:tasks.rados.rados.0.trial043.stdout:764: left oid 7 (ObjNum 362 snap 98 seq_num 362) 2026-02-20T11:19:11.775 INFO:tasks.rados.rados.0.trial043.stdout:764: done (0 left) 2026-02-20T11:19:11.823 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:19:11.897 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:11.896+0000 7f6f0d30b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:11.900 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:11.900+0000 7f6f0d30b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:11.926 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:19:11.941 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:19:11.941 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:19:12.777 INFO:tasks.rados.rados.0.trial043.stdout:767: done (0 left) 2026-02-20T11:19:12.777 INFO:tasks.rados.rados.0.trial043.stdout:768: read oid 26 snap -1 2026-02-20T11:19:12.777 INFO:tasks.rados.rados.0.trial043.stdout:768: expect (ObjNum 323 snap 88 seq_num 323) 2026-02-20T11:19:12.777 INFO:tasks.rados.rados.0.trial043.stdout:769: write oid 44 current snap is 99 2026-02-20T11:19:12.777 INFO:tasks.rados.rados.0.trial043.stdout:769: seq_num 363 ranges {500941=783978,1942140=158801} 2026-02-20T11:19:12.782 INFO:tasks.rados.rados.0.trial043.stdout:769: writing trial04342147-44 from 500941 to 1284919 tid 1 2026-02-20T11:19:12.786 INFO:tasks.rados.rados.0.trial043.stdout:769: writing trial04342147-44 from 1942140 to 2100941 tid 2 2026-02-20T11:19:12.786 INFO:tasks.rados.rados.0.trial043.stdout:770: read oid 5 snap -1 2026-02-20T11:19:12.786 INFO:tasks.rados.rados.0.trial043.stdout:770: expect deleted 2026-02-20T11:19:12.786 INFO:tasks.rados.rados.0.trial043.stdout:771: write (excl) oid 38 current snap is 99 2026-02-20T11:19:12.786 INFO:tasks.rados.rados.0.trial043.stdout:771: seq_num 364 ranges {517748=459227,1600622=406110} 2026-02-20T11:19:12.789 INFO:tasks.rados.rados.0.trial043.stdout:771: writing trial04342147-38 from 517748 to 976975 tid 1 2026-02-20T11:19:12.793 INFO:tasks.rados.rados.0.trial043.stdout:771: writing trial04342147-38 from 1600622 to 2006732 tid 2 2026-02-20T11:19:12.794 INFO:tasks.rados.rados.0.trial043.stdout:772: write oid 3 current snap is 99 2026-02-20T11:19:12.794 INFO:tasks.rados.rados.0.trial043.stdout:772: seq_num 365 ranges {534555=650829,1659104=475451} 2026-02-20T11:19:12.798 INFO:tasks.rados.rados.0.trial043.stdout:772: writing trial04342147-3 from 534555 to 1185384 tid 1 2026-02-20T11:19:12.801 INFO:tasks.rados.rados.0.trial043.stdout:772: writing trial04342147-3 from 1659104 to 2134555 tid 2 2026-02-20T11:19:12.802 INFO:tasks.rados.rados.0.trial043.stdout:773: delete oid 21 current snap is 99 2026-02-20T11:19:12.811 INFO:tasks.rados.rados.0.trial043.stdout:768: expect (ObjNum 323 snap 88 seq_num 323) 2026-02-20T11:19:12.811 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:12.849 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:12.918 INFO:tasks.rados.rados.0.trial043.stdout:768: done (5 left) 2026-02-20T11:19:12.919 INFO:tasks.rados.rados.0.trial043.stdout:770: done (4 left) 2026-02-20T11:19:12.919 INFO:tasks.rados.rados.0.trial043.stdout:773: done (3 left) 2026-02-20T11:19:12.919 INFO:tasks.rados.rados.0.trial043.stdout:774: write (excl) oid 16 current snap is 99 2026-02-20T11:19:12.919 INFO:tasks.rados.rados.0.trial043.stdout:771: finishing write tid 1 to trial04342147-38 2026-02-20T11:19:12.919 INFO:tasks.rados.rados.0.trial043.stdout:774: seq_num 367 ranges {568169=401327,1376068=669790} 2026-02-20T11:19:12.922 INFO:tasks.rados.rados.0.trial043.stdout:774: writing trial04342147-16 from 568169 to 969496 tid 1 2026-02-20T11:19:12.927 INFO:tasks.rados.rados.0.trial043.stdout:774: writing trial04342147-16 from 1376068 to 2045858 tid 2 2026-02-20T11:19:12.927 INFO:tasks.rados.rados.0.trial043.stdout:775: read oid 14 snap 98 2026-02-20T11:19:12.928 INFO:tasks.rados.rados.0.trial043.stdout:775: expect (ObjNum 251 snap 70 seq_num 251) 2026-02-20T11:19:12.928 INFO:tasks.rados.rados.0.trial043.stdout:776: read oid 8 snap -1 2026-02-20T11:19:12.928 INFO:tasks.rados.rados.0.trial043.stdout:776: expect (ObjNum 293 snap 81 seq_num 293) 2026-02-20T11:19:12.928 INFO:tasks.rados.rados.0.trial043.stdout:777: read oid 48 snap -1 2026-02-20T11:19:12.928 INFO:tasks.rados.rados.0.trial043.stdout:777: expect (ObjNum 337 snap 92 seq_num 337) 2026-02-20T11:19:12.928 INFO:tasks.rados.rados.0.trial043.stdout:778: read oid 25 snap -1 2026-02-20T11:19:12.928 INFO:tasks.rados.rados.0.trial043.stdout:778: expect (ObjNum 328 snap 89 seq_num 328) 2026-02-20T11:19:12.928 INFO:tasks.rados.rados.0.trial043.stdout:779: snap_create 2026-02-20T11:19:12.928 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 8 2026-02-20T11:19:12.928 INFO:tasks.rados.rados.0.trial043.stdout:772: finishing write tid 1 to trial04342147-3 2026-02-20T11:19:12.929 INFO:tasks.rados.rados.0.trial043.stdout:772: finishing write tid 2 to trial04342147-3 2026-02-20T11:19:12.929 INFO:tasks.rados.rados.0.trial043.stdout:772: finishing write tid 3 to trial04342147-3 2026-02-20T11:19:12.929 INFO:tasks.rados.rados.0.trial043.stdout:772: finishing write tid 4 to trial04342147-3 2026-02-20T11:19:12.929 INFO:tasks.rados.rados.0.trial043.stdout:772: oid 3 updating version 0 to 211 2026-02-20T11:19:12.929 INFO:tasks.rados.rados.0.trial043.stdout:772: oid 3 version 211 is already newer than 211 2026-02-20T11:19:12.929 INFO:tasks.rados.rados.0.trial043.stdout:772: oid 3 version 211 is already newer than 211 2026-02-20T11:19:12.929 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 3 v 211 (ObjNum 365 snap 99 seq_num 365) dirty exists 2026-02-20T11:19:12.929 INFO:tasks.rados.rados.0.trial043.stdout:772: left oid 3 (ObjNum 365 snap 99 seq_num 365) 2026-02-20T11:19:12.929 INFO:tasks.rados.rados.0.trial043.stdout:769: finishing write tid 1 to trial04342147-44 2026-02-20T11:19:12.929 INFO:tasks.rados.rados.0.trial043.stdout:771: finishing write tid 2 to trial04342147-38 2026-02-20T11:19:12.929 INFO:tasks.rados.rados.0.trial043.stdout:772: done (7 left) 2026-02-20T11:19:12.929 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 7 2026-02-20T11:19:12.930 INFO:tasks.rados.rados.0.trial043.stdout:769: finishing write tid 2 to trial04342147-44 2026-02-20T11:19:12.930 INFO:tasks.rados.rados.0.trial043.stdout:771: finishing write tid 3 to trial04342147-38 2026-02-20T11:19:12.930 INFO:tasks.rados.rados.0.trial043.stdout:771: finishing write tid 4 to trial04342147-38 2026-02-20T11:19:12.930 INFO:tasks.rados.rados.0.trial043.stdout:771: oid 38 updating version 0 to 26 2026-02-20T11:19:12.930 INFO:tasks.rados.rados.0.trial043.stdout:771: oid 38 version 26 is already newer than 24 2026-02-20T11:19:12.930 INFO:tasks.rados.rados.0.trial043.stdout:771: oid 38 version 26 is already newer than 26 2026-02-20T11:19:12.930 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 38 v 26 (ObjNum 364 snap 99 seq_num 364) dirty exists 2026-02-20T11:19:12.930 INFO:tasks.rados.rados.0.trial043.stdout:771: left oid 38 (ObjNum 364 snap 99 seq_num 364) 2026-02-20T11:19:12.930 INFO:tasks.rados.rados.0.trial043.stdout:769: finishing write tid 3 to trial04342147-44 2026-02-20T11:19:12.930 INFO:tasks.rados.rados.0.trial043.stdout:769: finishing write tid 4 to trial04342147-44 2026-02-20T11:19:12.930 INFO:tasks.rados.rados.0.trial043.stdout:769: oid 44 updating version 0 to 102 2026-02-20T11:19:12.930 INFO:tasks.rados.rados.0.trial043.stdout:769: oid 44 version 102 is already newer than 102 2026-02-20T11:19:12.930 INFO:tasks.rados.rados.0.trial043.stdout:769: oid 44 version 102 is already newer than 102 2026-02-20T11:19:12.931 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 44 v 102 (ObjNum 363 snap 99 seq_num 363) dirty exists 2026-02-20T11:19:12.931 INFO:tasks.rados.rados.0.trial043.stdout:769: left oid 44 (ObjNum 363 snap 99 seq_num 363) 2026-02-20T11:19:12.931 INFO:tasks.rados.rados.0.trial043.stdout:769: done (6 left) 2026-02-20T11:19:12.931 INFO:tasks.rados.rados.0.trial043.stdout:771: done (5 left) 2026-02-20T11:19:12.931 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 5 2026-02-20T11:19:12.942 INFO:tasks.rados.rados.0.trial043.stdout:777: expect (ObjNum 337 snap 92 seq_num 337) 2026-02-20T11:19:12.948 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2994747 2026-02-20T11:19:12.991 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:13.023 INFO:tasks.rados.rados.0.trial043.stdout:777: done (4 left) 2026-02-20T11:19:13.023 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 4 2026-02-20T11:19:13.023 INFO:tasks.rados.rados.0.trial043.stdout:778: expect (ObjNum 328 snap 89 seq_num 328) 2026-02-20T11:19:13.030 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2605382 2026-02-20T11:19:13.085 INFO:tasks.rados.rados.0.trial043.stdout:774: finishing write tid 1 to trial04342147-16 2026-02-20T11:19:13.086 INFO:tasks.rados.rados.0.trial043.stdout:775: expect (ObjNum 251 snap 70 seq_num 251) 2026-02-20T11:19:13.105 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1093487 2026-02-20T11:19:13.119 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1008325 2026-02-20T11:19:13.130 INFO:tasks.rados.rados.0.trial043.stdout:774: finishing write tid 2 to trial04342147-16 2026-02-20T11:19:13.130 INFO:tasks.rados.rados.0.trial043.stdout:775: done (3 left) 2026-02-20T11:19:13.130 INFO:tasks.rados.rados.0.trial043.stdout:778: done (2 left) 2026-02-20T11:19:13.130 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:19:13.130 INFO:tasks.rados.rados.0.trial043.stdout:774: finishing write tid 3 to trial04342147-16 2026-02-20T11:19:13.130 INFO:tasks.rados.rados.0.trial043.stdout:774: finishing write tid 4 to trial04342147-16 2026-02-20T11:19:13.130 INFO:tasks.rados.rados.0.trial043.stdout:774: oid 16 updating version 0 to 294 2026-02-20T11:19:13.131 INFO:tasks.rados.rados.0.trial043.stdout:774: oid 16 version 294 is already newer than 294 2026-02-20T11:19:13.131 INFO:tasks.rados.rados.0.trial043.stdout:774: oid 16 version 294 is already newer than 292 2026-02-20T11:19:13.131 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 16 v 294 (ObjNum 367 snap 99 seq_num 367) dirty exists 2026-02-20T11:19:13.131 INFO:tasks.rados.rados.0.trial043.stdout:774: left oid 16 (ObjNum 367 snap 99 seq_num 367) 2026-02-20T11:19:13.131 INFO:tasks.rados.rados.0.trial043.stdout:776: expect (ObjNum 293 snap 81 seq_num 293) 2026-02-20T11:19:13.131 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:13.215 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:13.258 INFO:tasks.rados.rados.0.trial043.stdout:774: done (1 left) 2026-02-20T11:19:13.258 INFO:tasks.rados.rados.0.trial043.stdout:776: done (0 left) 2026-02-20T11:19:13.777 INFO:tasks.rados.rados.0.trial043.stdout:779: done (0 left) 2026-02-20T11:19:13.778 INFO:tasks.rados.rados.0.trial043.stdout:780: snap_remove snap 98 2026-02-20T11:19:13.943 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:19:14.019 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:14.018+0000 7fc6e80cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:14.022 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:14.022+0000 7fc6e80cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:14.048 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:19:14.064 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:19:14.065 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:19:14.091 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:19:14.161 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:19:14.163 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:19:14.164 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:19:14.491 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~209s 2026-02-20T11:19:14.782 INFO:tasks.rados.rados.0.trial043.stdout:780: done (0 left) 2026-02-20T11:19:14.782 INFO:tasks.rados.rados.0.trial043.stdout:781: snap_remove snap 99 2026-02-20T11:19:15.788 INFO:tasks.rados.rados.0.trial043.stdout:781: done (0 left) 2026-02-20T11:19:15.788 INFO:tasks.rados.rados.0.trial043.stdout:782: read oid 24 snap -1 2026-02-20T11:19:15.788 INFO:tasks.rados.rados.0.trial043.stdout:782: expect (ObjNum 257 snap 71 seq_num 257) 2026-02-20T11:19:15.789 INFO:tasks.rados.rados.0.trial043.stdout:783: read oid 50 snap -1 2026-02-20T11:19:15.789 INFO:tasks.rados.rados.0.trial043.stdout:783: expect deleted 2026-02-20T11:19:15.789 INFO:tasks.rados.rados.0.trial043.stdout:784: read oid 31 snap -1 2026-02-20T11:19:15.789 INFO:tasks.rados.rados.0.trial043.stdout:784: expect deleted 2026-02-20T11:19:15.789 INFO:tasks.rados.rados.0.trial043.stdout:785: read oid 33 snap -1 2026-02-20T11:19:15.789 INFO:tasks.rados.rados.0.trial043.stdout:785: expect (ObjNum 335 snap 91 seq_num 335) 2026-02-20T11:19:15.790 INFO:tasks.rados.rados.0.trial043.stdout:786: delete oid 37 current snap is 100 2026-02-20T11:19:15.801 INFO:tasks.rados.rados.0.trial043.stdout:786: done (4 left) 2026-02-20T11:19:15.801 INFO:tasks.rados.rados.0.trial043.stdout:787: delete oid 17 current snap is 100 2026-02-20T11:19:15.820 INFO:tasks.rados.rados.0.trial043.stdout:783: done (4 left) 2026-02-20T11:19:15.820 INFO:tasks.rados.rados.0.trial043.stdout:784: done (3 left) 2026-02-20T11:19:15.821 INFO:tasks.rados.rados.0.trial043.stdout:787: done (2 left) 2026-02-20T11:19:15.821 INFO:tasks.rados.rados.0.trial043.stdout:788: write oid 12 current snap is 100 2026-02-20T11:19:15.821 INFO:tasks.rados.rados.0.trial043.stdout:788: seq_num 370 ranges {618590=627074,1835161=383429} 2026-02-20T11:19:15.825 INFO:tasks.rados.rados.0.trial043.stdout:788: writing trial04342147-12 from 618590 to 1245664 tid 1 2026-02-20T11:19:15.829 INFO:tasks.rados.rados.0.trial043.stdout:788: writing trial04342147-12 from 1835161 to 2218590 tid 2 2026-02-20T11:19:15.830 INFO:tasks.rados.rados.0.trial043.stdout:789: read oid 35 snap -1 2026-02-20T11:19:15.830 INFO:tasks.rados.rados.0.trial043.stdout:789: expect (ObjNum 361 snap 97 seq_num 361) 2026-02-20T11:19:15.830 INFO:tasks.rados.rados.0.trial043.stdout:790: delete oid 48 current snap is 100 2026-02-20T11:19:15.830 INFO:tasks.rados.rados.0.trial043.stdout:782: expect (ObjNum 257 snap 71 seq_num 257) 2026-02-20T11:19:15.866 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 135697 2026-02-20T11:19:15.923 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1784349 2026-02-20T11:19:15.932 INFO:tasks.rados.rados.0.trial043.stdout:785: expect (ObjNum 335 snap 91 seq_num 335) 2026-02-20T11:19:15.980 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:16.066 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:19:16.089 INFO:tasks.rados.rados.0.trial043.stdout:788: finishing write tid 1 to trial04342147-12 2026-02-20T11:19:16.089 INFO:tasks.rados.rados.0.trial043.stdout:789: expect (ObjNum 361 snap 97 seq_num 361) 2026-02-20T11:19:16.101 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:16.119 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:16.137 INFO:tasks.rados.rados.0.trial043.stdout:788: finishing write tid 2 to trial04342147-12 2026-02-20T11:19:16.137 INFO:tasks.rados.rados.0.trial043.stdout:788: finishing write tid 3 to trial04342147-12 2026-02-20T11:19:16.137 INFO:tasks.rados.rados.0.trial043.stdout:782: done (4 left) 2026-02-20T11:19:16.137 INFO:tasks.rados.rados.0.trial043.stdout:785: done (3 left) 2026-02-20T11:19:16.137 INFO:tasks.rados.rados.0.trial043.stdout:789: done (2 left) 2026-02-20T11:19:16.138 INFO:tasks.rados.rados.0.trial043.stdout:790: done (1 left) 2026-02-20T11:19:16.138 INFO:tasks.rados.rados.0.trial043.stdout:791: write oid 18 current snap is 100 2026-02-20T11:19:16.138 INFO:tasks.rados.rados.0.trial043.stdout:788: finishing write tid 4 to trial04342147-12 2026-02-20T11:19:16.138 INFO:tasks.rados.rados.0.trial043.stdout:788: oid 12 updating version 0 to 105 2026-02-20T11:19:16.138 INFO:tasks.rados.rados.0.trial043.stdout:788: oid 12 version 105 is already newer than 105 2026-02-20T11:19:16.138 INFO:tasks.rados.rados.0.trial043.stdout:788: oid 12 version 105 is already newer than 105 2026-02-20T11:19:16.138 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 12 v 105 (ObjNum 370 snap 100 seq_num 370) dirty exists 2026-02-20T11:19:16.138 INFO:tasks.rados.rados.0.trial043.stdout:788: left oid 12 (ObjNum 370 snap 100 seq_num 370) 2026-02-20T11:19:16.138 INFO:tasks.rados.rados.0.trial043.stdout:791: seq_num 372 ranges {652204=777572,1952125=300079} 2026-02-20T11:19:16.143 INFO:tasks.rados.rados.0.trial043.stdout:791: writing trial04342147-18 from 652204 to 1429776 tid 1 2026-02-20T11:19:16.143 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:16.142+0000 7f25adfdc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:16.145 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:16.146+0000 7f25adfdc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:16.146 INFO:tasks.rados.rados.0.trial043.stdout:791: writing trial04342147-18 from 1952125 to 2252204 tid 2 2026-02-20T11:19:16.146 INFO:tasks.rados.rados.0.trial043.stdout:788: done (1 left) 2026-02-20T11:19:16.146 INFO:tasks.rados.rados.0.trial043.stdout:792: write (excl) oid 36 current snap is 100 2026-02-20T11:19:16.146 INFO:tasks.rados.rados.0.trial043.stdout:792: seq_num 373 ranges {669011=569174,2010607=258404} 2026-02-20T11:19:16.150 INFO:tasks.rados.rados.0.trial043.stdout:792: writing trial04342147-36 from 669011 to 1238185 tid 1 2026-02-20T11:19:16.156 INFO:tasks.rados.rados.0.trial043.stdout:792: writing trial04342147-36 from 2010607 to 2269011 tid 2 2026-02-20T11:19:16.156 INFO:tasks.rados.rados.0.trial043.stdout:793: read oid 20 snap -1 2026-02-20T11:19:16.156 INFO:tasks.rados.rados.0.trial043.stdout:793: expect (ObjNum 128 snap 17 seq_num 128) 2026-02-20T11:19:16.156 INFO:tasks.rados.rados.0.trial043.stdout:794: read oid 25 snap -1 2026-02-20T11:19:16.156 INFO:tasks.rados.rados.0.trial043.stdout:794: expect (ObjNum 328 snap 89 seq_num 328) 2026-02-20T11:19:16.157 INFO:tasks.rados.rados.0.trial043.stdout:795: read oid 42 snap -1 2026-02-20T11:19:16.157 INFO:tasks.rados.rados.0.trial043.stdout:795: expect deleted 2026-02-20T11:19:16.157 INFO:tasks.rados.rados.0.trial043.stdout:796: read oid 4 snap -1 2026-02-20T11:19:16.157 INFO:tasks.rados.rados.0.trial043.stdout:796: expect (ObjNum 310 snap 87 seq_num 310) 2026-02-20T11:19:16.157 INFO:tasks.rados.rados.0.trial043.stdout:797: read oid 37 snap -1 2026-02-20T11:19:16.157 INFO:tasks.rados.rados.0.trial043.stdout:797: expect deleted 2026-02-20T11:19:16.157 INFO:tasks.rados.rados.0.trial043.stdout:798: read oid 9 snap -1 2026-02-20T11:19:16.157 INFO:tasks.rados.rados.0.trial043.stdout:798: expect (ObjNum 282 snap 78 seq_num 282) 2026-02-20T11:19:16.157 INFO:tasks.rados.rados.0.trial043.stdout:799: read oid 2 snap -1 2026-02-20T11:19:16.157 INFO:tasks.rados.rados.0.trial043.stdout:799: expect deleted 2026-02-20T11:19:16.157 INFO:tasks.rados.rados.0.trial043.stdout:800: read oid 19 snap -1 2026-02-20T11:19:16.157 INFO:tasks.rados.rados.0.trial043.stdout:800: expect deleted 2026-02-20T11:19:16.157 INFO:tasks.rados.rados.0.trial043.stdout:801: write (excl) oid 24 current snap is 100 2026-02-20T11:19:16.158 INFO:tasks.rados.rados.0.trial043.stdout:801: seq_num 374 ranges {685818=644423,2069089=216729} 2026-02-20T11:19:16.160 INFO:tasks.rados.rados.0.trial043.stdout:801: writing trial04342147-24 from 685818 to 1330241 tid 1 2026-02-20T11:19:16.164 INFO:tasks.rados.rados.0.trial043.stdout:801: writing trial04342147-24 from 2069089 to 2285818 tid 2 2026-02-20T11:19:16.164 INFO:tasks.rados.rados.0.trial043.stdout:802: write (excl) oid 3 current snap is 100 2026-02-20T11:19:16.164 INFO:tasks.rados.rados.0.trial043.stdout:802: seq_num 375 ranges {702625=719672,2127571=175054} 2026-02-20T11:19:16.169 INFO:tasks.rados.rados.0.trial043.stdout:802: writing trial04342147-3 from 702625 to 1422297 tid 1 2026-02-20T11:19:16.173 INFO:tasks.rados.rados.0.trial043.stdout:802: writing trial04342147-3 from 2127571 to 2302625 tid 2 2026-02-20T11:19:16.173 INFO:tasks.rados.rados.0.trial043.stdout:803: read oid 7 snap -1 2026-02-20T11:19:16.173 INFO:tasks.rados.rados.0.trial043.stdout:803: expect (ObjNum 362 snap 98 seq_num 362) 2026-02-20T11:19:16.173 INFO:tasks.rados.rados.0.trial043.stdout:804: write (excl) oid 38 current snap is 100 2026-02-20T11:19:16.173 INFO:tasks.rados.rados.0.trial043.stdout:804: seq_num 376 ranges {719432=794921,2186053=133379} 2026-02-20T11:19:16.178 INFO:tasks.rados.rados.0.trial043.stdout:804: writing trial04342147-38 from 719432 to 1514353 tid 1 2026-02-20T11:19:16.182 INFO:tasks.rados.rados.0.trial043.stdout:804: writing trial04342147-38 from 2186053 to 2319432 tid 2 2026-02-20T11:19:16.182 INFO:tasks.rados.rados.0.trial043.stdout:805: read oid 39 snap -1 2026-02-20T11:19:16.182 INFO:tasks.rados.rados.0.trial043.stdout:805: expect deleted 2026-02-20T11:19:16.183 INFO:tasks.rados.rados.0.trial043.stdout:806: read oid 16 snap -1 2026-02-20T11:19:16.183 INFO:tasks.rados.rados.0.trial043.stdout:806: expect (ObjNum 367 snap 99 seq_num 367) 2026-02-20T11:19:16.183 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 16 2026-02-20T11:19:16.183 INFO:tasks.rados.rados.0.trial043.stdout:795: done (15 left) 2026-02-20T11:19:16.183 INFO:tasks.rados.rados.0.trial043.stdout:799: done (14 left) 2026-02-20T11:19:16.183 INFO:tasks.rados.rados.0.trial043.stdout:800: done (13 left) 2026-02-20T11:19:16.183 INFO:tasks.rados.rados.0.trial043.stdout:807: read oid 45 snap -1 2026-02-20T11:19:16.183 INFO:tasks.rados.rados.0.trial043.stdout:807: expect deleted 2026-02-20T11:19:16.183 INFO:tasks.rados.rados.0.trial043.stdout:808: delete oid 40 current snap is 100 2026-02-20T11:19:16.183 INFO:tasks.rados.rados.0.trial043.stdout:792: finishing write tid 1 to trial04342147-36 2026-02-20T11:19:16.183 INFO:tasks.rados.rados.0.trial043.stdout:794: expect (ObjNum 328 snap 89 seq_num 328) 2026-02-20T11:19:16.183 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:16.194 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:19:16.233 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:16.249 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:19:16.250 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:19:16.264 INFO:tasks.rados.rados.0.trial043.stdout:794: done (14 left) 2026-02-20T11:19:16.264 INFO:tasks.rados.rados.0.trial043.stdout:808: done (13 left) 2026-02-20T11:19:16.264 INFO:tasks.rados.rados.0.trial043.stdout:809: delete oid 12 current snap is 100 2026-02-20T11:19:16.264 INFO:tasks.rados.rados.0.trial043.stdout:791: finishing write tid 1 to trial04342147-18 2026-02-20T11:19:16.264 INFO:tasks.rados.rados.0.trial043.stdout:793: expect (ObjNum 128 snap 17 seq_num 128) 2026-02-20T11:19:16.265 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:16.305 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2101593 2026-02-20T11:19:16.310 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 813783 2026-02-20T11:19:16.327 INFO:tasks.rados.rados.0.trial043.stdout:791: finishing write tid 2 to trial04342147-18 2026-02-20T11:19:16.327 INFO:tasks.rados.rados.0.trial043.stdout:803: expect (ObjNum 362 snap 98 seq_num 362) 2026-02-20T11:19:16.351 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2080694 2026-02-20T11:19:16.414 INFO:tasks.rados.rados.0.trial043.stdout:793: done (13 left) 2026-02-20T11:19:16.414 INFO:tasks.rados.rados.0.trial043.stdout:797: done (12 left) 2026-02-20T11:19:16.415 INFO:tasks.rados.rados.0.trial043.stdout:803: done (11 left) 2026-02-20T11:19:16.415 INFO:tasks.rados.rados.0.trial043.stdout:805: done (10 left) 2026-02-20T11:19:16.415 INFO:tasks.rados.rados.0.trial043.stdout:809: done (9 left) 2026-02-20T11:19:16.415 INFO:tasks.rados.rados.0.trial043.stdout:810: snap_create 2026-02-20T11:19:16.415 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 9 2026-02-20T11:19:16.415 INFO:tasks.rados.rados.0.trial043.stdout:802: finishing write tid 1 to trial04342147-3 2026-02-20T11:19:16.415 INFO:tasks.rados.rados.0.trial043.stdout:792: finishing write tid 2 to trial04342147-36 2026-02-20T11:19:16.415 INFO:tasks.rados.rados.0.trial043.stdout:798: expect (ObjNum 282 snap 78 seq_num 282) 2026-02-20T11:19:16.415 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:16.500 INFO:tasks.rados.rados.0.trial043.stdout:798: done (8 left) 2026-02-20T11:19:16.500 INFO:tasks.rados.rados.0.trial043.stdout:807: done (7 left) 2026-02-20T11:19:16.500 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 7 2026-02-20T11:19:16.500 INFO:tasks.rados.rados.0.trial043.stdout:796: expect (ObjNum 310 snap 87 seq_num 310) 2026-02-20T11:19:16.500 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:16.676 INFO:tasks.rados.rados.0.trial043.stdout:806: expect (ObjNum 367 snap 99 seq_num 367) 2026-02-20T11:19:16.748 INFO:tasks.rados.rados.0.trial043.stdout:801: finishing write tid 1 to trial04342147-24 2026-02-20T11:19:16.748 INFO:tasks.rados.rados.0.trial043.stdout:796: done (6 left) 2026-02-20T11:19:16.749 INFO:tasks.rados.rados.0.trial043.stdout:806: done (5 left) 2026-02-20T11:19:16.749 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 5 2026-02-20T11:19:16.749 INFO:tasks.rados.rados.0.trial043.stdout:791: finishing write tid 3 to trial04342147-18 2026-02-20T11:19:16.749 INFO:tasks.rados.rados.0.trial043.stdout:802: finishing write tid 2 to trial04342147-3 2026-02-20T11:19:16.749 INFO:tasks.rados.rados.0.trial043.stdout:802: finishing write tid 3 to trial04342147-3 2026-02-20T11:19:16.749 INFO:tasks.rados.rados.0.trial043.stdout:791: finishing write tid 4 to trial04342147-18 2026-02-20T11:19:16.749 INFO:tasks.rados.rados.0.trial043.stdout:791: oid 18 updating version 0 to 56 2026-02-20T11:19:16.749 INFO:tasks.rados.rados.0.trial043.stdout:791: oid 18 version 56 is already newer than 56 2026-02-20T11:19:16.749 INFO:tasks.rados.rados.0.trial043.stdout:791: oid 18 version 56 is already newer than 56 2026-02-20T11:19:16.749 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 18 v 56 (ObjNum 372 snap 100 seq_num 372) dirty exists 2026-02-20T11:19:16.749 INFO:tasks.rados.rados.0.trial043.stdout:791: left oid 18 (ObjNum 372 snap 100 seq_num 372) 2026-02-20T11:19:16.749 INFO:tasks.rados.rados.0.trial043.stdout:802: finishing write tid 4 to trial04342147-3 2026-02-20T11:19:16.749 INFO:tasks.rados.rados.0.trial043.stdout:802: oid 3 updating version 0 to 212 2026-02-20T11:19:16.749 INFO:tasks.rados.rados.0.trial043.stdout:802: oid 3 updating version 212 to 214 2026-02-20T11:19:16.749 INFO:tasks.rados.rados.0.trial043.stdout:802: oid 3 version 214 is already newer than 214 2026-02-20T11:19:16.749 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 3 v 214 (ObjNum 375 snap 100 seq_num 375) dirty exists 2026-02-20T11:19:16.749 INFO:tasks.rados.rados.0.trial043.stdout:802: left oid 3 (ObjNum 375 snap 100 seq_num 375) 2026-02-20T11:19:16.749 INFO:tasks.rados.rados.0.trial043.stdout:801: finishing write tid 2 to trial04342147-24 2026-02-20T11:19:16.749 INFO:tasks.rados.rados.0.trial043.stdout:804: finishing write tid 1 to trial04342147-38 2026-02-20T11:19:16.749 INFO:tasks.rados.rados.0.trial043.stdout:801: finishing write tid 3 to trial04342147-24 2026-02-20T11:19:16.749 INFO:tasks.rados.rados.0.trial043.stdout:801: finishing write tid 4 to trial04342147-24 2026-02-20T11:19:16.749 INFO:tasks.rados.rados.0.trial043.stdout:801: oid 24 updating version 0 to 64 2026-02-20T11:19:16.749 INFO:tasks.rados.rados.0.trial043.stdout:801: oid 24 version 64 is already newer than 64 2026-02-20T11:19:16.749 INFO:tasks.rados.rados.0.trial043.stdout:801: oid 24 version 64 is already newer than 62 2026-02-20T11:19:16.750 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 24 v 64 (ObjNum 374 snap 100 seq_num 374) dirty exists 2026-02-20T11:19:16.750 INFO:tasks.rados.rados.0.trial043.stdout:801: left oid 24 (ObjNum 374 snap 100 seq_num 374) 2026-02-20T11:19:16.750 INFO:tasks.rados.rados.0.trial043.stdout:791: done (4 left) 2026-02-20T11:19:16.750 INFO:tasks.rados.rados.0.trial043.stdout:801: done (3 left) 2026-02-20T11:19:16.750 INFO:tasks.rados.rados.0.trial043.stdout:802: done (2 left) 2026-02-20T11:19:16.750 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:19:16.750 INFO:tasks.rados.rados.0.trial043.stdout:804: finishing write tid 2 to trial04342147-38 2026-02-20T11:19:16.750 INFO:tasks.rados.rados.0.trial043.stdout:792: finishing write tid 3 to trial04342147-36 2026-02-20T11:19:16.750 INFO:tasks.rados.rados.0.trial043.stdout:792: finishing write tid 4 to trial04342147-36 2026-02-20T11:19:16.750 INFO:tasks.rados.rados.0.trial043.stdout:792: oid 36 updating version 0 to 43 2026-02-20T11:19:16.750 INFO:tasks.rados.rados.0.trial043.stdout:792: oid 36 version 43 is already newer than 41 2026-02-20T11:19:16.750 INFO:tasks.rados.rados.0.trial043.stdout:792: oid 36 version 43 is already newer than 43 2026-02-20T11:19:16.750 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 36 v 43 (ObjNum 373 snap 100 seq_num 373) dirty exists 2026-02-20T11:19:16.750 INFO:tasks.rados.rados.0.trial043.stdout:792: left oid 36 (ObjNum 373 snap 100 seq_num 373) 2026-02-20T11:19:16.750 INFO:tasks.rados.rados.0.trial043.stdout:804: finishing write tid 3 to trial04342147-38 2026-02-20T11:19:16.750 INFO:tasks.rados.rados.0.trial043.stdout:804: finishing write tid 4 to trial04342147-38 2026-02-20T11:19:16.750 INFO:tasks.rados.rados.0.trial043.stdout:804: oid 38 updating version 0 to 27 2026-02-20T11:19:16.750 INFO:tasks.rados.rados.0.trial043.stdout:804: oid 38 updating version 27 to 29 2026-02-20T11:19:16.750 INFO:tasks.rados.rados.0.trial043.stdout:804: oid 38 version 29 is already newer than 29 2026-02-20T11:19:16.750 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 38 v 29 (ObjNum 376 snap 100 seq_num 376) dirty exists 2026-02-20T11:19:16.750 INFO:tasks.rados.rados.0.trial043.stdout:804: left oid 38 (ObjNum 376 snap 100 seq_num 376) 2026-02-20T11:19:16.750 INFO:tasks.rados.rados.0.trial043.stdout:792: done (1 left) 2026-02-20T11:19:16.750 INFO:tasks.rados.rados.0.trial043.stdout:804: done (0 left) 2026-02-20T11:19:16.788 INFO:tasks.rados.rados.0.trial043.stdout:810: done (0 left) 2026-02-20T11:19:16.788 INFO:tasks.rados.rados.0.trial043.stdout:811: delete oid 17 current snap is 101 2026-02-20T11:19:16.810 INFO:tasks.rados.rados.0.trial043.stdout:811: done (0 left) 2026-02-20T11:19:16.810 INFO:tasks.rados.rados.0.trial043.stdout:812: write oid 1 current snap is 101 2026-02-20T11:19:16.810 INFO:tasks.rados.rados.0.trial043.stdout:812: seq_num 380 ranges {786660=695917,1903628=435462} 2026-02-20T11:19:16.816 INFO:tasks.rados.rados.0.trial043.stdout:812: writing trial04342147-1 from 786660 to 1482577 tid 1 2026-02-20T11:19:16.819 INFO:tasks.rados.rados.0.trial043.stdout:812: writing trial04342147-1 from 1903628 to 2339090 tid 2 2026-02-20T11:19:16.819 INFO:tasks.rados.rados.0.trial043.stdout:813: snap_remove snap 100 2026-02-20T11:19:17.792 INFO:tasks.rados.rados.0.trial043.stdout:813: done (1 left) 2026-02-20T11:19:17.793 INFO:tasks.rados.rados.0.trial043.stdout:814: read oid 7 snap -1 2026-02-20T11:19:17.793 INFO:tasks.rados.rados.0.trial043.stdout:814: expect (ObjNum 362 snap 98 seq_num 362) 2026-02-20T11:19:17.793 INFO:tasks.rados.rados.0.trial043.stdout:815: delete oid 50 current snap is 101 2026-02-20T11:19:17.793 INFO:tasks.rados.rados.0.trial043.stdout:812: finishing write tid 1 to trial04342147-1 2026-02-20T11:19:17.793 INFO:tasks.rados.rados.0.trial043.stdout:812: finishing write tid 2 to trial04342147-1 2026-02-20T11:19:17.793 INFO:tasks.rados.rados.0.trial043.stdout:812: finishing write tid 3 to trial04342147-1 2026-02-20T11:19:17.793 INFO:tasks.rados.rados.0.trial043.stdout:812: finishing write tid 4 to trial04342147-1 2026-02-20T11:19:17.793 INFO:tasks.rados.rados.0.trial043.stdout:812: oid 1 updating version 0 to 95 2026-02-20T11:19:17.793 INFO:tasks.rados.rados.0.trial043.stdout:812: oid 1 version 95 is already newer than 95 2026-02-20T11:19:17.794 INFO:tasks.rados.rados.0.trial043.stdout:812: oid 1 version 95 is already newer than 95 2026-02-20T11:19:17.794 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 1 v 95 (ObjNum 380 snap 101 seq_num 380) dirty exists 2026-02-20T11:19:17.794 INFO:tasks.rados.rados.0.trial043.stdout:812: left oid 1 (ObjNum 380 snap 101 seq_num 380) 2026-02-20T11:19:17.811 INFO:tasks.rados.rados.0.trial043.stdout:812: done (2 left) 2026-02-20T11:19:17.811 INFO:tasks.rados.rados.0.trial043.stdout:815: done (1 left) 2026-02-20T11:19:17.811 INFO:tasks.rados.rados.0.trial043.stdout:816: read oid 22 snap -1 2026-02-20T11:19:17.811 INFO:tasks.rados.rados.0.trial043.stdout:816: expect (ObjNum 281 snap 77 seq_num 281) 2026-02-20T11:19:17.811 INFO:tasks.rados.rados.0.trial043.stdout:817: read oid 21 snap -1 2026-02-20T11:19:17.812 INFO:tasks.rados.rados.0.trial043.stdout:817: expect deleted 2026-02-20T11:19:17.812 INFO:tasks.rados.rados.0.trial043.stdout:818: delete oid 19 current snap is 101 2026-02-20T11:19:17.813 INFO:tasks.rados.rados.0.trial043.stdout:818: done (3 left) 2026-02-20T11:19:17.813 INFO:tasks.rados.rados.0.trial043.stdout:819: write (excl) oid 35 current snap is 101 2026-02-20T11:19:17.813 INFO:tasks.rados.rados.0.trial043.stdout:819: seq_num 383 ranges {437081=638017,1679074=582789} 2026-02-20T11:19:17.817 INFO:tasks.rados.rados.0.trial043.stdout:819: writing trial04342147-35 from 437081 to 1075098 tid 1 2026-02-20T11:19:17.822 INFO:tasks.rados.rados.0.trial043.stdout:819: writing trial04342147-35 from 1679074 to 2261863 tid 2 2026-02-20T11:19:17.822 INFO:tasks.rados.rados.0.trial043.stdout:820: write (excl) oid 13 current snap is 101 2026-02-20T11:19:17.823 INFO:tasks.rados.rados.0.trial043.stdout:820: seq_num 384 ranges {453888=713266,1737556=442800} 2026-02-20T11:19:17.827 INFO:tasks.rados.rados.0.trial043.stdout:820: writing trial04342147-13 from 453888 to 1167154 tid 1 2026-02-20T11:19:17.831 INFO:tasks.rados.rados.0.trial043.stdout:820: writing trial04342147-13 from 1737556 to 2180356 tid 2 2026-02-20T11:19:17.831 INFO:tasks.rados.rados.0.trial043.stdout:821: read oid 29 snap -1 2026-02-20T11:19:17.831 INFO:tasks.rados.rados.0.trial043.stdout:821: expect deleted 2026-02-20T11:19:17.831 INFO:tasks.rados.rados.0.trial043.stdout:822: read oid 33 snap -1 2026-02-20T11:19:17.831 INFO:tasks.rados.rados.0.trial043.stdout:822: expect (ObjNum 335 snap 91 seq_num 335) 2026-02-20T11:19:17.831 INFO:tasks.rados.rados.0.trial043.stdout:823: write oid 47 current snap is 101 2026-02-20T11:19:17.831 INFO:tasks.rados.rados.0.trial043.stdout:823: seq_num 385 ranges {470695=788515,1679685=586458} 2026-02-20T11:19:17.835 INFO:tasks.rados.rados.0.trial043.stdout:823: writing trial04342147-47 from 470695 to 1259210 tid 1 2026-02-20T11:19:17.839 INFO:tasks.rados.rados.0.trial043.stdout:823: writing trial04342147-47 from 1679685 to 2266143 tid 2 2026-02-20T11:19:17.839 INFO:tasks.rados.rados.0.trial043.stdout:824: write oid 43 current snap is 101 2026-02-20T11:19:17.840 INFO:tasks.rados.rados.0.trial043.stdout:824: seq_num 386 ranges {487502=463764,1738167=446469} 2026-02-20T11:19:17.843 INFO:tasks.rados.rados.0.trial043.stdout:824: writing trial04342147-43 from 487502 to 951266 tid 1 2026-02-20T11:19:17.848 INFO:tasks.rados.rados.0.trial043.stdout:824: writing trial04342147-43 from 1738167 to 2184636 tid 2 2026-02-20T11:19:17.848 INFO:tasks.rados.rados.0.trial043.stdout:825: write (excl) oid 17 current snap is 101 2026-02-20T11:19:17.848 INFO:tasks.rados.rados.0.trial043.stdout:825: seq_num 387 ranges {504309=539013,1796649=590127} 2026-02-20T11:19:17.852 INFO:tasks.rados.rados.0.trial043.stdout:825: writing trial04342147-17 from 504309 to 1043322 tid 1 2026-02-20T11:19:17.857 INFO:tasks.rados.rados.0.trial043.stdout:825: writing trial04342147-17 from 1796649 to 2386776 tid 2 2026-02-20T11:19:17.857 INFO:tasks.rados.rados.0.trial043.stdout:826: snap_create 2026-02-20T11:19:17.857 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 10 2026-02-20T11:19:17.857 INFO:tasks.rados.rados.0.trial043.stdout:817: done (9 left) 2026-02-20T11:19:17.857 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 9 2026-02-20T11:19:17.857 INFO:tasks.rados.rados.0.trial043.stdout:814: expect (ObjNum 362 snap 98 seq_num 362) 2026-02-20T11:19:17.893 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 481462 2026-02-20T11:19:17.936 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1764406 2026-02-20T11:19:17.950 INFO:tasks.rados.rados.0.trial043.stdout:819: finishing write tid 1 to trial04342147-35 2026-02-20T11:19:17.951 INFO:tasks.rados.rados.0.trial043.stdout:816: expect (ObjNum 281 snap 77 seq_num 281) 2026-02-20T11:19:18.094 INFO:tasks.rados.rados.0.trial043.stdout:819: finishing write tid 2 to trial04342147-35 2026-02-20T11:19:18.095 INFO:tasks.rados.rados.0.trial043.stdout:814: done (8 left) 2026-02-20T11:19:18.095 INFO:tasks.rados.rados.0.trial043.stdout:816: done (7 left) 2026-02-20T11:19:18.095 INFO:tasks.rados.rados.0.trial043.stdout:821: done (6 left) 2026-02-20T11:19:18.095 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 6 2026-02-20T11:19:18.095 INFO:tasks.rados.rados.0.trial043.stdout:819: finishing write tid 3 to trial04342147-35 2026-02-20T11:19:18.095 INFO:tasks.rados.rados.0.trial043.stdout:819: finishing write tid 4 to trial04342147-35 2026-02-20T11:19:18.095 INFO:tasks.rados.rados.0.trial043.stdout:819: oid 35 updating version 0 to 215 2026-02-20T11:19:18.095 INFO:tasks.rados.rados.0.trial043.stdout:819: oid 35 updating version 215 to 217 2026-02-20T11:19:18.096 INFO:tasks.rados.rados.0.trial043.stdout:819: oid 35 version 217 is already newer than 217 2026-02-20T11:19:18.096 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 35 v 217 (ObjNum 383 snap 101 seq_num 383) dirty exists 2026-02-20T11:19:18.096 INFO:tasks.rados.rados.0.trial043.stdout:819: left oid 35 (ObjNum 383 snap 101 seq_num 383) 2026-02-20T11:19:18.096 INFO:tasks.rados.rados.0.trial043.stdout:823: finishing write tid 1 to trial04342147-47 2026-02-20T11:19:18.096 INFO:tasks.rados.rados.0.trial043.stdout:820: finishing write tid 1 to trial04342147-13 2026-02-20T11:19:18.096 INFO:tasks.rados.rados.0.trial043.stdout:823: finishing write tid 2 to trial04342147-47 2026-02-20T11:19:18.097 INFO:tasks.rados.rados.0.trial043.stdout:823: finishing write tid 3 to trial04342147-47 2026-02-20T11:19:18.097 INFO:tasks.rados.rados.0.trial043.stdout:822: expect (ObjNum 335 snap 91 seq_num 335) 2026-02-20T11:19:18.100 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1562714 2026-02-20T11:19:18.194 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 49830 2026-02-20T11:19:18.247 INFO:tasks.rados.rados.0.trial043.stdout:824: finishing write tid 1 to trial04342147-43 2026-02-20T11:19:18.247 INFO:tasks.rados.rados.0.trial043.stdout:823: finishing write tid 4 to trial04342147-47 2026-02-20T11:19:18.247 INFO:tasks.rados.rados.0.trial043.stdout:823: oid 47 updating version 0 to 109 2026-02-20T11:19:18.247 INFO:tasks.rados.rados.0.trial043.stdout:823: oid 47 version 109 is already newer than 109 2026-02-20T11:19:18.247 INFO:tasks.rados.rados.0.trial043.stdout:823: oid 47 version 109 is already newer than 109 2026-02-20T11:19:18.247 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 47 v 109 (ObjNum 385 snap 101 seq_num 385) dirty exists 2026-02-20T11:19:18.247 INFO:tasks.rados.rados.0.trial043.stdout:823: left oid 47 (ObjNum 385 snap 101 seq_num 385) 2026-02-20T11:19:18.247 INFO:tasks.rados.rados.0.trial043.stdout:820: finishing write tid 2 to trial04342147-13 2026-02-20T11:19:18.247 INFO:tasks.rados.rados.0.trial043.stdout:824: finishing write tid 2 to trial04342147-43 2026-02-20T11:19:18.247 INFO:tasks.rados.rados.0.trial043.stdout:819: done (5 left) 2026-02-20T11:19:18.247 INFO:tasks.rados.rados.0.trial043.stdout:822: done (4 left) 2026-02-20T11:19:18.247 INFO:tasks.rados.rados.0.trial043.stdout:823: done (3 left) 2026-02-20T11:19:18.247 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:19:18.247 INFO:tasks.rados.rados.0.trial043.stdout:825: finishing write tid 1 to trial04342147-17 2026-02-20T11:19:18.247 INFO:tasks.rados.rados.0.trial043.stdout:820: finishing write tid 3 to trial04342147-13 2026-02-20T11:19:18.247 INFO:tasks.rados.rados.0.trial043.stdout:820: finishing write tid 4 to trial04342147-13 2026-02-20T11:19:18.247 INFO:tasks.rados.rados.0.trial043.stdout:820: oid 13 updating version 0 to 218 2026-02-20T11:19:18.247 INFO:tasks.rados.rados.0.trial043.stdout:820: oid 13 updating version 218 to 220 2026-02-20T11:19:18.247 INFO:tasks.rados.rados.0.trial043.stdout:820: oid 13 version 220 is already newer than 220 2026-02-20T11:19:18.247 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 13 v 220 (ObjNum 384 snap 101 seq_num 384) dirty exists 2026-02-20T11:19:18.247 INFO:tasks.rados.rados.0.trial043.stdout:820: left oid 13 (ObjNum 384 snap 101 seq_num 384) 2026-02-20T11:19:18.247 INFO:tasks.rados.rados.0.trial043.stdout:825: finishing write tid 2 to trial04342147-17 2026-02-20T11:19:18.247 INFO:tasks.rados.rados.0.trial043.stdout:825: finishing write tid 3 to trial04342147-17 2026-02-20T11:19:18.247 INFO:tasks.rados.rados.0.trial043.stdout:825: finishing write tid 4 to trial04342147-17 2026-02-20T11:19:18.247 INFO:tasks.rados.rados.0.trial043.stdout:825: oid 17 updating version 0 to 260 2026-02-20T11:19:18.247 INFO:tasks.rados.rados.0.trial043.stdout:825: oid 17 version 260 is already newer than 258 2026-02-20T11:19:18.247 INFO:tasks.rados.rados.0.trial043.stdout:825: oid 17 version 260 is already newer than 260 2026-02-20T11:19:18.247 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 17 v 260 (ObjNum 387 snap 101 seq_num 387) dirty exists 2026-02-20T11:19:18.247 INFO:tasks.rados.rados.0.trial043.stdout:825: left oid 17 (ObjNum 387 snap 101 seq_num 387) 2026-02-20T11:19:18.248 INFO:tasks.rados.rados.0.trial043.stdout:824: finishing write tid 3 to trial04342147-43 2026-02-20T11:19:18.248 INFO:tasks.rados.rados.0.trial043.stdout:824: finishing write tid 4 to trial04342147-43 2026-02-20T11:19:18.248 INFO:tasks.rados.rados.0.trial043.stdout:824: oid 43 updating version 0 to 299 2026-02-20T11:19:18.248 INFO:tasks.rados.rados.0.trial043.stdout:824: oid 43 version 299 is already newer than 299 2026-02-20T11:19:18.248 INFO:tasks.rados.rados.0.trial043.stdout:824: oid 43 version 299 is already newer than 299 2026-02-20T11:19:18.248 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 43 v 299 (ObjNum 386 snap 101 seq_num 386) dirty exists 2026-02-20T11:19:18.248 INFO:tasks.rados.rados.0.trial043.stdout:824: left oid 43 (ObjNum 386 snap 101 seq_num 386) 2026-02-20T11:19:18.248 INFO:tasks.rados.rados.0.trial043.stdout:820: done (2 left) 2026-02-20T11:19:18.248 INFO:tasks.rados.rados.0.trial043.stdout:824: done (1 left) 2026-02-20T11:19:18.248 INFO:tasks.rados.rados.0.trial043.stdout:825: done (0 left) 2026-02-20T11:19:18.251 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:19:18.326 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:18.325+0000 7f76559d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:18.328 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:18.328+0000 7f76559d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:18.353 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:19:18.370 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:19:18.370 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:19:18.773 INFO:tasks.rados.rados.0.trial043.stdout:826: done (0 left) 2026-02-20T11:19:18.773 INFO:tasks.rados.rados.0.trial043.stdout:827: write oid 2 current snap is 102 2026-02-20T11:19:18.773 INFO:tasks.rados.rados.0.trial043.stdout:827: seq_num 388 ranges {521116=730615,1971484=549632} 2026-02-20T11:19:18.777 INFO:tasks.rados.rados.0.trial043.stdout:827: writing trial04342147-2 from 521116 to 1251731 tid 1 2026-02-20T11:19:18.783 INFO:tasks.rados.rados.0.trial043.stdout:827: writing trial04342147-2 from 1971484 to 2521116 tid 2 2026-02-20T11:19:18.783 INFO:tasks.rados.rados.0.trial043.stdout:828: read oid 6 snap -1 2026-02-20T11:19:18.783 INFO:tasks.rados.rados.0.trial043.stdout:828: expect (ObjNum 244 snap 66 seq_num 244) 2026-02-20T11:19:18.783 INFO:tasks.rados.rados.0.trial043.stdout:829: snap_remove snap 101 2026-02-20T11:19:19.165 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:19:19.237 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:19:19.240 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:19:19.241 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:19:19.775 INFO:tasks.rados.rados.0.trial043.stdout:829: done (2 left) 2026-02-20T11:19:19.775 INFO:tasks.rados.rados.0.trial043.stdout:830: read oid 8 snap -1 2026-02-20T11:19:19.775 INFO:tasks.rados.rados.0.trial043.stdout:830: expect (ObjNum 293 snap 81 seq_num 293) 2026-02-20T11:19:19.775 INFO:tasks.rados.rados.0.trial043.stdout:831: write oid 17 current snap is 102 2026-02-20T11:19:19.775 INFO:tasks.rados.rados.0.trial043.stdout:831: seq_num 389 ranges {537923=405864,1513613=593796} 2026-02-20T11:19:19.778 INFO:tasks.rados.rados.0.trial043.stdout:831: writing trial04342147-17 from 537923 to 943787 tid 1 2026-02-20T11:19:19.782 INFO:tasks.rados.rados.0.trial043.stdout:831: writing trial04342147-17 from 1513613 to 2107409 tid 2 2026-02-20T11:19:19.783 INFO:tasks.rados.rados.0.trial043.stdout:832: read oid 50 snap -1 2026-02-20T11:19:19.783 INFO:tasks.rados.rados.0.trial043.stdout:832: expect deleted 2026-02-20T11:19:19.783 INFO:tasks.rados.rados.0.trial043.stdout:833: write oid 29 current snap is 102 2026-02-20T11:19:19.783 INFO:tasks.rados.rados.0.trial043.stdout:833: seq_num 390 ranges {554730=481113,1572095=737454} 2026-02-20T11:19:19.786 INFO:tasks.rados.rados.0.trial043.stdout:833: writing trial04342147-29 from 554730 to 1035843 tid 1 2026-02-20T11:19:19.791 INFO:tasks.rados.rados.0.trial043.stdout:833: writing trial04342147-29 from 1572095 to 2309549 tid 2 2026-02-20T11:19:19.791 INFO:tasks.rados.rados.0.trial043.stdout:834: read oid 35 snap -1 2026-02-20T11:19:19.791 INFO:tasks.rados.rados.0.trial043.stdout:834: expect (ObjNum 383 snap 101 seq_num 383) 2026-02-20T11:19:19.791 INFO:tasks.rados.rados.0.trial043.stdout:835: snap_create 2026-02-20T11:19:19.791 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 7 2026-02-20T11:19:19.792 INFO:tasks.rados.rados.0.trial043.stdout:827: finishing write tid 1 to trial04342147-2 2026-02-20T11:19:19.792 INFO:tasks.rados.rados.0.trial043.stdout:828: expect (ObjNum 244 snap 66 seq_num 244) 2026-02-20T11:19:19.845 INFO:tasks.rados.rados.0.trial043.stdout:827: finishing write tid 2 to trial04342147-2 2026-02-20T11:19:19.846 INFO:tasks.rados.rados.0.trial043.stdout:827: finishing write tid 3 to trial04342147-2 2026-02-20T11:19:19.846 INFO:tasks.rados.rados.0.trial043.stdout:828: done (6 left) 2026-02-20T11:19:19.846 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 6 2026-02-20T11:19:19.846 INFO:tasks.rados.rados.0.trial043.stdout:827: finishing write tid 4 to trial04342147-2 2026-02-20T11:19:19.846 INFO:tasks.rados.rados.0.trial043.stdout:827: oid 2 updating version 0 to 112 2026-02-20T11:19:19.846 INFO:tasks.rados.rados.0.trial043.stdout:827: oid 2 version 112 is already newer than 112 2026-02-20T11:19:19.846 INFO:tasks.rados.rados.0.trial043.stdout:827: oid 2 version 112 is already newer than 112 2026-02-20T11:19:19.846 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 2 v 112 (ObjNum 388 snap 102 seq_num 388) dirty exists 2026-02-20T11:19:19.846 INFO:tasks.rados.rados.0.trial043.stdout:827: left oid 2 (ObjNum 388 snap 102 seq_num 388) 2026-02-20T11:19:19.846 INFO:tasks.rados.rados.0.trial043.stdout:827: done (5 left) 2026-02-20T11:19:19.846 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 5 2026-02-20T11:19:19.846 INFO:tasks.rados.rados.0.trial043.stdout:834: expect (ObjNum 383 snap 101 seq_num 383) 2026-02-20T11:19:19.896 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:19.925 INFO:tasks.rados.rados.0.trial043.stdout:831: finishing write tid 1 to trial04342147-17 2026-02-20T11:19:19.925 INFO:tasks.rados.rados.0.trial043.stdout:834: done (4 left) 2026-02-20T11:19:19.925 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 4 2026-02-20T11:19:19.925 INFO:tasks.rados.rados.0.trial043.stdout:831: finishing write tid 2 to trial04342147-17 2026-02-20T11:19:19.925 INFO:tasks.rados.rados.0.trial043.stdout:831: finishing write tid 3 to trial04342147-17 2026-02-20T11:19:19.925 INFO:tasks.rados.rados.0.trial043.stdout:831: finishing write tid 4 to trial04342147-17 2026-02-20T11:19:19.925 INFO:tasks.rados.rados.0.trial043.stdout:831: oid 17 updating version 0 to 263 2026-02-20T11:19:19.925 INFO:tasks.rados.rados.0.trial043.stdout:831: oid 17 version 263 is already newer than 263 2026-02-20T11:19:19.926 INFO:tasks.rados.rados.0.trial043.stdout:831: oid 17 version 263 is already newer than 263 2026-02-20T11:19:19.926 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 17 v 263 (ObjNum 389 snap 102 seq_num 389) dirty exists 2026-02-20T11:19:19.926 INFO:tasks.rados.rados.0.trial043.stdout:831: left oid 17 (ObjNum 389 snap 102 seq_num 389) 2026-02-20T11:19:19.926 INFO:tasks.rados.rados.0.trial043.stdout:833: finishing write tid 1 to trial04342147-29 2026-02-20T11:19:19.926 INFO:tasks.rados.rados.0.trial043.stdout:833: finishing write tid 2 to trial04342147-29 2026-02-20T11:19:19.926 INFO:tasks.rados.rados.0.trial043.stdout:830: expect (ObjNum 293 snap 81 seq_num 293) 2026-02-20T11:19:19.926 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 378578 2026-02-20T11:19:19.995 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~215s 2026-02-20T11:19:20.009 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:20.052 INFO:tasks.rados.rados.0.trial043.stdout:833: finishing write tid 3 to trial04342147-29 2026-02-20T11:19:20.052 INFO:tasks.rados.rados.0.trial043.stdout:833: finishing write tid 4 to trial04342147-29 2026-02-20T11:19:20.052 INFO:tasks.rados.rados.0.trial043.stdout:833: oid 29 updating version 0 to 302 2026-02-20T11:19:20.052 INFO:tasks.rados.rados.0.trial043.stdout:833: oid 29 version 302 is already newer than 302 2026-02-20T11:19:20.052 INFO:tasks.rados.rados.0.trial043.stdout:833: oid 29 version 302 is already newer than 302 2026-02-20T11:19:20.052 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 29 v 302 (ObjNum 390 snap 102 seq_num 390) dirty exists 2026-02-20T11:19:20.052 INFO:tasks.rados.rados.0.trial043.stdout:833: left oid 29 (ObjNum 390 snap 102 seq_num 390) 2026-02-20T11:19:20.052 INFO:tasks.rados.rados.0.trial043.stdout:830: done (3 left) 2026-02-20T11:19:20.052 INFO:tasks.rados.rados.0.trial043.stdout:831: done (2 left) 2026-02-20T11:19:20.052 INFO:tasks.rados.rados.0.trial043.stdout:832: done (1 left) 2026-02-20T11:19:20.053 INFO:tasks.rados.rados.0.trial043.stdout:833: done (0 left) 2026-02-20T11:19:20.372 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:19:20.447 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:20.447+0000 7f1d2501a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:20.450 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:20.450+0000 7f1d2501a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:20.475 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:19:20.492 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:19:20.492 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:19:20.776 INFO:tasks.rados.rados.0.trial043.stdout:835: done (0 left) 2026-02-20T11:19:20.777 INFO:tasks.rados.rados.0.trial043.stdout:836: delete oid 44 current snap is 103 2026-02-20T11:19:20.813 INFO:tasks.rados.rados.0.trial043.stdout:836: done (0 left) 2026-02-20T11:19:20.814 INFO:tasks.rados.rados.0.trial043.stdout:837: read oid 40 snap -1 2026-02-20T11:19:20.814 INFO:tasks.rados.rados.0.trial043.stdout:837: expect deleted 2026-02-20T11:19:20.814 INFO:tasks.rados.rados.0.trial043.stdout:838: write oid 22 current snap is 103 2026-02-20T11:19:20.814 INFO:tasks.rados.rados.0.trial043.stdout:838: seq_num 392 ranges {588344=631611,1689059=741123} 2026-02-20T11:19:20.818 INFO:tasks.rados.rados.0.trial043.stdout:838: writing trial04342147-22 from 588344 to 1219955 tid 1 2026-02-20T11:19:20.823 INFO:tasks.rados.rados.0.trial043.stdout:838: writing trial04342147-22 from 1689059 to 2430182 tid 2 2026-02-20T11:19:20.823 INFO:tasks.rados.rados.0.trial043.stdout:839: snap_remove snap 102 2026-02-20T11:19:21.779 INFO:tasks.rados.rados.0.trial043.stdout:839: done (2 left) 2026-02-20T11:19:21.779 INFO:tasks.rados.rados.0.trial043.stdout:840: read oid 17 snap -1 2026-02-20T11:19:21.779 INFO:tasks.rados.rados.0.trial043.stdout:840: expect (ObjNum 389 snap 102 seq_num 389) 2026-02-20T11:19:21.779 INFO:tasks.rados.rados.0.trial043.stdout:841: write oid 33 current snap is 103 2026-02-20T11:19:21.779 INFO:tasks.rados.rados.0.trial043.stdout:841: seq_num 393 ranges {605151=706860,2031188=573963} 2026-02-20T11:19:21.783 INFO:tasks.rados.rados.0.trial043.stdout:841: writing trial04342147-33 from 605151 to 1312011 tid 1 2026-02-20T11:19:21.790 INFO:tasks.rados.rados.0.trial043.stdout:841: writing trial04342147-33 from 2031188 to 2605151 tid 2 2026-02-20T11:19:21.790 INFO:tasks.rados.rados.0.trial043.stdout:842: read oid 3 snap -1 2026-02-20T11:19:21.790 INFO:tasks.rados.rados.0.trial043.stdout:842: expect (ObjNum 375 snap 100 seq_num 375) 2026-02-20T11:19:21.790 INFO:tasks.rados.rados.0.trial043.stdout:843: read oid 21 snap -1 2026-02-20T11:19:21.791 INFO:tasks.rados.rados.0.trial043.stdout:843: expect deleted 2026-02-20T11:19:21.791 INFO:tasks.rados.rados.0.trial043.stdout:844: snap_create 2026-02-20T11:19:21.791 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 6 2026-02-20T11:19:21.791 INFO:tasks.rados.rados.0.trial043.stdout:838: finishing write tid 1 to trial04342147-22 2026-02-20T11:19:21.791 INFO:tasks.rados.rados.0.trial043.stdout:838: finishing write tid 2 to trial04342147-22 2026-02-20T11:19:21.791 INFO:tasks.rados.rados.0.trial043.stdout:838: finishing write tid 3 to trial04342147-22 2026-02-20T11:19:21.791 INFO:tasks.rados.rados.0.trial043.stdout:838: finishing write tid 4 to trial04342147-22 2026-02-20T11:19:21.791 INFO:tasks.rados.rados.0.trial043.stdout:838: oid 22 updating version 0 to 90 2026-02-20T11:19:21.791 INFO:tasks.rados.rados.0.trial043.stdout:838: oid 22 version 90 is already newer than 90 2026-02-20T11:19:21.791 INFO:tasks.rados.rados.0.trial043.stdout:838: oid 22 version 90 is already newer than 90 2026-02-20T11:19:21.791 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 22 v 90 (ObjNum 392 snap 103 seq_num 392) dirty exists 2026-02-20T11:19:21.791 INFO:tasks.rados.rados.0.trial043.stdout:838: left oid 22 (ObjNum 392 snap 103 seq_num 392) 2026-02-20T11:19:21.791 INFO:tasks.rados.rados.0.trial043.stdout:837: done (5 left) 2026-02-20T11:19:21.792 INFO:tasks.rados.rados.0.trial043.stdout:838: done (4 left) 2026-02-20T11:19:21.792 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 4 2026-02-20T11:19:21.799 INFO:tasks.rados.rados.0.trial043.stdout:843: done (3 left) 2026-02-20T11:19:21.799 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:19:21.819 INFO:tasks.rados.rados.0.trial043.stdout:842: expect (ObjNum 375 snap 100 seq_num 375) 2026-02-20T11:19:21.826 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1206721 2026-02-20T11:19:21.893 INFO:tasks.rados.rados.0.trial043.stdout:841: finishing write tid 1 to trial04342147-33 2026-02-20T11:19:21.894 INFO:tasks.rados.rados.0.trial043.stdout:841: finishing write tid 2 to trial04342147-33 2026-02-20T11:19:21.894 INFO:tasks.rados.rados.0.trial043.stdout:840: expect (ObjNum 389 snap 102 seq_num 389) 2026-02-20T11:19:21.899 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1060818 2026-02-20T11:19:21.980 INFO:tasks.rados.rados.0.trial043.stdout:841: finishing write tid 3 to trial04342147-33 2026-02-20T11:19:21.980 INFO:tasks.rados.rados.0.trial043.stdout:841: finishing write tid 4 to trial04342147-33 2026-02-20T11:19:21.981 INFO:tasks.rados.rados.0.trial043.stdout:841: oid 33 updating version 0 to 305 2026-02-20T11:19:21.981 INFO:tasks.rados.rados.0.trial043.stdout:841: oid 33 version 305 is already newer than 305 2026-02-20T11:19:21.981 INFO:tasks.rados.rados.0.trial043.stdout:841: oid 33 version 305 is already newer than 305 2026-02-20T11:19:21.981 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 33 v 305 (ObjNum 393 snap 103 seq_num 393) dirty exists 2026-02-20T11:19:21.981 INFO:tasks.rados.rados.0.trial043.stdout:841: left oid 33 (ObjNum 393 snap 103 seq_num 393) 2026-02-20T11:19:21.981 INFO:tasks.rados.rados.0.trial043.stdout:840: done (2 left) 2026-02-20T11:19:21.982 INFO:tasks.rados.rados.0.trial043.stdout:841: done (1 left) 2026-02-20T11:19:21.982 INFO:tasks.rados.rados.0.trial043.stdout:842: done (0 left) 2026-02-20T11:19:22.493 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:19:22.569 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:22.568+0000 7f98d7acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:22.572 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:22.572+0000 7f98d7acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:22.597 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:19:22.614 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:19:22.615 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:19:22.784 INFO:tasks.rados.rados.0.trial043.stdout:844: done (0 left) 2026-02-20T11:19:22.784 INFO:tasks.rados.rados.0.trial043.stdout:845: snap_create 2026-02-20T11:19:23.786 INFO:tasks.rados.rados.0.trial043.stdout:845: done (0 left) 2026-02-20T11:19:23.786 INFO:tasks.rados.rados.0.trial043.stdout:846: snap_remove snap 104 2026-02-20T11:19:24.242 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:19:24.315 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:19:24.318 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:19:24.318 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:19:24.616 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:19:24.693 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:24.691+0000 7f0e12cc2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:24.696 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:24.695+0000 7f0e12cc2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:24.720 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:19:24.737 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:19:24.737 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:19:24.789 INFO:tasks.rados.rados.0.trial043.stdout:846: done (0 left) 2026-02-20T11:19:24.790 INFO:tasks.rados.rados.0.trial043.stdout:847: read oid 44 snap -1 2026-02-20T11:19:24.790 INFO:tasks.rados.rados.0.trial043.stdout:847: expect deleted 2026-02-20T11:19:24.790 INFO:tasks.rados.rados.0.trial043.stdout:848: read oid 31 snap -1 2026-02-20T11:19:24.790 INFO:tasks.rados.rados.0.trial043.stdout:848: expect deleted 2026-02-20T11:19:24.790 INFO:tasks.rados.rados.0.trial043.stdout:849: read oid 2 snap -1 2026-02-20T11:19:24.790 INFO:tasks.rados.rados.0.trial043.stdout:849: expect (ObjNum 388 snap 102 seq_num 388) 2026-02-20T11:19:24.790 INFO:tasks.rados.rados.0.trial043.stdout:850: write (excl) oid 15 current snap is 105 2026-02-20T11:19:24.790 INFO:tasks.rados.rados.0.trial043.stdout:850: seq_num 394 ranges {621958=782109,2089670=532288} 2026-02-20T11:19:24.794 INFO:tasks.rados.rados.0.trial043.stdout:850: writing trial04342147-15 from 621958 to 1404067 tid 1 2026-02-20T11:19:24.800 INFO:tasks.rados.rados.0.trial043.stdout:850: writing trial04342147-15 from 2089670 to 2621958 tid 2 2026-02-20T11:19:24.800 INFO:tasks.rados.rados.0.trial043.stdout:851: read oid 40 snap -1 2026-02-20T11:19:24.800 INFO:tasks.rados.rados.0.trial043.stdout:851: expect deleted 2026-02-20T11:19:24.800 INFO:tasks.rados.rados.0.trial043.stdout:852: delete oid 7 current snap is 105 2026-02-20T11:19:24.823 INFO:tasks.rados.rados.0.trial043.stdout:849: expect (ObjNum 388 snap 102 seq_num 388) 2026-02-20T11:19:24.825 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1412765 2026-02-20T11:19:24.875 INFO:tasks.rados.rados.0.trial043.stdout:850: finishing write tid 1 to trial04342147-15 2026-02-20T11:19:24.875 INFO:tasks.rados.rados.0.trial043.stdout:850: finishing write tid 2 to trial04342147-15 2026-02-20T11:19:24.875 INFO:tasks.rados.rados.0.trial043.stdout:850: finishing write tid 3 to trial04342147-15 2026-02-20T11:19:24.875 INFO:tasks.rados.rados.0.trial043.stdout:850: finishing write tid 4 to trial04342147-15 2026-02-20T11:19:24.875 INFO:tasks.rados.rados.0.trial043.stdout:850: oid 15 updating version 0 to 66 2026-02-20T11:19:24.875 INFO:tasks.rados.rados.0.trial043.stdout:850: oid 15 updating version 66 to 68 2026-02-20T11:19:24.875 INFO:tasks.rados.rados.0.trial043.stdout:850: oid 15 version 68 is already newer than 68 2026-02-20T11:19:24.875 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 15 v 68 (ObjNum 394 snap 105 seq_num 394) dirty exists 2026-02-20T11:19:24.876 INFO:tasks.rados.rados.0.trial043.stdout:850: left oid 15 (ObjNum 394 snap 105 seq_num 394) 2026-02-20T11:19:24.876 INFO:tasks.rados.rados.0.trial043.stdout:847: done (5 left) 2026-02-20T11:19:24.876 INFO:tasks.rados.rados.0.trial043.stdout:848: done (4 left) 2026-02-20T11:19:24.876 INFO:tasks.rados.rados.0.trial043.stdout:849: done (3 left) 2026-02-20T11:19:24.876 INFO:tasks.rados.rados.0.trial043.stdout:850: done (2 left) 2026-02-20T11:19:24.876 INFO:tasks.rados.rados.0.trial043.stdout:851: done (1 left) 2026-02-20T11:19:24.876 INFO:tasks.rados.rados.0.trial043.stdout:852: done (0 left) 2026-02-20T11:19:24.876 INFO:tasks.rados.rados.0.trial043.stdout:853: write (excl) oid 22 current snap is 105 2026-02-20T11:19:24.876 INFO:tasks.rados.rados.0.trial043.stdout:853: seq_num 396 ranges {655572=648960,1922987=732585} 2026-02-20T11:19:24.879 INFO:tasks.rados.rados.0.trial043.stdout:853: writing trial04342147-22 from 655572 to 1304532 tid 1 2026-02-20T11:19:24.886 INFO:tasks.rados.rados.0.trial043.stdout:853: writing trial04342147-22 from 1922987 to 2655572 tid 2 2026-02-20T11:19:24.887 INFO:tasks.rados.rados.0.trial043.stdout:854: write oid 8 current snap is 105 2026-02-20T11:19:24.887 INFO:tasks.rados.rados.0.trial043.stdout:854: seq_num 397 ranges {672379=724209,1865116=608472} 2026-02-20T11:19:24.891 INFO:tasks.rados.rados.0.trial043.stdout:854: writing trial04342147-8 from 672379 to 1396588 tid 1 2026-02-20T11:19:24.896 INFO:tasks.rados.rados.0.trial043.stdout:854: writing trial04342147-8 from 1865116 to 2473588 tid 2 2026-02-20T11:19:24.896 INFO:tasks.rados.rados.0.trial043.stdout:855: write (excl) oid 29 current snap is 105 2026-02-20T11:19:24.897 INFO:tasks.rados.rados.0.trial043.stdout:855: seq_num 398 ranges {689186=799458,1923598=752130} 2026-02-20T11:19:24.902 INFO:tasks.rados.rados.0.trial043.stdout:855: writing trial04342147-29 from 689186 to 1488644 tid 1 2026-02-20T11:19:24.906 INFO:tasks.rados.rados.0.trial043.stdout:855: writing trial04342147-29 from 1923598 to 2675728 tid 2 2026-02-20T11:19:24.907 INFO:tasks.rados.rados.0.trial043.stdout:856: write (excl) oid 47 current snap is 105 2026-02-20T11:19:24.907 INFO:tasks.rados.rados.0.trial043.stdout:856: seq_num 399 ranges {705993=474707,1582080=612141} 2026-02-20T11:19:24.911 INFO:tasks.rados.rados.0.trial043.stdout:856: writing trial04342147-47 from 705993 to 1180700 tid 1 2026-02-20T11:19:24.915 INFO:tasks.rados.rados.0.trial043.stdout:856: writing trial04342147-47 from 1582080 to 2194221 tid 2 2026-02-20T11:19:24.915 INFO:tasks.rados.rados.0.trial043.stdout:857: write oid 35 current snap is 105 2026-02-20T11:19:24.915 INFO:tasks.rados.rados.0.trial043.stdout:853: finishing write tid 1 to trial04342147-22 2026-02-20T11:19:24.915 INFO:tasks.rados.rados.0.trial043.stdout:857: seq_num 400 ranges {722800=549956,2040562=682238} 2026-02-20T11:19:24.920 INFO:tasks.rados.rados.0.trial043.stdout:857: writing trial04342147-35 from 722800 to 1272756 tid 1 2026-02-20T11:19:24.928 INFO:tasks.rados.rados.0.trial043.stdout:857: writing trial04342147-35 from 2040562 to 2722800 tid 2 2026-02-20T11:19:24.928 INFO:tasks.rados.rados.0.trial043.stdout:858: write oid 6 current snap is 105 2026-02-20T11:19:24.928 INFO:tasks.rados.rados.0.trial043.stdout:858: seq_num 401 ranges {739607=625205,1982691=499457} 2026-02-20T11:19:24.932 INFO:tasks.rados.rados.0.trial043.stdout:858: writing trial04342147-6 from 739607 to 1364812 tid 1 2026-02-20T11:19:24.937 INFO:tasks.rados.rados.0.trial043.stdout:858: writing trial04342147-6 from 1982691 to 2482148 tid 2 2026-02-20T11:19:24.937 INFO:tasks.rados.rados.0.trial043.stdout:859: read oid 21 snap -1 2026-02-20T11:19:24.937 INFO:tasks.rados.rados.0.trial043.stdout:859: expect deleted 2026-02-20T11:19:24.938 INFO:tasks.rados.rados.0.trial043.stdout:860: snap_remove snap 103 2026-02-20T11:19:25.499 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~220s 2026-02-20T11:19:25.799 INFO:tasks.rados.rados.0.trial043.stdout:860: done (7 left) 2026-02-20T11:19:25.799 INFO:tasks.rados.rados.0.trial043.stdout:861: delete oid 40 current snap is 105 2026-02-20T11:19:25.799 INFO:tasks.rados.rados.0.trial043.stdout:853: finishing write tid 2 to trial04342147-22 2026-02-20T11:19:25.799 INFO:tasks.rados.rados.0.trial043.stdout:853: finishing write tid 3 to trial04342147-22 2026-02-20T11:19:25.799 INFO:tasks.rados.rados.0.trial043.stdout:853: finishing write tid 4 to trial04342147-22 2026-02-20T11:19:25.799 INFO:tasks.rados.rados.0.trial043.stdout:853: oid 22 updating version 0 to 92 2026-02-20T11:19:25.799 INFO:tasks.rados.rados.0.trial043.stdout:853: oid 22 updating version 92 to 94 2026-02-20T11:19:25.799 INFO:tasks.rados.rados.0.trial043.stdout:853: oid 22 version 94 is already newer than 94 2026-02-20T11:19:25.799 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 22 v 94 (ObjNum 396 snap 105 seq_num 396) dirty exists 2026-02-20T11:19:25.800 INFO:tasks.rados.rados.0.trial043.stdout:853: left oid 22 (ObjNum 396 snap 105 seq_num 396) 2026-02-20T11:19:25.800 INFO:tasks.rados.rados.0.trial043.stdout:857: finishing write tid 1 to trial04342147-35 2026-02-20T11:19:25.800 INFO:tasks.rados.rados.0.trial043.stdout:857: finishing write tid 2 to trial04342147-35 2026-02-20T11:19:25.800 INFO:tasks.rados.rados.0.trial043.stdout:856: finishing write tid 1 to trial04342147-47 2026-02-20T11:19:25.800 INFO:tasks.rados.rados.0.trial043.stdout:857: finishing write tid 3 to trial04342147-35 2026-02-20T11:19:25.800 INFO:tasks.rados.rados.0.trial043.stdout:855: finishing write tid 1 to trial04342147-29 2026-02-20T11:19:25.800 INFO:tasks.rados.rados.0.trial043.stdout:857: finishing write tid 4 to trial04342147-35 2026-02-20T11:19:25.800 INFO:tasks.rados.rados.0.trial043.stdout:857: oid 35 updating version 0 to 224 2026-02-20T11:19:25.800 INFO:tasks.rados.rados.0.trial043.stdout:857: oid 35 version 224 is already newer than 224 2026-02-20T11:19:25.800 INFO:tasks.rados.rados.0.trial043.stdout:857: oid 35 version 224 is already newer than 224 2026-02-20T11:19:25.800 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 35 v 224 (ObjNum 400 snap 105 seq_num 400) dirty exists 2026-02-20T11:19:25.800 INFO:tasks.rados.rados.0.trial043.stdout:857: left oid 35 (ObjNum 400 snap 105 seq_num 400) 2026-02-20T11:19:25.801 INFO:tasks.rados.rados.0.trial043.stdout:858: finishing write tid 1 to trial04342147-6 2026-02-20T11:19:25.801 INFO:tasks.rados.rados.0.trial043.stdout:855: finishing write tid 2 to trial04342147-29 2026-02-20T11:19:25.801 INFO:tasks.rados.rados.0.trial043.stdout:856: finishing write tid 2 to trial04342147-47 2026-02-20T11:19:25.801 INFO:tasks.rados.rados.0.trial043.stdout:855: finishing write tid 3 to trial04342147-29 2026-02-20T11:19:25.801 INFO:tasks.rados.rados.0.trial043.stdout:856: finishing write tid 3 to trial04342147-47 2026-02-20T11:19:25.801 INFO:tasks.rados.rados.0.trial043.stdout:855: finishing write tid 4 to trial04342147-29 2026-02-20T11:19:25.801 INFO:tasks.rados.rados.0.trial043.stdout:855: oid 29 updating version 0 to 309 2026-02-20T11:19:25.801 INFO:tasks.rados.rados.0.trial043.stdout:855: oid 29 version 309 is already newer than 307 2026-02-20T11:19:25.801 INFO:tasks.rados.rados.0.trial043.stdout:855: oid 29 version 309 is already newer than 309 2026-02-20T11:19:25.801 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 29 v 309 (ObjNum 398 snap 105 seq_num 398) dirty exists 2026-02-20T11:19:25.801 INFO:tasks.rados.rados.0.trial043.stdout:855: left oid 29 (ObjNum 398 snap 105 seq_num 398) 2026-02-20T11:19:25.801 INFO:tasks.rados.rados.0.trial043.stdout:858: finishing write tid 2 to trial04342147-6 2026-02-20T11:19:25.802 INFO:tasks.rados.rados.0.trial043.stdout:856: finishing write tid 4 to trial04342147-47 2026-02-20T11:19:25.802 INFO:tasks.rados.rados.0.trial043.stdout:856: oid 47 updating version 0 to 118 2026-02-20T11:19:25.802 INFO:tasks.rados.rados.0.trial043.stdout:856: oid 47 version 118 is already newer than 118 2026-02-20T11:19:25.802 INFO:tasks.rados.rados.0.trial043.stdout:856: oid 47 version 118 is already newer than 116 2026-02-20T11:19:25.802 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 47 v 118 (ObjNum 399 snap 105 seq_num 399) dirty exists 2026-02-20T11:19:25.802 INFO:tasks.rados.rados.0.trial043.stdout:856: left oid 47 (ObjNum 399 snap 105 seq_num 399) 2026-02-20T11:19:25.802 INFO:tasks.rados.rados.0.trial043.stdout:858: finishing write tid 3 to trial04342147-6 2026-02-20T11:19:25.802 INFO:tasks.rados.rados.0.trial043.stdout:858: finishing write tid 4 to trial04342147-6 2026-02-20T11:19:25.802 INFO:tasks.rados.rados.0.trial043.stdout:858: oid 6 updating version 0 to 33 2026-02-20T11:19:25.802 INFO:tasks.rados.rados.0.trial043.stdout:858: oid 6 version 33 is already newer than 33 2026-02-20T11:19:25.802 INFO:tasks.rados.rados.0.trial043.stdout:858: oid 6 version 33 is already newer than 33 2026-02-20T11:19:25.802 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 6 v 33 (ObjNum 401 snap 105 seq_num 401) dirty exists 2026-02-20T11:19:25.802 INFO:tasks.rados.rados.0.trial043.stdout:858: left oid 6 (ObjNum 401 snap 105 seq_num 401) 2026-02-20T11:19:25.803 INFO:tasks.rados.rados.0.trial043.stdout:854: finishing write tid 1 to trial04342147-8 2026-02-20T11:19:25.803 INFO:tasks.rados.rados.0.trial043.stdout:854: finishing write tid 2 to trial04342147-8 2026-02-20T11:19:25.803 INFO:tasks.rados.rados.0.trial043.stdout:854: finishing write tid 3 to trial04342147-8 2026-02-20T11:19:25.803 INFO:tasks.rados.rados.0.trial043.stdout:854: finishing write tid 4 to trial04342147-8 2026-02-20T11:19:25.803 INFO:tasks.rados.rados.0.trial043.stdout:854: oid 8 updating version 0 to 98 2026-02-20T11:19:25.803 INFO:tasks.rados.rados.0.trial043.stdout:854: oid 8 version 98 is already newer than 98 2026-02-20T11:19:25.803 INFO:tasks.rados.rados.0.trial043.stdout:854: oid 8 version 98 is already newer than 98 2026-02-20T11:19:25.803 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 8 v 98 (ObjNum 397 snap 105 seq_num 397) dirty exists 2026-02-20T11:19:25.803 INFO:tasks.rados.rados.0.trial043.stdout:854: left oid 8 (ObjNum 397 snap 105 seq_num 397) 2026-02-20T11:19:25.821 INFO:tasks.rados.rados.0.trial043.stdout:853: done (7 left) 2026-02-20T11:19:25.821 INFO:tasks.rados.rados.0.trial043.stdout:854: done (6 left) 2026-02-20T11:19:25.821 INFO:tasks.rados.rados.0.trial043.stdout:855: done (5 left) 2026-02-20T11:19:25.821 INFO:tasks.rados.rados.0.trial043.stdout:856: done (4 left) 2026-02-20T11:19:25.821 INFO:tasks.rados.rados.0.trial043.stdout:857: done (3 left) 2026-02-20T11:19:25.821 INFO:tasks.rados.rados.0.trial043.stdout:858: done (2 left) 2026-02-20T11:19:25.821 INFO:tasks.rados.rados.0.trial043.stdout:859: done (1 left) 2026-02-20T11:19:25.822 INFO:tasks.rados.rados.0.trial043.stdout:861: done (0 left) 2026-02-20T11:19:25.822 INFO:tasks.rados.rados.0.trial043.stdout:862: write oid 14 current snap is 105 2026-02-20T11:19:25.822 INFO:tasks.rados.rados.0.trial043.stdout:862: seq_num 403 ranges {773221=492056,1816008=503126} 2026-02-20T11:19:25.825 INFO:tasks.rados.rados.0.trial043.stdout:862: writing trial04342147-14 from 773221 to 1265277 tid 1 2026-02-20T11:19:25.830 INFO:tasks.rados.rados.0.trial043.stdout:862: writing trial04342147-14 from 1816008 to 2319134 tid 2 2026-02-20T11:19:25.830 INFO:tasks.rados.rados.0.trial043.stdout:863: read oid 20 snap -1 2026-02-20T11:19:25.830 INFO:tasks.rados.rados.0.trial043.stdout:863: expect (ObjNum 128 snap 17 seq_num 128) 2026-02-20T11:19:25.830 INFO:tasks.rados.rados.0.trial043.stdout:864: read oid 10 snap -1 2026-02-20T11:19:25.830 INFO:tasks.rados.rados.0.trial043.stdout:864: expect (ObjNum 321 snap 88 seq_num 321) 2026-02-20T11:19:25.830 INFO:tasks.rados.rados.0.trial043.stdout:865: snap_create 2026-02-20T11:19:25.830 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:19:25.853 INFO:tasks.rados.rados.0.trial043.stdout:862: finishing write tid 1 to trial04342147-14 2026-02-20T11:19:25.855 INFO:tasks.rados.rados.0.trial043.stdout:864: expect (ObjNum 321 snap 88 seq_num 321) 2026-02-20T11:19:25.859 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 513672 2026-02-20T11:19:25.906 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:26.003 INFO:tasks.rados.rados.0.trial043.stdout:862: finishing write tid 2 to trial04342147-14 2026-02-20T11:19:26.003 INFO:tasks.rados.rados.0.trial043.stdout:863: expect (ObjNum 128 snap 17 seq_num 128) 2026-02-20T11:19:26.004 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:26.073 INFO:tasks.rados.rados.0.trial043.stdout:862: finishing write tid 3 to trial04342147-14 2026-02-20T11:19:26.073 INFO:tasks.rados.rados.0.trial043.stdout:862: finishing write tid 4 to trial04342147-14 2026-02-20T11:19:26.073 INFO:tasks.rados.rados.0.trial043.stdout:862: oid 14 updating version 0 to 266 2026-02-20T11:19:26.073 INFO:tasks.rados.rados.0.trial043.stdout:862: oid 14 version 266 is already newer than 266 2026-02-20T11:19:26.073 INFO:tasks.rados.rados.0.trial043.stdout:862: oid 14 version 266 is already newer than 266 2026-02-20T11:19:26.073 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 14 v 266 (ObjNum 403 snap 105 seq_num 403) dirty exists 2026-02-20T11:19:26.073 INFO:tasks.rados.rados.0.trial043.stdout:862: left oid 14 (ObjNum 403 snap 105 seq_num 403) 2026-02-20T11:19:26.073 INFO:tasks.rados.rados.0.trial043.stdout:862: done (2 left) 2026-02-20T11:19:26.074 INFO:tasks.rados.rados.0.trial043.stdout:863: done (1 left) 2026-02-20T11:19:26.074 INFO:tasks.rados.rados.0.trial043.stdout:864: done (0 left) 2026-02-20T11:19:26.738 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:19:26.801 INFO:tasks.rados.rados.0.trial043.stdout:865: done (0 left) 2026-02-20T11:19:26.801 INFO:tasks.rados.rados.0.trial043.stdout:866: snap_create 2026-02-20T11:19:26.816 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:26.814+0000 7ff761acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:26.818 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:26.818+0000 7ff761acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:26.845 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:19:26.861 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:19:26.861 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:19:27.800 INFO:tasks.rados.rados.0.trial043.stdout:866: done (0 left) 2026-02-20T11:19:27.800 INFO:tasks.rados.rados.0.trial043.stdout:867: write oid 44 current snap is 107 2026-02-20T11:19:27.800 INFO:tasks.rados.rados.0.trial043.stdout:867: seq_num 404 ranges {790028=567305,1874490=763137} 2026-02-20T11:19:27.805 INFO:tasks.rados.rados.0.trial043.stdout:867: writing trial04342147-44 from 790028 to 1357333 tid 1 2026-02-20T11:19:27.811 INFO:tasks.rados.rados.0.trial043.stdout:867: writing trial04342147-44 from 1874490 to 2637627 tid 2 2026-02-20T11:19:27.811 INFO:tasks.rados.rados.0.trial043.stdout:868: read oid 18 snap -1 2026-02-20T11:19:27.811 INFO:tasks.rados.rados.0.trial043.stdout:868: expect (ObjNum 372 snap 100 seq_num 372) 2026-02-20T11:19:27.811 INFO:tasks.rados.rados.0.trial043.stdout:869: read oid 19 snap -1 2026-02-20T11:19:27.811 INFO:tasks.rados.rados.0.trial043.stdout:869: expect deleted 2026-02-20T11:19:27.811 INFO:tasks.rados.rados.0.trial043.stdout:870: write oid 31 current snap is 107 2026-02-20T11:19:27.811 INFO:tasks.rados.rados.0.trial043.stdout:870: seq_num 405 ranges {406835=642554,1532972=506795,2772312=34523} 2026-02-20T11:19:27.815 INFO:tasks.rados.rados.0.trial043.stdout:870: writing trial04342147-31 from 406835 to 1049389 tid 1 2026-02-20T11:19:27.819 INFO:tasks.rados.rados.0.trial043.stdout:870: writing trial04342147-31 from 1532972 to 2039767 tid 2 2026-02-20T11:19:27.822 INFO:tasks.rados.rados.0.trial043.stdout:870: writing trial04342147-31 from 2772312 to 2806835 tid 3 2026-02-20T11:19:27.822 INFO:tasks.rados.rados.0.trial043.stdout:871: snap_remove snap 106 2026-02-20T11:19:28.775 INFO:tasks.rados.rados.0.trial043.stdout:871: done (4 left) 2026-02-20T11:19:28.775 INFO:tasks.rados.rados.0.trial043.stdout:872: write oid 37 current snap is 107 2026-02-20T11:19:28.775 INFO:tasks.rados.rados.0.trial043.stdout:872: seq_num 406 ranges {423642=717803,1875101=766806} 2026-02-20T11:19:28.779 INFO:tasks.rados.rados.0.trial043.stdout:872: writing trial04342147-37 from 423642 to 1141445 tid 1 2026-02-20T11:19:28.785 INFO:tasks.rados.rados.0.trial043.stdout:872: writing trial04342147-37 from 1875101 to 2641907 tid 2 2026-02-20T11:19:28.785 INFO:tasks.rados.rados.0.trial043.stdout:873: delete oid 17 current snap is 107 2026-02-20T11:19:28.786 INFO:tasks.rados.rados.0.trial043.stdout:870: finishing write tid 1 to trial04342147-31 2026-02-20T11:19:28.786 INFO:tasks.rados.rados.0.trial043.stdout:867: finishing write tid 1 to trial04342147-44 2026-02-20T11:19:28.786 INFO:tasks.rados.rados.0.trial043.stdout:870: finishing write tid 2 to trial04342147-31 2026-02-20T11:19:28.786 INFO:tasks.rados.rados.0.trial043.stdout:868: expect (ObjNum 372 snap 100 seq_num 372) 2026-02-20T11:19:28.836 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1571046 2026-02-20T11:19:28.846 INFO:tasks.rados.rados.0.trial043.stdout:870: finishing write tid 3 to trial04342147-31 2026-02-20T11:19:28.846 INFO:tasks.rados.rados.0.trial043.stdout:867: finishing write tid 2 to trial04342147-44 2026-02-20T11:19:28.846 INFO:tasks.rados.rados.0.trial043.stdout:868: done (5 left) 2026-02-20T11:19:28.846 INFO:tasks.rados.rados.0.trial043.stdout:869: done (4 left) 2026-02-20T11:19:28.846 INFO:tasks.rados.rados.0.trial043.stdout:873: done (3 left) 2026-02-20T11:19:28.846 INFO:tasks.rados.rados.0.trial043.stdout:874: read oid 3 snap -1 2026-02-20T11:19:28.846 INFO:tasks.rados.rados.0.trial043.stdout:874: expect (ObjNum 375 snap 100 seq_num 375) 2026-02-20T11:19:28.847 INFO:tasks.rados.rados.0.trial043.stdout:875: write oid 18 current snap is 107 2026-02-20T11:19:28.847 INFO:tasks.rados.rados.0.trial043.stdout:870: finishing write tid 4 to trial04342147-31 2026-02-20T11:19:28.847 INFO:tasks.rados.rados.0.trial043.stdout:870: finishing write tid 5 to trial04342147-31 2026-02-20T11:19:28.847 INFO:tasks.rados.rados.0.trial043.stdout:870: oid 31 updating version 0 to 72 2026-02-20T11:19:28.847 INFO:tasks.rados.rados.0.trial043.stdout:870: oid 31 version 72 is already newer than 72 2026-02-20T11:19:28.847 INFO:tasks.rados.rados.0.trial043.stdout:870: oid 31 version 72 is already newer than 72 2026-02-20T11:19:28.847 INFO:tasks.rados.rados.0.trial043.stdout:870: oid 31 version 72 is already newer than 72 2026-02-20T11:19:28.847 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 31 v 72 (ObjNum 405 snap 107 seq_num 405) dirty exists 2026-02-20T11:19:28.847 INFO:tasks.rados.rados.0.trial043.stdout:870: left oid 31 (ObjNum 405 snap 107 seq_num 405) 2026-02-20T11:19:28.847 INFO:tasks.rados.rados.0.trial043.stdout:867: finishing write tid 3 to trial04342147-44 2026-02-20T11:19:28.847 INFO:tasks.rados.rados.0.trial043.stdout:867: finishing write tid 4 to trial04342147-44 2026-02-20T11:19:28.847 INFO:tasks.rados.rados.0.trial043.stdout:867: oid 44 updating version 0 to 121 2026-02-20T11:19:28.847 INFO:tasks.rados.rados.0.trial043.stdout:867: oid 44 version 121 is already newer than 121 2026-02-20T11:19:28.847 INFO:tasks.rados.rados.0.trial043.stdout:867: oid 44 version 121 is already newer than 121 2026-02-20T11:19:28.847 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 44 v 121 (ObjNum 404 snap 107 seq_num 404) dirty exists 2026-02-20T11:19:28.847 INFO:tasks.rados.rados.0.trial043.stdout:867: left oid 44 (ObjNum 404 snap 107 seq_num 404) 2026-02-20T11:19:28.847 INFO:tasks.rados.rados.0.trial043.stdout:875: seq_num 408 ranges {457256=468301,1592065=770475,2854581=2675} 2026-02-20T11:19:28.849 INFO:tasks.rados.rados.0.trial043.stdout:875: writing trial04342147-18 from 457256 to 925557 tid 1 2026-02-20T11:19:28.855 INFO:tasks.rados.rados.0.trial043.stdout:875: writing trial04342147-18 from 1592065 to 2362540 tid 2 2026-02-20T11:19:28.857 INFO:tasks.rados.rados.0.trial043.stdout:875: writing trial04342147-18 from 2854581 to 2857256 tid 3 2026-02-20T11:19:28.857 INFO:tasks.rados.rados.0.trial043.stdout:867: done (4 left) 2026-02-20T11:19:28.857 INFO:tasks.rados.rados.0.trial043.stdout:870: done (3 left) 2026-02-20T11:19:28.857 INFO:tasks.rados.rados.0.trial043.stdout:876: snap_remove snap 105 2026-02-20T11:19:28.862 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:19:28.938 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:28.936+0000 7fd285b9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:28.941 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:28.940+0000 7fd285b9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:28.966 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:19:28.983 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:19:28.983 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:19:29.319 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:19:29.390 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:19:29.392 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:19:29.393 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:19:29.807 INFO:tasks.rados.rados.0.trial043.stdout:876: done (3 left) 2026-02-20T11:19:29.807 INFO:tasks.rados.rados.0.trial043.stdout:877: snap_create 2026-02-20T11:19:29.807 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:19:29.807 INFO:tasks.rados.rados.0.trial043.stdout:872: finishing write tid 1 to trial04342147-37 2026-02-20T11:19:29.807 INFO:tasks.rados.rados.0.trial043.stdout:872: finishing write tid 2 to trial04342147-37 2026-02-20T11:19:29.807 INFO:tasks.rados.rados.0.trial043.stdout:872: finishing write tid 3 to trial04342147-37 2026-02-20T11:19:29.807 INFO:tasks.rados.rados.0.trial043.stdout:872: finishing write tid 4 to trial04342147-37 2026-02-20T11:19:29.808 INFO:tasks.rados.rados.0.trial043.stdout:872: oid 37 updating version 0 to 227 2026-02-20T11:19:29.808 INFO:tasks.rados.rados.0.trial043.stdout:872: oid 37 version 227 is already newer than 227 2026-02-20T11:19:29.808 INFO:tasks.rados.rados.0.trial043.stdout:872: oid 37 version 227 is already newer than 227 2026-02-20T11:19:29.808 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 37 v 227 (ObjNum 406 snap 107 seq_num 406) dirty exists 2026-02-20T11:19:29.808 INFO:tasks.rados.rados.0.trial043.stdout:872: left oid 37 (ObjNum 406 snap 107 seq_num 406) 2026-02-20T11:19:29.808 INFO:tasks.rados.rados.0.trial043.stdout:872: done (2 left) 2026-02-20T11:19:29.808 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:19:29.808 INFO:tasks.rados.rados.0.trial043.stdout:874: expect (ObjNum 375 snap 100 seq_num 375) 2026-02-20T11:19:29.811 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 707418 2026-02-20T11:19:29.831 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:29.878 INFO:tasks.rados.rados.0.trial043.stdout:875: finishing write tid 1 to trial04342147-18 2026-02-20T11:19:29.878 INFO:tasks.rados.rados.0.trial043.stdout:875: finishing write tid 2 to trial04342147-18 2026-02-20T11:19:29.879 INFO:tasks.rados.rados.0.trial043.stdout:874: done (1 left) 2026-02-20T11:19:29.879 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:19:29.879 INFO:tasks.rados.rados.0.trial043.stdout:875: finishing write tid 3 to trial04342147-18 2026-02-20T11:19:29.879 INFO:tasks.rados.rados.0.trial043.stdout:875: finishing write tid 4 to trial04342147-18 2026-02-20T11:19:29.879 INFO:tasks.rados.rados.0.trial043.stdout:875: finishing write tid 5 to trial04342147-18 2026-02-20T11:19:29.879 INFO:tasks.rados.rados.0.trial043.stdout:875: oid 18 updating version 0 to 61 2026-02-20T11:19:29.879 INFO:tasks.rados.rados.0.trial043.stdout:875: oid 18 version 61 is already newer than 61 2026-02-20T11:19:29.879 INFO:tasks.rados.rados.0.trial043.stdout:875: oid 18 version 61 is already newer than 61 2026-02-20T11:19:29.879 INFO:tasks.rados.rados.0.trial043.stdout:875: oid 18 version 61 is already newer than 61 2026-02-20T11:19:29.879 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 18 v 61 (ObjNum 408 snap 107 seq_num 408) dirty exists 2026-02-20T11:19:29.879 INFO:tasks.rados.rados.0.trial043.stdout:875: left oid 18 (ObjNum 408 snap 107 seq_num 408) 2026-02-20T11:19:29.879 INFO:tasks.rados.rados.0.trial043.stdout:875: done (0 left) 2026-02-20T11:19:30.811 INFO:tasks.rados.rados.0.trial043.stdout:877: done (0 left) 2026-02-20T11:19:30.811 INFO:tasks.rados.rados.0.trial043.stdout:878: snap_create 2026-02-20T11:19:30.984 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:19:31.003 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~226s 2026-02-20T11:19:31.062 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:31.060+0000 7f2cc850b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:31.065 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:31.064+0000 7f2cc850b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:31.091 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:19:31.106 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:19:31.107 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:19:31.814 INFO:tasks.rados.rados.0.trial043.stdout:878: done (0 left) 2026-02-20T11:19:31.814 INFO:tasks.rados.rados.0.trial043.stdout:879: write oid 23 current snap is 109 2026-02-20T11:19:31.814 INFO:tasks.rados.rados.0.trial043.stdout:879: seq_num 409 ranges {474063=543550,1650547=514133,2765651=108412} 2026-02-20T11:19:31.818 INFO:tasks.rados.rados.0.trial043.stdout:879: writing trial04342147-23 from 474063 to 1017613 tid 1 2026-02-20T11:19:31.822 INFO:tasks.rados.rados.0.trial043.stdout:879: writing trial04342147-23 from 1650547 to 2164680 tid 2 2026-02-20T11:19:31.825 INFO:tasks.rados.rados.0.trial043.stdout:879: writing trial04342147-23 from 2765651 to 2874063 tid 3 2026-02-20T11:19:31.825 INFO:tasks.rados.rados.0.trial043.stdout:880: snap_create 2026-02-20T11:19:31.826 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:19:31.858 INFO:tasks.rados.rados.0.trial043.stdout:879: finishing write tid 1 to trial04342147-23 2026-02-20T11:19:31.865 INFO:tasks.rados.rados.0.trial043.stdout:879: finishing write tid 2 to trial04342147-23 2026-02-20T11:19:31.871 INFO:tasks.rados.rados.0.trial043.stdout:879: finishing write tid 3 to trial04342147-23 2026-02-20T11:19:31.875 INFO:tasks.rados.rados.0.trial043.stdout:879: finishing write tid 4 to trial04342147-23 2026-02-20T11:19:31.876 INFO:tasks.rados.rados.0.trial043.stdout:879: finishing write tid 5 to trial04342147-23 2026-02-20T11:19:31.877 INFO:tasks.rados.rados.0.trial043.stdout:879: oid 23 updating version 0 to 101 2026-02-20T11:19:31.877 INFO:tasks.rados.rados.0.trial043.stdout:879: oid 23 version 101 is already newer than 101 2026-02-20T11:19:31.877 INFO:tasks.rados.rados.0.trial043.stdout:879: oid 23 version 101 is already newer than 101 2026-02-20T11:19:31.877 INFO:tasks.rados.rados.0.trial043.stdout:879: oid 23 version 101 is already newer than 101 2026-02-20T11:19:31.877 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 23 v 101 (ObjNum 409 snap 109 seq_num 409) dirty exists 2026-02-20T11:19:31.877 INFO:tasks.rados.rados.0.trial043.stdout:879: left oid 23 (ObjNum 409 snap 109 seq_num 409) 2026-02-20T11:19:31.877 INFO:tasks.rados.rados.0.trial043.stdout:879: done (0 left) 2026-02-20T11:19:32.820 INFO:tasks.rados.rados.0.trial043.stdout:880: done (0 left) 2026-02-20T11:19:32.820 INFO:tasks.rados.rados.0.trial043.stdout:881: snap_remove snap 108 2026-02-20T11:19:33.109 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:19:33.185 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:33.183+0000 7f804c9dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:33.188 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:33.187+0000 7f804c9dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:33.214 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:19:33.229 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:19:33.230 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:19:33.819 INFO:tasks.rados.rados.0.trial043.stdout:881: done (0 left) 2026-02-20T11:19:33.819 INFO:tasks.rados.rados.0.trial043.stdout:882: read oid 15 snap 107 2026-02-20T11:19:33.819 INFO:tasks.rados.rados.0.trial043.stdout:882: expect (ObjNum 394 snap 105 seq_num 394) 2026-02-20T11:19:33.819 INFO:tasks.rados.rados.0.trial043.stdout:883: write (excl) oid 49 current snap is 110 2026-02-20T11:19:33.819 INFO:tasks.rados.rados.0.trial043.stdout:883: seq_num 410 ranges {490870=618799,1592676=774144,2793074=97796} 2026-02-20T11:19:33.823 INFO:tasks.rados.rados.0.trial043.stdout:883: writing trial04342147-49 from 490870 to 1109669 tid 1 2026-02-20T11:19:33.828 INFO:tasks.rados.rados.0.trial043.stdout:883: writing trial04342147-49 from 1592676 to 2366820 tid 2 2026-02-20T11:19:33.830 INFO:tasks.rados.rados.0.trial043.stdout:883: writing trial04342147-49 from 2793074 to 2890870 tid 3 2026-02-20T11:19:33.830 INFO:tasks.rados.rados.0.trial043.stdout:884: delete oid 27 current snap is 110 2026-02-20T11:19:33.847 INFO:tasks.rados.rados.0.trial043.stdout:884: done (2 left) 2026-02-20T11:19:33.847 INFO:tasks.rados.rados.0.trial043.stdout:885: snap_create 2026-02-20T11:19:33.847 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:19:33.858 INFO:tasks.rados.rados.0.trial043.stdout:882: expect (ObjNum 394 snap 105 seq_num 394) 2026-02-20T11:19:33.939 INFO:tasks.rados.rados.0.trial043.stdout:883: finishing write tid 1 to trial04342147-49 2026-02-20T11:19:33.939 INFO:tasks.rados.rados.0.trial043.stdout:883: finishing write tid 2 to trial04342147-49 2026-02-20T11:19:33.939 INFO:tasks.rados.rados.0.trial043.stdout:883: finishing write tid 3 to trial04342147-49 2026-02-20T11:19:33.939 INFO:tasks.rados.rados.0.trial043.stdout:883: finishing write tid 4 to trial04342147-49 2026-02-20T11:19:33.939 INFO:tasks.rados.rados.0.trial043.stdout:883: finishing write tid 5 to trial04342147-49 2026-02-20T11:19:33.939 INFO:tasks.rados.rados.0.trial043.stdout:883: oid 49 updating version 0 to 77 2026-02-20T11:19:33.939 INFO:tasks.rados.rados.0.trial043.stdout:883: oid 49 version 77 is already newer than 77 2026-02-20T11:19:33.939 INFO:tasks.rados.rados.0.trial043.stdout:883: oid 49 version 77 is already newer than 77 2026-02-20T11:19:33.940 INFO:tasks.rados.rados.0.trial043.stdout:883: oid 49 version 77 is already newer than 74 2026-02-20T11:19:33.940 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 49 v 77 (ObjNum 410 snap 110 seq_num 410) dirty exists 2026-02-20T11:19:33.940 INFO:tasks.rados.rados.0.trial043.stdout:883: left oid 49 (ObjNum 410 snap 110 seq_num 410) 2026-02-20T11:19:33.940 INFO:tasks.rados.rados.0.trial043.stdout:882: done (1 left) 2026-02-20T11:19:33.940 INFO:tasks.rados.rados.0.trial043.stdout:883: done (0 left) 2026-02-20T11:19:34.395 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:19:34.467 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:19:34.469 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:19:34.470 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:19:34.825 INFO:tasks.rados.rados.0.trial043.stdout:885: done (0 left) 2026-02-20T11:19:34.826 INFO:tasks.rados.rados.0.trial043.stdout:886: snap_remove snap 110 2026-02-20T11:19:35.231 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:19:35.308 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:35.306+0000 7f706090b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:35.310 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:35.309+0000 7f706090b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:35.337 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:19:35.353 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:19:35.353 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:19:35.827 INFO:tasks.rados.rados.0.trial043.stdout:886: done (0 left) 2026-02-20T11:19:35.827 INFO:tasks.rados.rados.0.trial043.stdout:887: write (excl) oid 32 current snap is 111 2026-02-20T11:19:35.827 INFO:tasks.rados.rados.0.trial043.stdout:887: seq_num 412 ranges {524484=485650,1425993=661460,2847920=76564} 2026-02-20T11:19:35.831 INFO:tasks.rados.rados.0.trial043.stdout:887: writing trial04342147-32 from 524484 to 1010134 tid 1 2026-02-20T11:19:35.836 INFO:tasks.rados.rados.0.trial043.stdout:887: writing trial04342147-32 from 1425993 to 2087453 tid 2 2026-02-20T11:19:35.839 INFO:tasks.rados.rados.0.trial043.stdout:887: writing trial04342147-32 from 2847920 to 2924484 tid 3 2026-02-20T11:19:35.839 INFO:tasks.rados.rados.0.trial043.stdout:888: snap_remove snap 107 2026-02-20T11:19:36.507 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~231s 2026-02-20T11:19:36.833 INFO:tasks.rados.rados.0.trial043.stdout:888: done (1 left) 2026-02-20T11:19:36.833 INFO:tasks.rados.rados.0.trial043.stdout:889: snap_remove snap 109 2026-02-20T11:19:37.355 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:19:37.434 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:37.432+0000 7fb65025a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:37.436 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:37.435+0000 7fb65025a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:37.462 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:19:37.479 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:19:37.480 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:19:37.836 INFO:tasks.rados.rados.0.trial043.stdout:889: done (1 left) 2026-02-20T11:19:37.836 INFO:tasks.rados.rados.0.trial043.stdout:890: delete oid 37 current snap is 111 2026-02-20T11:19:37.837 INFO:tasks.rados.rados.0.trial043.stdout:887: finishing write tid 1 to trial04342147-32 2026-02-20T11:19:37.837 INFO:tasks.rados.rados.0.trial043.stdout:887: finishing write tid 2 to trial04342147-32 2026-02-20T11:19:37.837 INFO:tasks.rados.rados.0.trial043.stdout:887: finishing write tid 3 to trial04342147-32 2026-02-20T11:19:37.837 INFO:tasks.rados.rados.0.trial043.stdout:887: finishing write tid 4 to trial04342147-32 2026-02-20T11:19:37.837 INFO:tasks.rados.rados.0.trial043.stdout:887: finishing write tid 5 to trial04342147-32 2026-02-20T11:19:37.837 INFO:tasks.rados.rados.0.trial043.stdout:887: oid 32 updating version 0 to 6 2026-02-20T11:19:37.837 INFO:tasks.rados.rados.0.trial043.stdout:887: oid 32 updating version 6 to 9 2026-02-20T11:19:37.837 INFO:tasks.rados.rados.0.trial043.stdout:887: oid 32 version 9 is already newer than 9 2026-02-20T11:19:37.837 INFO:tasks.rados.rados.0.trial043.stdout:887: oid 32 version 9 is already newer than 9 2026-02-20T11:19:37.837 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 32 v 9 (ObjNum 412 snap 111 seq_num 412) dirty exists 2026-02-20T11:19:37.837 INFO:tasks.rados.rados.0.trial043.stdout:887: left oid 32 (ObjNum 412 snap 111 seq_num 412) 2026-02-20T11:19:37.866 INFO:tasks.rados.rados.0.trial043.stdout:887: done (1 left) 2026-02-20T11:19:37.866 INFO:tasks.rados.rados.0.trial043.stdout:890: done (0 left) 2026-02-20T11:19:37.866 INFO:tasks.rados.rados.0.trial043.stdout:891: write oid 25 current snap is 111 2026-02-20T11:19:37.867 INFO:tasks.rados.rados.0.trial043.stdout:891: seq_num 414 ranges {558098=636148,1826604=665129} 2026-02-20T11:19:37.871 INFO:tasks.rados.rados.0.trial043.stdout:891: writing trial04342147-25 from 558098 to 1194246 tid 1 2026-02-20T11:19:37.876 INFO:tasks.rados.rados.0.trial043.stdout:891: writing trial04342147-25 from 1826604 to 2491733 tid 2 2026-02-20T11:19:37.877 INFO:tasks.rados.rados.0.trial043.stdout:892: read oid 48 snap -1 2026-02-20T11:19:37.877 INFO:tasks.rados.rados.0.trial043.stdout:892: expect deleted 2026-02-20T11:19:37.877 INFO:tasks.rados.rados.0.trial043.stdout:893: write (excl) oid 22 current snap is 111 2026-02-20T11:19:37.877 INFO:tasks.rados.rados.0.trial043.stdout:893: seq_num 415 ranges {574905=711397,1885086=525140,2930189=44716} 2026-02-20T11:19:37.881 INFO:tasks.rados.rados.0.trial043.stdout:893: writing trial04342147-22 from 574905 to 1286302 tid 1 2026-02-20T11:19:37.885 INFO:tasks.rados.rados.0.trial043.stdout:893: writing trial04342147-22 from 1885086 to 2410226 tid 2 2026-02-20T11:19:37.888 INFO:tasks.rados.rados.0.trial043.stdout:893: writing trial04342147-22 from 2930189 to 2974905 tid 3 2026-02-20T11:19:37.888 INFO:tasks.rados.rados.0.trial043.stdout:894: snap_create 2026-02-20T11:19:37.888 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:19:37.888 INFO:tasks.rados.rados.0.trial043.stdout:892: done (2 left) 2026-02-20T11:19:37.888 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:19:37.896 INFO:tasks.rados.rados.0.trial043.stdout:891: finishing write tid 1 to trial04342147-25 2026-02-20T11:19:37.902 INFO:tasks.rados.rados.0.trial043.stdout:893: finishing write tid 1 to trial04342147-22 2026-02-20T11:19:37.904 INFO:tasks.rados.rados.0.trial043.stdout:891: finishing write tid 2 to trial04342147-25 2026-02-20T11:19:37.908 INFO:tasks.rados.rados.0.trial043.stdout:891: finishing write tid 3 to trial04342147-25 2026-02-20T11:19:37.910 INFO:tasks.rados.rados.0.trial043.stdout:891: finishing write tid 4 to trial04342147-25 2026-02-20T11:19:37.910 INFO:tasks.rados.rados.0.trial043.stdout:891: oid 25 updating version 0 to 98 2026-02-20T11:19:37.911 INFO:tasks.rados.rados.0.trial043.stdout:891: oid 25 version 98 is already newer than 98 2026-02-20T11:19:37.911 INFO:tasks.rados.rados.0.trial043.stdout:891: oid 25 version 98 is already newer than 98 2026-02-20T11:19:37.911 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 25 v 98 (ObjNum 414 snap 111 seq_num 414) dirty exists 2026-02-20T11:19:37.911 INFO:tasks.rados.rados.0.trial043.stdout:891: left oid 25 (ObjNum 414 snap 111 seq_num 414) 2026-02-20T11:19:37.911 INFO:tasks.rados.rados.0.trial043.stdout:891: done (1 left) 2026-02-20T11:19:37.911 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:19:37.918 INFO:tasks.rados.rados.0.trial043.stdout:893: finishing write tid 2 to trial04342147-22 2026-02-20T11:19:37.923 INFO:tasks.rados.rados.0.trial043.stdout:893: finishing write tid 3 to trial04342147-22 2026-02-20T11:19:37.927 INFO:tasks.rados.rados.0.trial043.stdout:893: finishing write tid 4 to trial04342147-22 2026-02-20T11:19:37.929 INFO:tasks.rados.rados.0.trial043.stdout:893: finishing write tid 5 to trial04342147-22 2026-02-20T11:19:37.930 INFO:tasks.rados.rados.0.trial043.stdout:893: oid 22 updating version 0 to 99 2026-02-20T11:19:37.930 INFO:tasks.rados.rados.0.trial043.stdout:893: oid 22 updating version 99 to 102 2026-02-20T11:19:37.930 INFO:tasks.rados.rados.0.trial043.stdout:893: oid 22 version 102 is already newer than 102 2026-02-20T11:19:37.930 INFO:tasks.rados.rados.0.trial043.stdout:893: oid 22 version 102 is already newer than 102 2026-02-20T11:19:37.930 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 22 v 102 (ObjNum 415 snap 111 seq_num 415) dirty exists 2026-02-20T11:19:37.930 INFO:tasks.rados.rados.0.trial043.stdout:893: left oid 22 (ObjNum 415 snap 111 seq_num 415) 2026-02-20T11:19:37.930 INFO:tasks.rados.rados.0.trial043.stdout:893: done (0 left) 2026-02-20T11:19:38.777 INFO:tasks.rados.rados.0.trial043.stdout:894: done (0 left) 2026-02-20T11:19:38.777 INFO:tasks.rados.rados.0.trial043.stdout:895: read oid 31 snap 111 2026-02-20T11:19:38.778 INFO:tasks.rados.rados.0.trial043.stdout:895: expect (ObjNum 405 snap 107 seq_num 405) 2026-02-20T11:19:38.778 INFO:tasks.rados.rados.0.trial043.stdout:896: write oid 4 current snap is 112 2026-02-20T11:19:38.778 INFO:tasks.rados.rados.0.trial043.stdout:896: seq_num 416 ranges {591712=786646,1943568=668798} 2026-02-20T11:19:38.782 INFO:tasks.rados.rados.0.trial043.stdout:896: writing trial04342147-4 from 591712 to 1378358 tid 1 2026-02-20T11:19:38.787 INFO:tasks.rados.rados.0.trial043.stdout:896: writing trial04342147-4 from 1943568 to 2612366 tid 2 2026-02-20T11:19:38.788 INFO:tasks.rados.rados.0.trial043.stdout:897: write (excl) oid 33 current snap is 112 2026-02-20T11:19:38.788 INFO:tasks.rados.rados.0.trial043.stdout:897: seq_num 417 ranges {608519=461895,1602050=528809,2585035=423484} 2026-02-20T11:19:38.791 INFO:tasks.rados.rados.0.trial043.stdout:897: writing trial04342147-33 from 608519 to 1070414 tid 1 2026-02-20T11:19:38.795 INFO:tasks.rados.rados.0.trial043.stdout:897: writing trial04342147-33 from 1602050 to 2130859 tid 2 2026-02-20T11:19:38.799 INFO:tasks.rados.rados.0.trial043.stdout:897: writing trial04342147-33 from 2585035 to 3008519 tid 3 2026-02-20T11:19:38.800 INFO:tasks.rados.rados.0.trial043.stdout:898: read oid 12 snap 111 2026-02-20T11:19:38.800 INFO:tasks.rados.rados.0.trial043.stdout:898: expect deleted 2026-02-20T11:19:38.800 INFO:tasks.rados.rados.0.trial043.stdout:899: delete oid 18 current snap is 112 2026-02-20T11:19:38.805 INFO:tasks.rados.rados.0.trial043.stdout:895: expect (ObjNum 405 snap 107 seq_num 405) 2026-02-20T11:19:38.805 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:38.867 INFO:tasks.rados.rados.0.trial043.stdout:897: finishing write tid 1 to trial04342147-33 2026-02-20T11:19:38.876 INFO:tasks.rados.rados.0.trial043.stdout:897: finishing write tid 2 to trial04342147-33 2026-02-20T11:19:38.883 INFO:tasks.rados.rados.0.trial043.stdout:895: done (4 left) 2026-02-20T11:19:38.884 INFO:tasks.rados.rados.0.trial043.stdout:898: done (3 left) 2026-02-20T11:19:38.884 INFO:tasks.rados.rados.0.trial043.stdout:899: done (2 left) 2026-02-20T11:19:38.884 INFO:tasks.rados.rados.0.trial043.stdout:900: delete oid 3 current snap is 112 2026-02-20T11:19:38.884 INFO:tasks.rados.rados.0.trial043.stdout:897: finishing write tid 3 to trial04342147-33 2026-02-20T11:19:38.884 INFO:tasks.rados.rados.0.trial043.stdout:896: finishing write tid 1 to trial04342147-4 2026-02-20T11:19:38.888 INFO:tasks.rados.rados.0.trial043.stdout:897: finishing write tid 4 to trial04342147-33 2026-02-20T11:19:38.890 INFO:tasks.rados.rados.0.trial043.stdout:897: finishing write tid 5 to trial04342147-33 2026-02-20T11:19:38.890 INFO:tasks.rados.rados.0.trial043.stdout:897: oid 33 updating version 0 to 316 2026-02-20T11:19:38.890 INFO:tasks.rados.rados.0.trial043.stdout:897: oid 33 version 316 is already newer than 316 2026-02-20T11:19:38.890 INFO:tasks.rados.rados.0.trial043.stdout:897: oid 33 version 316 is already newer than 316 2026-02-20T11:19:38.890 INFO:tasks.rados.rados.0.trial043.stdout:897: oid 33 version 316 is already newer than 313 2026-02-20T11:19:38.890 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 33 v 316 (ObjNum 417 snap 112 seq_num 417) dirty exists 2026-02-20T11:19:38.890 INFO:tasks.rados.rados.0.trial043.stdout:897: left oid 33 (ObjNum 417 snap 112 seq_num 417) 2026-02-20T11:19:38.894 INFO:tasks.rados.rados.0.trial043.stdout:896: finishing write tid 2 to trial04342147-4 2026-02-20T11:19:38.899 INFO:tasks.rados.rados.0.trial043.stdout:896: finishing write tid 3 to trial04342147-4 2026-02-20T11:19:38.901 INFO:tasks.rados.rados.0.trial043.stdout:896: finishing write tid 4 to trial04342147-4 2026-02-20T11:19:38.901 INFO:tasks.rados.rados.0.trial043.stdout:896: oid 4 updating version 0 to 106 2026-02-20T11:19:38.901 INFO:tasks.rados.rados.0.trial043.stdout:896: oid 4 version 106 is already newer than 106 2026-02-20T11:19:38.901 INFO:tasks.rados.rados.0.trial043.stdout:896: oid 4 version 106 is already newer than 106 2026-02-20T11:19:38.901 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 4 v 106 (ObjNum 416 snap 112 seq_num 416) dirty exists 2026-02-20T11:19:38.901 INFO:tasks.rados.rados.0.trial043.stdout:896: left oid 4 (ObjNum 416 snap 112 seq_num 416) 2026-02-20T11:19:38.911 INFO:tasks.rados.rados.0.trial043.stdout:896: done (2 left) 2026-02-20T11:19:38.911 INFO:tasks.rados.rados.0.trial043.stdout:897: done (1 left) 2026-02-20T11:19:38.912 INFO:tasks.rados.rados.0.trial043.stdout:900: done (0 left) 2026-02-20T11:19:38.912 INFO:tasks.rados.rados.0.trial043.stdout:901: read oid 28 snap -1 2026-02-20T11:19:38.912 INFO:tasks.rados.rados.0.trial043.stdout:901: expect (ObjNum 320 snap 88 seq_num 320) 2026-02-20T11:19:38.912 INFO:tasks.rados.rados.0.trial043.stdout:902: snap_create 2026-02-20T11:19:38.912 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:19:38.950 INFO:tasks.rados.rados.0.trial043.stdout:901: expect (ObjNum 320 snap 88 seq_num 320) 2026-02-20T11:19:38.951 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:39.128 INFO:tasks.rados.rados.0.trial043.stdout:901: done (0 left) 2026-02-20T11:19:39.471 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:19:39.481 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:19:39.542 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:19:39.545 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:19:39.545 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:19:39.555 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:39.553+0000 7fe31e6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:39.557 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:39.557+0000 7fe31e6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:39.583 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:19:39.599 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:19:39.599 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:19:39.844 INFO:tasks.rados.rados.0.trial043.stdout:902: done (0 left) 2026-02-20T11:19:39.844 INFO:tasks.rados.rados.0.trial043.stdout:903: write oid 29 current snap is 113 2026-02-20T11:19:39.844 INFO:tasks.rados.rados.0.trial043.stdout:903: seq_num 420 ranges {658940=403995,1777496=676136,2950951=107989} 2026-02-20T11:19:39.848 INFO:tasks.rados.rados.0.trial043.stdout:903: writing trial04342147-29 from 658940 to 1062935 tid 1 2026-02-20T11:19:39.853 INFO:tasks.rados.rados.0.trial043.stdout:903: writing trial04342147-29 from 1777496 to 2453632 tid 2 2026-02-20T11:19:39.856 INFO:tasks.rados.rados.0.trial043.stdout:903: writing trial04342147-29 from 2950951 to 3058940 tid 3 2026-02-20T11:19:39.856 INFO:tasks.rados.rados.0.trial043.stdout:904: read oid 2 snap -1 2026-02-20T11:19:39.856 INFO:tasks.rados.rados.0.trial043.stdout:904: expect (ObjNum 388 snap 102 seq_num 388) 2026-02-20T11:19:39.856 INFO:tasks.rados.rados.0.trial043.stdout:905: snap_remove snap 112 2026-02-20T11:19:40.848 INFO:tasks.rados.rados.0.trial043.stdout:905: done (2 left) 2026-02-20T11:19:40.849 INFO:tasks.rados.rados.0.trial043.stdout:906: delete oid 36 current snap is 113 2026-02-20T11:19:40.849 INFO:tasks.rados.rados.0.trial043.stdout:904: expect (ObjNum 388 snap 102 seq_num 388) 2026-02-20T11:19:40.877 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:40.898 INFO:tasks.rados.rados.0.trial043.stdout:903: finishing write tid 1 to trial04342147-29 2026-02-20T11:19:40.898 INFO:tasks.rados.rados.0.trial043.stdout:903: finishing write tid 2 to trial04342147-29 2026-02-20T11:19:40.898 INFO:tasks.rados.rados.0.trial043.stdout:904: done (2 left) 2026-02-20T11:19:40.898 INFO:tasks.rados.rados.0.trial043.stdout:906: done (1 left) 2026-02-20T11:19:40.898 INFO:tasks.rados.rados.0.trial043.stdout:907: snap_remove snap 111 2026-02-20T11:19:41.601 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:19:41.678 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:41.676+0000 7f276cc5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:41.680 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:41.679+0000 7f276cc5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:41.708 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:19:41.725 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:19:41.725 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:19:41.851 INFO:tasks.rados.rados.0.trial043.stdout:907: done (1 left) 2026-02-20T11:19:41.851 INFO:tasks.rados.rados.0.trial043.stdout:908: write oid 47 current snap is 113 2026-02-20T11:19:41.851 INFO:tasks.rados.rados.0.trial043.stdout:903: finishing write tid 3 to trial04342147-29 2026-02-20T11:19:41.851 INFO:tasks.rados.rados.0.trial043.stdout:908: seq_num 422 ranges {692554=554493,1778107=679805,2889444=203110} 2026-02-20T11:19:41.855 INFO:tasks.rados.rados.0.trial043.stdout:908: writing trial04342147-47 from 692554 to 1247047 tid 1 2026-02-20T11:19:41.860 INFO:tasks.rados.rados.0.trial043.stdout:908: writing trial04342147-47 from 1778107 to 2457912 tid 2 2026-02-20T11:19:41.863 INFO:tasks.rados.rados.0.trial043.stdout:908: writing trial04342147-47 from 2889444 to 3092554 tid 3 2026-02-20T11:19:41.863 INFO:tasks.rados.rados.0.trial043.stdout:909: read oid 48 snap -1 2026-02-20T11:19:41.863 INFO:tasks.rados.rados.0.trial043.stdout:909: expect deleted 2026-02-20T11:19:41.863 INFO:tasks.rados.rados.0.trial043.stdout:910: snap_create 2026-02-20T11:19:41.863 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:19:41.863 INFO:tasks.rados.rados.0.trial043.stdout:903: finishing write tid 4 to trial04342147-29 2026-02-20T11:19:41.864 INFO:tasks.rados.rados.0.trial043.stdout:903: finishing write tid 5 to trial04342147-29 2026-02-20T11:19:41.864 INFO:tasks.rados.rados.0.trial043.stdout:903: oid 29 updating version 0 to 321 2026-02-20T11:19:41.864 INFO:tasks.rados.rados.0.trial043.stdout:903: oid 29 version 321 is already newer than 321 2026-02-20T11:19:41.864 INFO:tasks.rados.rados.0.trial043.stdout:903: oid 29 version 321 is already newer than 321 2026-02-20T11:19:41.864 INFO:tasks.rados.rados.0.trial043.stdout:903: oid 29 version 321 is already newer than 321 2026-02-20T11:19:41.864 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 29 v 321 (ObjNum 420 snap 113 seq_num 420) dirty exists 2026-02-20T11:19:41.864 INFO:tasks.rados.rados.0.trial043.stdout:903: left oid 29 (ObjNum 420 snap 113 seq_num 420) 2026-02-20T11:19:41.864 INFO:tasks.rados.rados.0.trial043.stdout:903: done (2 left) 2026-02-20T11:19:41.864 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:19:41.870 INFO:tasks.rados.rados.0.trial043.stdout:909: done (1 left) 2026-02-20T11:19:41.870 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:19:41.885 INFO:tasks.rados.rados.0.trial043.stdout:908: finishing write tid 1 to trial04342147-47 2026-02-20T11:19:41.894 INFO:tasks.rados.rados.0.trial043.stdout:908: finishing write tid 2 to trial04342147-47 2026-02-20T11:19:41.900 INFO:tasks.rados.rados.0.trial043.stdout:908: finishing write tid 3 to trial04342147-47 2026-02-20T11:19:41.904 INFO:tasks.rados.rados.0.trial043.stdout:908: finishing write tid 4 to trial04342147-47 2026-02-20T11:19:41.906 INFO:tasks.rados.rados.0.trial043.stdout:908: finishing write tid 5 to trial04342147-47 2026-02-20T11:19:41.906 INFO:tasks.rados.rados.0.trial043.stdout:908: oid 47 updating version 0 to 125 2026-02-20T11:19:41.906 INFO:tasks.rados.rados.0.trial043.stdout:908: oid 47 version 125 is already newer than 125 2026-02-20T11:19:41.906 INFO:tasks.rados.rados.0.trial043.stdout:908: oid 47 version 125 is already newer than 125 2026-02-20T11:19:41.906 INFO:tasks.rados.rados.0.trial043.stdout:908: oid 47 version 125 is already newer than 125 2026-02-20T11:19:41.906 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 47 v 125 (ObjNum 422 snap 113 seq_num 422) dirty exists 2026-02-20T11:19:41.907 INFO:tasks.rados.rados.0.trial043.stdout:908: left oid 47 (ObjNum 422 snap 113 seq_num 422) 2026-02-20T11:19:41.907 INFO:tasks.rados.rados.0.trial043.stdout:908: done (0 left) 2026-02-20T11:19:42.011 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~237s 2026-02-20T11:19:42.859 INFO:tasks.rados.rados.0.trial043.stdout:910: done (0 left) 2026-02-20T11:19:42.859 INFO:tasks.rados.rados.0.trial043.stdout:911: snap_create 2026-02-20T11:19:43.727 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:19:43.804 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:43.801+0000 7f74c51a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:43.806 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:43.805+0000 7f74c51a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:43.831 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:19:43.848 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:19:43.849 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:19:43.863 INFO:tasks.rados.rados.0.trial043.stdout:911: done (0 left) 2026-02-20T11:19:43.863 INFO:tasks.rados.rados.0.trial043.stdout:912: read oid 43 snap -1 2026-02-20T11:19:43.863 INFO:tasks.rados.rados.0.trial043.stdout:912: expect (ObjNum 386 snap 101 seq_num 386) 2026-02-20T11:19:43.864 INFO:tasks.rados.rados.0.trial043.stdout:913: read oid 17 snap 114 2026-02-20T11:19:43.864 INFO:tasks.rados.rados.0.trial043.stdout:913: expect deleted 2026-02-20T11:19:43.864 INFO:tasks.rados.rados.0.trial043.stdout:914: read oid 46 snap 114 2026-02-20T11:19:43.864 INFO:tasks.rados.rados.0.trial043.stdout:914: expect (ObjNum 326 snap 89 seq_num 326) 2026-02-20T11:19:43.864 INFO:tasks.rados.rados.0.trial043.stdout:915: snap_remove snap 113 2026-02-20T11:19:44.546 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:19:44.618 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:19:44.621 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:19:44.622 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:19:44.869 INFO:tasks.rados.rados.0.trial043.stdout:915: done (3 left) 2026-02-20T11:19:44.869 INFO:tasks.rados.rados.0.trial043.stdout:916: read oid 48 snap -1 2026-02-20T11:19:44.869 INFO:tasks.rados.rados.0.trial043.stdout:916: expect deleted 2026-02-20T11:19:44.869 INFO:tasks.rados.rados.0.trial043.stdout:917: delete oid 50 current snap is 115 2026-02-20T11:19:44.869 INFO:tasks.rados.rados.0.trial043.stdout:914: expect (ObjNum 326 snap 89 seq_num 326) 2026-02-20T11:19:44.890 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2314131 2026-02-20T11:19:44.969 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1341863 2026-02-20T11:19:45.000 INFO:tasks.rados.rados.0.trial043.stdout:912: expect (ObjNum 386 snap 101 seq_num 386) 2026-02-20T11:19:45.046 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:45.139 INFO:tasks.rados.rados.0.trial043.stdout:912: done (4 left) 2026-02-20T11:19:45.140 INFO:tasks.rados.rados.0.trial043.stdout:913: done (3 left) 2026-02-20T11:19:45.140 INFO:tasks.rados.rados.0.trial043.stdout:914: done (2 left) 2026-02-20T11:19:45.140 INFO:tasks.rados.rados.0.trial043.stdout:917: done (1 left) 2026-02-20T11:19:45.140 INFO:tasks.rados.rados.0.trial043.stdout:918: snap_create 2026-02-20T11:19:45.140 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:19:45.140 INFO:tasks.rados.rados.0.trial043.stdout:916: done (0 left) 2026-02-20T11:19:45.849 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:19:45.870 INFO:tasks.rados.rados.0.trial043.stdout:918: done (0 left) 2026-02-20T11:19:45.870 INFO:tasks.rados.rados.0.trial043.stdout:919: write oid 6 current snap is 116 2026-02-20T11:19:45.870 INFO:tasks.rados.rados.0.trial043.stdout:919: seq_num 424 ranges {726168=704991,1895071=683474} 2026-02-20T11:19:45.876 INFO:tasks.rados.rados.0.trial043.stdout:919: writing trial04342147-6 from 726168 to 1431159 tid 1 2026-02-20T11:19:45.880 INFO:tasks.rados.rados.0.trial043.stdout:919: writing trial04342147-6 from 1895071 to 2578545 tid 2 2026-02-20T11:19:45.880 INFO:tasks.rados.rados.0.trial043.stdout:920: write (excl) oid 39 current snap is 116 2026-02-20T11:19:45.881 INFO:tasks.rados.rados.0.trial043.stdout:920: seq_num 425 ranges {742975=780240,1953553=427132,2971713=171262} 2026-02-20T11:19:45.886 INFO:tasks.rados.rados.0.trial043.stdout:920: writing trial04342147-39 from 742975 to 1523215 tid 1 2026-02-20T11:19:45.889 INFO:tasks.rados.rados.0.trial043.stdout:920: writing trial04342147-39 from 1953553 to 2380685 tid 2 2026-02-20T11:19:45.892 INFO:tasks.rados.rados.0.trial043.stdout:920: writing trial04342147-39 from 2971713 to 3142975 tid 3 2026-02-20T11:19:45.892 INFO:tasks.rados.rados.0.trial043.stdout:921: write oid 49 current snap is 116 2026-02-20T11:19:45.892 INFO:tasks.rados.rados.0.trial043.stdout:921: seq_num 426 ranges {759782=571842,2012035=687143} 2026-02-20T11:19:45.897 INFO:tasks.rados.rados.0.trial043.stdout:921: writing trial04342147-49 from 759782 to 1331624 tid 1 2026-02-20T11:19:45.902 INFO:tasks.rados.rados.0.trial043.stdout:921: writing trial04342147-49 from 2012035 to 2699178 tid 2 2026-02-20T11:19:45.902 INFO:tasks.rados.rados.0.trial043.stdout:922: write oid 44 current snap is 116 2026-02-20T11:19:45.902 INFO:tasks.rados.rados.0.trial043.stdout:922: seq_num 427 ranges {776589=647091,2070517=430801,3026559=150030} 2026-02-20T11:19:45.907 INFO:tasks.rados.rados.0.trial043.stdout:922: writing trial04342147-44 from 776589 to 1423680 tid 1 2026-02-20T11:19:45.912 INFO:tasks.rados.rados.0.trial043.stdout:922: writing trial04342147-44 from 2070517 to 2501318 tid 2 2026-02-20T11:19:45.914 INFO:tasks.rados.rados.0.trial043.stdout:922: writing trial04342147-44 from 3026559 to 3176589 tid 3 2026-02-20T11:19:45.914 INFO:tasks.rados.rados.0.trial043.stdout:923: snap_create 2026-02-20T11:19:45.914 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 4 2026-02-20T11:19:45.914 INFO:tasks.rados.rados.0.trial043.stdout:920: finishing write tid 1 to trial04342147-39 2026-02-20T11:19:45.919 INFO:tasks.rados.rados.0.trial043.stdout:919: finishing write tid 1 to trial04342147-6 2026-02-20T11:19:45.932 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:45.930+0000 7fbeab88b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:45.933 INFO:tasks.rados.rados.0.trial043.stdout:919: finishing write tid 2 to trial04342147-6 2026-02-20T11:19:45.933 INFO:tasks.rados.rados.0.trial043.stdout:920: finishing write tid 2 to trial04342147-39 2026-02-20T11:19:45.935 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:45.934+0000 7fbeab88b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:45.941 INFO:tasks.rados.rados.0.trial043.stdout:920: finishing write tid 3 to trial04342147-39 2026-02-20T11:19:45.941 INFO:tasks.rados.rados.0.trial043.stdout:919: finishing write tid 3 to trial04342147-6 2026-02-20T11:19:45.944 INFO:tasks.rados.rados.0.trial043.stdout:919: finishing write tid 4 to trial04342147-6 2026-02-20T11:19:45.944 INFO:tasks.rados.rados.0.trial043.stdout:919: oid 6 updating version 0 to 37 2026-02-20T11:19:45.944 INFO:tasks.rados.rados.0.trial043.stdout:919: oid 6 version 37 is already newer than 37 2026-02-20T11:19:45.944 INFO:tasks.rados.rados.0.trial043.stdout:919: oid 6 version 37 is already newer than 37 2026-02-20T11:19:45.944 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 6 v 37 (ObjNum 424 snap 116 seq_num 424) dirty exists 2026-02-20T11:19:45.944 INFO:tasks.rados.rados.0.trial043.stdout:919: left oid 6 (ObjNum 424 snap 116 seq_num 424) 2026-02-20T11:19:45.944 INFO:tasks.rados.rados.0.trial043.stdout:919: done (3 left) 2026-02-20T11:19:45.944 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:19:45.947 INFO:tasks.rados.rados.0.trial043.stdout:922: finishing write tid 1 to trial04342147-44 2026-02-20T11:19:45.948 INFO:tasks.rados.rados.0.trial043.stdout:920: finishing write tid 4 to trial04342147-39 2026-02-20T11:19:45.951 INFO:tasks.rados.rados.0.trial043.stdout:920: finishing write tid 5 to trial04342147-39 2026-02-20T11:19:45.952 INFO:tasks.rados.rados.0.trial043.stdout:920: oid 39 updating version 0 to 46 2026-02-20T11:19:45.952 INFO:tasks.rados.rados.0.trial043.stdout:920: oid 39 updating version 46 to 49 2026-02-20T11:19:45.952 INFO:tasks.rados.rados.0.trial043.stdout:920: oid 39 version 49 is already newer than 49 2026-02-20T11:19:45.952 INFO:tasks.rados.rados.0.trial043.stdout:920: oid 39 version 49 is already newer than 49 2026-02-20T11:19:45.952 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 39 v 49 (ObjNum 425 snap 116 seq_num 425) dirty exists 2026-02-20T11:19:45.952 INFO:tasks.rados.rados.0.trial043.stdout:920: left oid 39 (ObjNum 425 snap 116 seq_num 425) 2026-02-20T11:19:45.952 INFO:tasks.rados.rados.0.trial043.stdout:920: done (2 left) 2026-02-20T11:19:45.952 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:19:45.954 INFO:tasks.rados.rados.0.trial043.stdout:922: finishing write tid 2 to trial04342147-44 2026-02-20T11:19:45.965 INFO:tasks.rados.rados.0.trial043.stdout:922: finishing write tid 3 to trial04342147-44 2026-02-20T11:19:45.965 INFO:tasks.rados.rados.0.trial043.stdout:922: finishing write tid 4 to trial04342147-44 2026-02-20T11:19:45.967 INFO:tasks.rados.rados.0.trial043.stdout:921: finishing write tid 1 to trial04342147-49 2026-02-20T11:19:45.969 INFO:tasks.rados.rados.0.trial043.stdout:922: finishing write tid 5 to trial04342147-44 2026-02-20T11:19:45.969 INFO:tasks.rados.rados.0.trial043.stdout:922: oid 44 updating version 0 to 130 2026-02-20T11:19:45.969 INFO:tasks.rados.rados.0.trial043.stdout:922: oid 44 version 130 is already newer than 130 2026-02-20T11:19:45.969 INFO:tasks.rados.rados.0.trial043.stdout:922: oid 44 version 130 is already newer than 130 2026-02-20T11:19:45.969 INFO:tasks.rados.rados.0.trial043.stdout:922: oid 44 version 130 is already newer than 130 2026-02-20T11:19:45.969 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 44 v 130 (ObjNum 427 snap 116 seq_num 427) dirty exists 2026-02-20T11:19:45.969 INFO:tasks.rados.rados.0.trial043.stdout:922: left oid 44 (ObjNum 427 snap 116 seq_num 427) 2026-02-20T11:19:45.969 INFO:tasks.rados.rados.0.trial043.stdout:922: done (1 left) 2026-02-20T11:19:45.969 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:19:45.971 INFO:tasks.rados.rados.0.trial043.stdout:921: finishing write tid 2 to trial04342147-49 2026-02-20T11:19:45.976 INFO:tasks.rados.rados.0.trial043.stdout:921: finishing write tid 3 to trial04342147-49 2026-02-20T11:19:45.978 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:19:45.979 INFO:tasks.rados.rados.0.trial043.stdout:921: finishing write tid 4 to trial04342147-49 2026-02-20T11:19:45.979 INFO:tasks.rados.rados.0.trial043.stdout:921: oid 49 updating version 0 to 81 2026-02-20T11:19:45.979 INFO:tasks.rados.rados.0.trial043.stdout:921: oid 49 version 81 is already newer than 81 2026-02-20T11:19:45.979 INFO:tasks.rados.rados.0.trial043.stdout:921: oid 49 version 81 is already newer than 81 2026-02-20T11:19:45.979 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 49 v 81 (ObjNum 426 snap 116 seq_num 426) dirty exists 2026-02-20T11:19:45.979 INFO:tasks.rados.rados.0.trial043.stdout:921: left oid 49 (ObjNum 426 snap 116 seq_num 426) 2026-02-20T11:19:45.979 INFO:tasks.rados.rados.0.trial043.stdout:921: done (0 left) 2026-02-20T11:19:45.994 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:19:45.994 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:19:46.875 INFO:tasks.rados.rados.0.trial043.stdout:923: done (0 left) 2026-02-20T11:19:46.875 INFO:tasks.rados.rados.0.trial043.stdout:924: write oid 32 current snap is 117 2026-02-20T11:19:46.875 INFO:tasks.rados.rados.0.trial043.stdout:924: seq_num 428 ranges {793396=722340,2128999=690812} 2026-02-20T11:19:46.881 INFO:tasks.rados.rados.0.trial043.stdout:924: writing trial04342147-32 from 793396 to 1515736 tid 1 2026-02-20T11:19:46.886 INFO:tasks.rados.rados.0.trial043.stdout:924: writing trial04342147-32 from 2128999 to 2819811 tid 2 2026-02-20T11:19:46.886 INFO:tasks.rados.rados.0.trial043.stdout:925: delete oid 34 current snap is 117 2026-02-20T11:19:46.890 INFO:tasks.rados.rados.0.trial043.stdout:925: done (1 left) 2026-02-20T11:19:46.890 INFO:tasks.rados.rados.0.trial043.stdout:926: read oid 20 snap -1 2026-02-20T11:19:46.890 INFO:tasks.rados.rados.0.trial043.stdout:926: expect (ObjNum 128 snap 17 seq_num 128) 2026-02-20T11:19:46.890 INFO:tasks.rados.rados.0.trial043.stdout:927: snap_create 2026-02-20T11:19:46.890 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:19:46.908 INFO:tasks.rados.rados.0.trial043.stdout:926: expect (ObjNum 128 snap 17 seq_num 128) 2026-02-20T11:19:46.909 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:46.934 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 170976 2026-02-20T11:19:46.963 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1009321 2026-02-20T11:19:46.979 INFO:tasks.rados.rados.0.trial043.stdout:926: done (1 left) 2026-02-20T11:19:46.979 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:19:46.979 INFO:tasks.rados.rados.0.trial043.stdout:924: finishing write tid 1 to trial04342147-32 2026-02-20T11:19:46.980 INFO:tasks.rados.rados.0.trial043.stdout:924: finishing write tid 2 to trial04342147-32 2026-02-20T11:19:46.980 INFO:tasks.rados.rados.0.trial043.stdout:924: finishing write tid 3 to trial04342147-32 2026-02-20T11:19:46.980 INFO:tasks.rados.rados.0.trial043.stdout:924: finishing write tid 4 to trial04342147-32 2026-02-20T11:19:46.980 INFO:tasks.rados.rados.0.trial043.stdout:924: oid 32 updating version 0 to 13 2026-02-20T11:19:46.980 INFO:tasks.rados.rados.0.trial043.stdout:924: oid 32 version 13 is already newer than 13 2026-02-20T11:19:46.980 INFO:tasks.rados.rados.0.trial043.stdout:924: oid 32 version 13 is already newer than 13 2026-02-20T11:19:46.980 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 32 v 13 (ObjNum 428 snap 117 seq_num 428) dirty exists 2026-02-20T11:19:46.980 INFO:tasks.rados.rados.0.trial043.stdout:924: left oid 32 (ObjNum 428 snap 117 seq_num 428) 2026-02-20T11:19:46.980 INFO:tasks.rados.rados.0.trial043.stdout:924: done (0 left) 2026-02-20T11:19:47.515 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~242s 2026-02-20T11:19:47.879 INFO:tasks.rados.rados.0.trial043.stdout:927: done (0 left) 2026-02-20T11:19:47.879 INFO:tasks.rados.rados.0.trial043.stdout:928: delete oid 7 current snap is 118 2026-02-20T11:19:47.895 INFO:tasks.rados.rados.0.trial043.stdout:928: done (0 left) 2026-02-20T11:19:47.895 INFO:tasks.rados.rados.0.trial043.stdout:929: read oid 22 snap -1 2026-02-20T11:19:47.895 INFO:tasks.rados.rados.0.trial043.stdout:929: expect (ObjNum 415 snap 111 seq_num 415) 2026-02-20T11:19:47.895 INFO:tasks.rados.rados.0.trial043.stdout:930: read oid 46 snap 115 2026-02-20T11:19:47.895 INFO:tasks.rados.rados.0.trial043.stdout:930: expect (ObjNum 326 snap 89 seq_num 326) 2026-02-20T11:19:47.895 INFO:tasks.rados.rados.0.trial043.stdout:931: read oid 15 snap -1 2026-02-20T11:19:47.895 INFO:tasks.rados.rados.0.trial043.stdout:931: expect (ObjNum 394 snap 105 seq_num 394) 2026-02-20T11:19:47.896 INFO:tasks.rados.rados.0.trial043.stdout:932: read oid 39 snap -1 2026-02-20T11:19:47.896 INFO:tasks.rados.rados.0.trial043.stdout:932: expect (ObjNum 425 snap 116 seq_num 425) 2026-02-20T11:19:47.896 INFO:tasks.rados.rados.0.trial043.stdout:933: read oid 3 snap 116 2026-02-20T11:19:47.896 INFO:tasks.rados.rados.0.trial043.stdout:933: expect deleted 2026-02-20T11:19:47.896 INFO:tasks.rados.rados.0.trial043.stdout:934: snap_remove snap 114 2026-02-20T11:19:47.996 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:19:48.070 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:48.069+0000 7f0767acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:48.073 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:48.072+0000 7f0767acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:48.098 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:19:48.115 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:19:48.115 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:19:48.779 INFO:tasks.rados.rados.0.trial043.stdout:934: done (5 left) 2026-02-20T11:19:48.779 INFO:tasks.rados.rados.0.trial043.stdout:935: write oid 50 current snap is 118 2026-02-20T11:19:48.779 INFO:tasks.rados.rados.0.trial043.stdout:935: seq_num 431 ranges {443817=548087,1788092=438139,3019898=223919} 2026-02-20T11:19:48.786 INFO:tasks.rados.rados.0.trial043.stdout:935: writing trial04342147-50 from 443817 to 991904 tid 1 2026-02-20T11:19:48.791 INFO:tasks.rados.rados.0.trial043.stdout:935: writing trial04342147-50 from 1788092 to 2226231 tid 2 2026-02-20T11:19:48.796 INFO:tasks.rados.rados.0.trial043.stdout:935: writing trial04342147-50 from 3019898 to 3243817 tid 3 2026-02-20T11:19:48.796 INFO:tasks.rados.rados.0.trial043.stdout:936: read oid 26 snap 117 2026-02-20T11:19:48.796 INFO:tasks.rados.rados.0.trial043.stdout:936: expect (ObjNum 323 snap 88 seq_num 323) 2026-02-20T11:19:48.796 INFO:tasks.rados.rados.0.trial043.stdout:937: write (excl) oid 12 current snap is 118 2026-02-20T11:19:48.796 INFO:tasks.rados.rados.0.trial043.stdout:937: seq_num 432 ranges {460624=623336,1846574=698150,3163674=96950} 2026-02-20T11:19:48.799 INFO:tasks.rados.rados.0.trial043.stdout:937: writing trial04342147-12 from 460624 to 1083960 tid 1 2026-02-20T11:19:48.805 INFO:tasks.rados.rados.0.trial043.stdout:937: writing trial04342147-12 from 1846574 to 2544724 tid 2 2026-02-20T11:19:48.808 INFO:tasks.rados.rados.0.trial043.stdout:937: writing trial04342147-12 from 3163674 to 3260624 tid 3 2026-02-20T11:19:48.808 INFO:tasks.rados.rados.0.trial043.stdout:938: write (excl) oid 7 current snap is 118 2026-02-20T11:19:48.808 INFO:tasks.rados.rados.0.trial043.stdout:938: seq_num 433 ranges {477431=698585,1905056=441808,3074744=202687} 2026-02-20T11:19:48.812 INFO:tasks.rados.rados.0.trial043.stdout:938: writing trial04342147-7 from 477431 to 1176016 tid 1 2026-02-20T11:19:48.816 INFO:tasks.rados.rados.0.trial043.stdout:938: writing trial04342147-7 from 1905056 to 2346864 tid 2 2026-02-20T11:19:48.820 INFO:tasks.rados.rados.0.trial043.stdout:938: writing trial04342147-7 from 3074744 to 3277431 tid 3 2026-02-20T11:19:48.820 INFO:tasks.rados.rados.0.trial043.stdout:939: snap_create 2026-02-20T11:19:48.821 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 9 2026-02-20T11:19:48.821 INFO:tasks.rados.rados.0.trial043.stdout:933: done (8 left) 2026-02-20T11:19:48.821 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 8 2026-02-20T11:19:48.821 INFO:tasks.rados.rados.0.trial043.stdout:929: expect (ObjNum 415 snap 111 seq_num 415) 2026-02-20T11:19:48.873 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:48.949 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1743052 2026-02-20T11:19:48.969 INFO:tasks.rados.rados.0.trial043.stdout:930: expect (ObjNum 326 snap 89 seq_num 326) 2026-02-20T11:19:49.014 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 647510 2026-02-20T11:19:49.104 INFO:tasks.rados.rados.0.trial043.stdout:931: expect (ObjNum 394 snap 105 seq_num 394) 2026-02-20T11:19:49.105 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:49.133 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:49.162 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:49.191 INFO:tasks.rados.rados.0.trial043.stdout:932: expect (ObjNum 425 snap 116 seq_num 425) 2026-02-20T11:19:49.260 INFO:tasks.rados.rados.0.trial043.stdout:929: done (7 left) 2026-02-20T11:19:49.260 INFO:tasks.rados.rados.0.trial043.stdout:930: done (6 left) 2026-02-20T11:19:49.260 INFO:tasks.rados.rados.0.trial043.stdout:931: done (5 left) 2026-02-20T11:19:49.261 INFO:tasks.rados.rados.0.trial043.stdout:932: done (4 left) 2026-02-20T11:19:49.261 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 4 2026-02-20T11:19:49.261 INFO:tasks.rados.rados.0.trial043.stdout:937: finishing write tid 1 to trial04342147-12 2026-02-20T11:19:49.261 INFO:tasks.rados.rados.0.trial043.stdout:935: finishing write tid 1 to trial04342147-50 2026-02-20T11:19:49.261 INFO:tasks.rados.rados.0.trial043.stdout:938: finishing write tid 1 to trial04342147-7 2026-02-20T11:19:49.261 INFO:tasks.rados.rados.0.trial043.stdout:937: finishing write tid 2 to trial04342147-12 2026-02-20T11:19:49.261 INFO:tasks.rados.rados.0.trial043.stdout:935: finishing write tid 2 to trial04342147-50 2026-02-20T11:19:49.261 INFO:tasks.rados.rados.0.trial043.stdout:937: finishing write tid 3 to trial04342147-12 2026-02-20T11:19:49.261 INFO:tasks.rados.rados.0.trial043.stdout:936: expect (ObjNum 323 snap 88 seq_num 323) 2026-02-20T11:19:49.262 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:49.349 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 3352606 2026-02-20T11:19:49.354 INFO:tasks.rados.rados.0.trial043.stdout:935: finishing write tid 3 to trial04342147-50 2026-02-20T11:19:49.355 INFO:tasks.rados.rados.0.trial043.stdout:938: finishing write tid 2 to trial04342147-7 2026-02-20T11:19:49.355 INFO:tasks.rados.rados.0.trial043.stdout:937: finishing write tid 4 to trial04342147-12 2026-02-20T11:19:49.355 INFO:tasks.rados.rados.0.trial043.stdout:937: finishing write tid 5 to trial04342147-12 2026-02-20T11:19:49.355 INFO:tasks.rados.rados.0.trial043.stdout:937: oid 12 updating version 0 to 134 2026-02-20T11:19:49.355 INFO:tasks.rados.rados.0.trial043.stdout:937: oid 12 version 134 is already newer than 131 2026-02-20T11:19:49.355 INFO:tasks.rados.rados.0.trial043.stdout:937: oid 12 version 134 is already newer than 134 2026-02-20T11:19:49.355 INFO:tasks.rados.rados.0.trial043.stdout:937: oid 12 version 134 is already newer than 134 2026-02-20T11:19:49.355 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 12 v 134 (ObjNum 432 snap 118 seq_num 432) dirty exists 2026-02-20T11:19:49.356 INFO:tasks.rados.rados.0.trial043.stdout:937: left oid 12 (ObjNum 432 snap 118 seq_num 432) 2026-02-20T11:19:49.356 INFO:tasks.rados.rados.0.trial043.stdout:935: finishing write tid 4 to trial04342147-50 2026-02-20T11:19:49.356 INFO:tasks.rados.rados.0.trial043.stdout:935: finishing write tid 5 to trial04342147-50 2026-02-20T11:19:49.356 INFO:tasks.rados.rados.0.trial043.stdout:935: oid 50 updating version 0 to 110 2026-02-20T11:19:49.356 INFO:tasks.rados.rados.0.trial043.stdout:935: oid 50 version 110 is already newer than 110 2026-02-20T11:19:49.356 INFO:tasks.rados.rados.0.trial043.stdout:935: oid 50 version 110 is already newer than 110 2026-02-20T11:19:49.356 INFO:tasks.rados.rados.0.trial043.stdout:935: oid 50 version 110 is already newer than 110 2026-02-20T11:19:49.356 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 50 v 110 (ObjNum 431 snap 118 seq_num 431) dirty exists 2026-02-20T11:19:49.356 INFO:tasks.rados.rados.0.trial043.stdout:935: left oid 50 (ObjNum 431 snap 118 seq_num 431) 2026-02-20T11:19:49.356 INFO:tasks.rados.rados.0.trial043.stdout:938: finishing write tid 3 to trial04342147-7 2026-02-20T11:19:49.357 INFO:tasks.rados.rados.0.trial043.stdout:938: finishing write tid 4 to trial04342147-7 2026-02-20T11:19:49.357 INFO:tasks.rados.rados.0.trial043.stdout:938: finishing write tid 5 to trial04342147-7 2026-02-20T11:19:49.357 INFO:tasks.rados.rados.0.trial043.stdout:938: oid 7 updating version 0 to 105 2026-02-20T11:19:49.357 INFO:tasks.rados.rados.0.trial043.stdout:938: oid 7 version 105 is already newer than 105 2026-02-20T11:19:49.357 INFO:tasks.rados.rados.0.trial043.stdout:938: oid 7 version 105 is already newer than 102 2026-02-20T11:19:49.357 INFO:tasks.rados.rados.0.trial043.stdout:938: oid 7 version 105 is already newer than 105 2026-02-20T11:19:49.357 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 7 v 105 (ObjNum 433 snap 118 seq_num 433) dirty exists 2026-02-20T11:19:49.357 INFO:tasks.rados.rados.0.trial043.stdout:938: left oid 7 (ObjNum 433 snap 118 seq_num 433) 2026-02-20T11:19:49.357 INFO:tasks.rados.rados.0.trial043.stdout:935: done (3 left) 2026-02-20T11:19:49.357 INFO:tasks.rados.rados.0.trial043.stdout:936: done (2 left) 2026-02-20T11:19:49.357 INFO:tasks.rados.rados.0.trial043.stdout:937: done (1 left) 2026-02-20T11:19:49.358 INFO:tasks.rados.rados.0.trial043.stdout:938: done (0 left) 2026-02-20T11:19:49.623 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:19:49.695 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:19:49.698 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:19:49.699 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:19:49.884 INFO:tasks.rados.rados.0.trial043.stdout:939: done (0 left) 2026-02-20T11:19:49.884 INFO:tasks.rados.rados.0.trial043.stdout:940: write oid 32 current snap is 119 2026-02-20T11:19:49.884 INFO:tasks.rados.rados.0.trial043.stdout:940: seq_num 434 ranges {494238=490187,1563538=701819,2818520=475718} 2026-02-20T11:19:49.888 INFO:tasks.rados.rados.0.trial043.stdout:940: writing trial04342147-32 from 494238 to 984425 tid 1 2026-02-20T11:19:49.893 INFO:tasks.rados.rados.0.trial043.stdout:940: writing trial04342147-32 from 1563538 to 2265357 tid 2 2026-02-20T11:19:49.897 INFO:tasks.rados.rados.0.trial043.stdout:940: writing trial04342147-32 from 2818520 to 3294238 tid 3 2026-02-20T11:19:49.897 INFO:tasks.rados.rados.0.trial043.stdout:941: write oid 28 current snap is 119 2026-02-20T11:19:49.898 INFO:tasks.rados.rados.0.trial043.stdout:941: seq_num 435 ranges {511045=565436,1622020=445477,2729590=556855} 2026-02-20T11:19:49.901 INFO:tasks.rados.rados.0.trial043.stdout:941: writing trial04342147-28 from 511045 to 1076481 tid 1 2026-02-20T11:19:49.905 INFO:tasks.rados.rados.0.trial043.stdout:941: writing trial04342147-28 from 1622020 to 2067497 tid 2 2026-02-20T11:19:49.910 INFO:tasks.rados.rados.0.trial043.stdout:941: writing trial04342147-28 from 2729590 to 3286445 tid 3 2026-02-20T11:19:49.910 INFO:tasks.rados.rados.0.trial043.stdout:942: read oid 26 snap 115 2026-02-20T11:19:49.910 INFO:tasks.rados.rados.0.trial043.stdout:942: expect (ObjNum 323 snap 88 seq_num 323) 2026-02-20T11:19:49.910 INFO:tasks.rados.rados.0.trial043.stdout:943: delete oid 7 current snap is 119 2026-02-20T11:19:49.933 INFO:tasks.rados.rados.0.trial043.stdout:940: finishing write tid 1 to trial04342147-32 2026-02-20T11:19:49.937 INFO:tasks.rados.rados.0.trial043.stdout:942: expect (ObjNum 323 snap 88 seq_num 323) 2026-02-20T11:19:49.937 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:49.976 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:50.029 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2489515 2026-02-20T11:19:50.042 INFO:tasks.rados.rados.0.trial043.stdout:942: done (3 left) 2026-02-20T11:19:50.042 INFO:tasks.rados.rados.0.trial043.stdout:943: done (2 left) 2026-02-20T11:19:50.042 INFO:tasks.rados.rados.0.trial043.stdout:944: write (excl) oid 24 current snap is 119 2026-02-20T11:19:50.042 INFO:tasks.rados.rados.0.trial043.stdout:940: finishing write tid 2 to trial04342147-32 2026-02-20T11:19:50.042 INFO:tasks.rados.rados.0.trial043.stdout:940: finishing write tid 3 to trial04342147-32 2026-02-20T11:19:50.042 INFO:tasks.rados.rados.0.trial043.stdout:940: finishing write tid 4 to trial04342147-32 2026-02-20T11:19:50.043 INFO:tasks.rados.rados.0.trial043.stdout:940: finishing write tid 5 to trial04342147-32 2026-02-20T11:19:50.043 INFO:tasks.rados.rados.0.trial043.stdout:940: oid 32 updating version 0 to 18 2026-02-20T11:19:50.043 INFO:tasks.rados.rados.0.trial043.stdout:940: oid 32 version 18 is already newer than 18 2026-02-20T11:19:50.043 INFO:tasks.rados.rados.0.trial043.stdout:940: oid 32 version 18 is already newer than 18 2026-02-20T11:19:50.043 INFO:tasks.rados.rados.0.trial043.stdout:940: oid 32 version 18 is already newer than 18 2026-02-20T11:19:50.043 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 32 v 18 (ObjNum 434 snap 119 seq_num 434) dirty exists 2026-02-20T11:19:50.043 INFO:tasks.rados.rados.0.trial043.stdout:940: left oid 32 (ObjNum 434 snap 119 seq_num 434) 2026-02-20T11:19:50.043 INFO:tasks.rados.rados.0.trial043.stdout:941: finishing write tid 1 to trial04342147-28 2026-02-20T11:19:50.043 INFO:tasks.rados.rados.0.trial043.stdout:941: finishing write tid 2 to trial04342147-28 2026-02-20T11:19:50.043 INFO:tasks.rados.rados.0.trial043.stdout:941: finishing write tid 3 to trial04342147-28 2026-02-20T11:19:50.044 INFO:tasks.rados.rados.0.trial043.stdout:941: finishing write tid 4 to trial04342147-28 2026-02-20T11:19:50.044 INFO:tasks.rados.rados.0.trial043.stdout:944: seq_num 437 ranges {544659=715934,1738984=449146,2784436=560223} 2026-02-20T11:19:50.046 INFO:tasks.rados.rados.0.trial043.stdout:944: writing trial04342147-24 from 544659 to 1260593 tid 1 2026-02-20T11:19:50.050 INFO:tasks.rados.rados.0.trial043.stdout:944: writing trial04342147-24 from 1738984 to 2188130 tid 2 2026-02-20T11:19:50.056 INFO:tasks.rados.rados.0.trial043.stdout:944: writing trial04342147-24 from 2784436 to 3344659 tid 3 2026-02-20T11:19:50.056 INFO:tasks.rados.rados.0.trial043.stdout:940: done (2 left) 2026-02-20T11:19:50.056 INFO:tasks.rados.rados.0.trial043.stdout:945: read oid 5 snap -1 2026-02-20T11:19:50.056 INFO:tasks.rados.rados.0.trial043.stdout:945: expect deleted 2026-02-20T11:19:50.056 INFO:tasks.rados.rados.0.trial043.stdout:946: write (excl) oid 32 current snap is 119 2026-02-20T11:19:50.056 INFO:tasks.rados.rados.0.trial043.stdout:941: finishing write tid 5 to trial04342147-28 2026-02-20T11:19:50.056 INFO:tasks.rados.rados.0.trial043.stdout:941: oid 28 updating version 0 to 103 2026-02-20T11:19:50.056 INFO:tasks.rados.rados.0.trial043.stdout:941: oid 28 version 103 is already newer than 103 2026-02-20T11:19:50.056 INFO:tasks.rados.rados.0.trial043.stdout:941: oid 28 version 103 is already newer than 103 2026-02-20T11:19:50.057 INFO:tasks.rados.rados.0.trial043.stdout:941: oid 28 version 103 is already newer than 103 2026-02-20T11:19:50.057 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 28 v 103 (ObjNum 435 snap 119 seq_num 435) dirty exists 2026-02-20T11:19:50.057 INFO:tasks.rados.rados.0.trial043.stdout:941: left oid 28 (ObjNum 435 snap 119 seq_num 435) 2026-02-20T11:19:50.057 INFO:tasks.rados.rados.0.trial043.stdout:946: seq_num 438 ranges {561466=791183,2081113=592804,3095506=265960} 2026-02-20T11:19:50.059 INFO:tasks.rados.rados.0.trial043.stdout:946: writing trial04342147-32 from 561466 to 1352649 tid 1 2026-02-20T11:19:50.065 INFO:tasks.rados.rados.0.trial043.stdout:946: writing trial04342147-32 from 2081113 to 2673917 tid 2 2026-02-20T11:19:50.068 INFO:tasks.rados.rados.0.trial043.stdout:946: writing trial04342147-32 from 3095506 to 3361466 tid 3 2026-02-20T11:19:50.068 INFO:tasks.rados.rados.0.trial043.stdout:941: done (3 left) 2026-02-20T11:19:50.068 INFO:tasks.rados.rados.0.trial043.stdout:947: write oid 48 current snap is 119 2026-02-20T11:19:50.069 INFO:tasks.rados.rados.0.trial043.stdout:947: seq_num 439 ranges {578273=466432,1739595=452815,2722929=655344} 2026-02-20T11:19:50.072 INFO:tasks.rados.rados.0.trial043.stdout:947: writing trial04342147-48 from 578273 to 1044705 tid 1 2026-02-20T11:19:50.077 INFO:tasks.rados.rados.0.trial043.stdout:947: writing trial04342147-48 from 1739595 to 2192410 tid 2 2026-02-20T11:19:50.082 INFO:tasks.rados.rados.0.trial043.stdout:947: writing trial04342147-48 from 2722929 to 3378273 tid 3 2026-02-20T11:19:50.082 INFO:tasks.rados.rados.0.trial043.stdout:948: read oid 28 snap -1 2026-02-20T11:19:50.082 INFO:tasks.rados.rados.0.trial043.stdout:948: expect (ObjNum 435 snap 119 seq_num 435) 2026-02-20T11:19:50.082 INFO:tasks.rados.rados.0.trial043.stdout:949: write (excl) oid 42 current snap is 119 2026-02-20T11:19:50.082 INFO:tasks.rados.rados.0.trial043.stdout:949: seq_num 440 ranges {595080=541681,1798077=596473,3150352=244728} 2026-02-20T11:19:50.086 INFO:tasks.rados.rados.0.trial043.stdout:949: writing trial04342147-42 from 595080 to 1136761 tid 1 2026-02-20T11:19:50.090 INFO:tasks.rados.rados.0.trial043.stdout:949: writing trial04342147-42 from 1798077 to 2394550 tid 2 2026-02-20T11:19:50.094 INFO:tasks.rados.rados.0.trial043.stdout:949: writing trial04342147-42 from 3150352 to 3395080 tid 3 2026-02-20T11:19:50.095 INFO:tasks.rados.rados.0.trial043.stdout:950: snap_remove snap 116 2026-02-20T11:19:50.116 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:19:50.195 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:50.194+0000 7fe5a2b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:50.198 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:50.197+0000 7fe5a2b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:50.224 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:19:50.239 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:19:50.240 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:19:50.887 INFO:tasks.rados.rados.0.trial043.stdout:950: done (6 left) 2026-02-20T11:19:50.887 INFO:tasks.rados.rados.0.trial043.stdout:951: write oid 13 current snap is 119 2026-02-20T11:19:50.887 INFO:tasks.rados.rados.0.trial043.stdout:951: seq_num 441 ranges {611887=733283,1972912=456484,2894128=517759} 2026-02-20T11:19:50.891 INFO:tasks.rados.rados.0.trial043.stdout:951: writing trial04342147-13 from 611887 to 1345170 tid 1 2026-02-20T11:19:50.896 INFO:tasks.rados.rados.0.trial043.stdout:951: writing trial04342147-13 from 1972912 to 2429396 tid 2 2026-02-20T11:19:50.900 INFO:tasks.rados.rados.0.trial043.stdout:951: writing trial04342147-13 from 2894128 to 3411887 tid 3 2026-02-20T11:19:50.901 INFO:tasks.rados.rados.0.trial043.stdout:945: done (6 left) 2026-02-20T11:19:50.901 INFO:tasks.rados.rados.0.trial043.stdout:952: write (excl) oid 40 current snap is 119 2026-02-20T11:19:50.901 INFO:tasks.rados.rados.0.trial043.stdout:946: finishing write tid 1 to trial04342147-32 2026-02-20T11:19:50.901 INFO:tasks.rados.rados.0.trial043.stdout:952: seq_num 442 ranges {628694=408532,1515041=600142,2805198=552112} 2026-02-20T11:19:50.904 INFO:tasks.rados.rados.0.trial043.stdout:952: writing trial04342147-40 from 628694 to 1037226 tid 1 2026-02-20T11:19:50.908 INFO:tasks.rados.rados.0.trial043.stdout:952: writing trial04342147-40 from 1515041 to 2115183 tid 2 2026-02-20T11:19:50.914 INFO:tasks.rados.rados.0.trial043.stdout:952: writing trial04342147-40 from 2805198 to 3357310 tid 3 2026-02-20T11:19:50.914 INFO:tasks.rados.rados.0.trial043.stdout:953: read oid 29 snap -1 2026-02-20T11:19:50.915 INFO:tasks.rados.rados.0.trial043.stdout:953: expect (ObjNum 420 snap 113 seq_num 420) 2026-02-20T11:19:50.915 INFO:tasks.rados.rados.0.trial043.stdout:954: read oid 23 snap -1 2026-02-20T11:19:50.915 INFO:tasks.rados.rados.0.trial043.stdout:954: expect (ObjNum 409 snap 109 seq_num 409) 2026-02-20T11:19:50.915 INFO:tasks.rados.rados.0.trial043.stdout:955: delete oid 43 current snap is 119 2026-02-20T11:19:50.915 INFO:tasks.rados.rados.0.trial043.stdout:947: finishing write tid 1 to trial04342147-48 2026-02-20T11:19:50.915 INFO:tasks.rados.rados.0.trial043.stdout:947: finishing write tid 2 to trial04342147-48 2026-02-20T11:19:50.915 INFO:tasks.rados.rados.0.trial043.stdout:949: finishing write tid 1 to trial04342147-42 2026-02-20T11:19:50.915 INFO:tasks.rados.rados.0.trial043.stdout:946: finishing write tid 2 to trial04342147-32 2026-02-20T11:19:50.915 INFO:tasks.rados.rados.0.trial043.stdout:947: finishing write tid 3 to trial04342147-48 2026-02-20T11:19:50.915 INFO:tasks.rados.rados.0.trial043.stdout:947: finishing write tid 4 to trial04342147-48 2026-02-20T11:19:50.915 INFO:tasks.rados.rados.0.trial043.stdout:947: finishing write tid 5 to trial04342147-48 2026-02-20T11:19:50.916 INFO:tasks.rados.rados.0.trial043.stdout:947: oid 48 updating version 0 to 85 2026-02-20T11:19:50.916 INFO:tasks.rados.rados.0.trial043.stdout:947: oid 48 version 85 is already newer than 85 2026-02-20T11:19:50.916 INFO:tasks.rados.rados.0.trial043.stdout:947: oid 48 version 85 is already newer than 85 2026-02-20T11:19:50.916 INFO:tasks.rados.rados.0.trial043.stdout:947: oid 48 version 85 is already newer than 85 2026-02-20T11:19:50.916 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 48 v 85 (ObjNum 439 snap 119 seq_num 439) dirty exists 2026-02-20T11:19:50.916 INFO:tasks.rados.rados.0.trial043.stdout:947: left oid 48 (ObjNum 439 snap 119 seq_num 439) 2026-02-20T11:19:50.916 INFO:tasks.rados.rados.0.trial043.stdout:949: finishing write tid 2 to trial04342147-42 2026-02-20T11:19:50.916 INFO:tasks.rados.rados.0.trial043.stdout:946: finishing write tid 3 to trial04342147-32 2026-02-20T11:19:50.916 INFO:tasks.rados.rados.0.trial043.stdout:948: expect (ObjNum 435 snap 119 seq_num 435) 2026-02-20T11:19:50.916 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:51.037 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:51.102 INFO:tasks.rados.rados.0.trial043.stdout:949: finishing write tid 3 to trial04342147-42 2026-02-20T11:19:51.102 INFO:tasks.rados.rados.0.trial043.stdout:946: finishing write tid 4 to trial04342147-32 2026-02-20T11:19:51.102 INFO:tasks.rados.rados.0.trial043.stdout:946: finishing write tid 5 to trial04342147-32 2026-02-20T11:19:51.102 INFO:tasks.rados.rados.0.trial043.stdout:946: oid 32 updating version 0 to 22 2026-02-20T11:19:51.103 INFO:tasks.rados.rados.0.trial043.stdout:946: oid 32 version 22 is already newer than 22 2026-02-20T11:19:51.103 INFO:tasks.rados.rados.0.trial043.stdout:946: oid 32 version 22 is already newer than 22 2026-02-20T11:19:51.103 INFO:tasks.rados.rados.0.trial043.stdout:946: oid 32 version 22 is already newer than 19 2026-02-20T11:19:51.103 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 32 v 22 (ObjNum 438 snap 119 seq_num 438) dirty exists 2026-02-20T11:19:51.103 INFO:tasks.rados.rados.0.trial043.stdout:946: left oid 32 (ObjNum 438 snap 119 seq_num 438) 2026-02-20T11:19:51.103 INFO:tasks.rados.rados.0.trial043.stdout:949: finishing write tid 4 to trial04342147-42 2026-02-20T11:19:51.103 INFO:tasks.rados.rados.0.trial043.stdout:946: done (9 left) 2026-02-20T11:19:51.103 INFO:tasks.rados.rados.0.trial043.stdout:947: done (8 left) 2026-02-20T11:19:51.103 INFO:tasks.rados.rados.0.trial043.stdout:948: done (7 left) 2026-02-20T11:19:51.103 INFO:tasks.rados.rados.0.trial043.stdout:955: done (6 left) 2026-02-20T11:19:51.104 INFO:tasks.rados.rados.0.trial043.stdout:956: delete oid 22 current snap is 119 2026-02-20T11:19:51.104 INFO:tasks.rados.rados.0.trial043.stdout:949: finishing write tid 5 to trial04342147-42 2026-02-20T11:19:51.104 INFO:tasks.rados.rados.0.trial043.stdout:949: oid 42 updating version 0 to 135 2026-02-20T11:19:51.104 INFO:tasks.rados.rados.0.trial043.stdout:949: oid 42 updating version 135 to 138 2026-02-20T11:19:51.104 INFO:tasks.rados.rados.0.trial043.stdout:949: oid 42 version 138 is already newer than 138 2026-02-20T11:19:51.104 INFO:tasks.rados.rados.0.trial043.stdout:949: oid 42 version 138 is already newer than 138 2026-02-20T11:19:51.104 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 42 v 138 (ObjNum 440 snap 119 seq_num 440) dirty exists 2026-02-20T11:19:51.104 INFO:tasks.rados.rados.0.trial043.stdout:949: left oid 42 (ObjNum 440 snap 119 seq_num 440) 2026-02-20T11:19:51.104 INFO:tasks.rados.rados.0.trial043.stdout:944: finishing write tid 1 to trial04342147-24 2026-02-20T11:19:51.104 INFO:tasks.rados.rados.0.trial043.stdout:944: finishing write tid 2 to trial04342147-24 2026-02-20T11:19:51.104 INFO:tasks.rados.rados.0.trial043.stdout:944: finishing write tid 3 to trial04342147-24 2026-02-20T11:19:51.104 INFO:tasks.rados.rados.0.trial043.stdout:944: finishing write tid 4 to trial04342147-24 2026-02-20T11:19:51.104 INFO:tasks.rados.rados.0.trial043.stdout:944: finishing write tid 5 to trial04342147-24 2026-02-20T11:19:51.105 INFO:tasks.rados.rados.0.trial043.stdout:944: oid 24 updating version 0 to 73 2026-02-20T11:19:51.105 INFO:tasks.rados.rados.0.trial043.stdout:944: oid 24 version 73 is already newer than 73 2026-02-20T11:19:51.105 INFO:tasks.rados.rados.0.trial043.stdout:944: oid 24 version 73 is already newer than 73 2026-02-20T11:19:51.105 INFO:tasks.rados.rados.0.trial043.stdout:944: oid 24 version 73 is already newer than 70 2026-02-20T11:19:51.105 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 24 v 73 (ObjNum 437 snap 119 seq_num 437) dirty exists 2026-02-20T11:19:51.105 INFO:tasks.rados.rados.0.trial043.stdout:944: left oid 24 (ObjNum 437 snap 119 seq_num 437) 2026-02-20T11:19:51.105 INFO:tasks.rados.rados.0.trial043.stdout:952: finishing write tid 1 to trial04342147-40 2026-02-20T11:19:51.105 INFO:tasks.rados.rados.0.trial043.stdout:952: finishing write tid 2 to trial04342147-40 2026-02-20T11:19:51.105 INFO:tasks.rados.rados.0.trial043.stdout:952: finishing write tid 3 to trial04342147-40 2026-02-20T11:19:51.105 INFO:tasks.rados.rados.0.trial043.stdout:952: finishing write tid 4 to trial04342147-40 2026-02-20T11:19:51.105 INFO:tasks.rados.rados.0.trial043.stdout:952: finishing write tid 5 to trial04342147-40 2026-02-20T11:19:51.105 INFO:tasks.rados.rados.0.trial043.stdout:952: oid 40 updating version 0 to 228 2026-02-20T11:19:51.106 INFO:tasks.rados.rados.0.trial043.stdout:952: oid 40 version 228 is already newer than 228 2026-02-20T11:19:51.106 INFO:tasks.rados.rados.0.trial043.stdout:952: oid 40 version 228 is already newer than 225 2026-02-20T11:19:51.106 INFO:tasks.rados.rados.0.trial043.stdout:952: oid 40 version 228 is already newer than 228 2026-02-20T11:19:51.106 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 40 v 228 (ObjNum 442 snap 119 seq_num 442) dirty exists 2026-02-20T11:19:51.106 INFO:tasks.rados.rados.0.trial043.stdout:952: left oid 40 (ObjNum 442 snap 119 seq_num 442) 2026-02-20T11:19:51.106 INFO:tasks.rados.rados.0.trial043.stdout:951: finishing write tid 1 to trial04342147-13 2026-02-20T11:19:51.106 INFO:tasks.rados.rados.0.trial043.stdout:953: expect (ObjNum 420 snap 113 seq_num 420) 2026-02-20T11:19:51.234 INFO:tasks.rados.rados.0.trial043.stdout:951: finishing write tid 2 to trial04342147-13 2026-02-20T11:19:51.234 INFO:tasks.rados.rados.0.trial043.stdout:951: finishing write tid 3 to trial04342147-13 2026-02-20T11:19:51.234 INFO:tasks.rados.rados.0.trial043.stdout:944: done (6 left) 2026-02-20T11:19:51.234 INFO:tasks.rados.rados.0.trial043.stdout:949: done (5 left) 2026-02-20T11:19:51.234 INFO:tasks.rados.rados.0.trial043.stdout:952: done (4 left) 2026-02-20T11:19:51.234 INFO:tasks.rados.rados.0.trial043.stdout:953: done (3 left) 2026-02-20T11:19:51.235 INFO:tasks.rados.rados.0.trial043.stdout:956: done (2 left) 2026-02-20T11:19:51.235 INFO:tasks.rados.rados.0.trial043.stdout:957: read oid 9 snap -1 2026-02-20T11:19:51.235 INFO:tasks.rados.rados.0.trial043.stdout:957: expect (ObjNum 282 snap 78 seq_num 282) 2026-02-20T11:19:51.235 INFO:tasks.rados.rados.0.trial043.stdout:958: delete oid 4 current snap is 119 2026-02-20T11:19:51.235 INFO:tasks.rados.rados.0.trial043.stdout:951: finishing write tid 4 to trial04342147-13 2026-02-20T11:19:51.235 INFO:tasks.rados.rados.0.trial043.stdout:951: finishing write tid 5 to trial04342147-13 2026-02-20T11:19:51.235 INFO:tasks.rados.rados.0.trial043.stdout:951: oid 13 updating version 0 to 235 2026-02-20T11:19:51.235 INFO:tasks.rados.rados.0.trial043.stdout:951: oid 13 version 235 is already newer than 235 2026-02-20T11:19:51.235 INFO:tasks.rados.rados.0.trial043.stdout:951: oid 13 version 235 is already newer than 235 2026-02-20T11:19:51.235 INFO:tasks.rados.rados.0.trial043.stdout:951: oid 13 version 235 is already newer than 235 2026-02-20T11:19:51.235 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 13 v 235 (ObjNum 441 snap 119 seq_num 441) dirty exists 2026-02-20T11:19:51.235 INFO:tasks.rados.rados.0.trial043.stdout:951: left oid 13 (ObjNum 441 snap 119 seq_num 441) 2026-02-20T11:19:51.236 INFO:tasks.rados.rados.0.trial043.stdout:954: expect (ObjNum 409 snap 109 seq_num 409) 2026-02-20T11:19:51.288 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:51.396 INFO:tasks.rados.rados.0.trial043.stdout:951: done (3 left) 2026-02-20T11:19:51.397 INFO:tasks.rados.rados.0.trial043.stdout:954: done (2 left) 2026-02-20T11:19:51.397 INFO:tasks.rados.rados.0.trial043.stdout:958: done (1 left) 2026-02-20T11:19:51.397 INFO:tasks.rados.rados.0.trial043.stdout:959: write (excl) oid 31 current snap is 119 2026-02-20T11:19:51.397 INFO:tasks.rados.rados.0.trial043.stdout:957: expect (ObjNum 282 snap 78 seq_num 282) 2026-02-20T11:19:51.469 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1076218 2026-02-20T11:19:51.490 INFO:tasks.rados.rados.0.trial043.stdout:959: seq_num 446 ranges {695922=709528,2032616=607480,3198537=297385} 2026-02-20T11:19:51.496 INFO:tasks.rados.rados.0.trial043.stdout:959: writing trial04342147-31 from 695922 to 1405450 tid 1 2026-02-20T11:19:51.501 INFO:tasks.rados.rados.0.trial043.stdout:959: writing trial04342147-31 from 2032616 to 2640096 tid 2 2026-02-20T11:19:51.504 INFO:tasks.rados.rados.0.trial043.stdout:959: writing trial04342147-31 from 3198537 to 3495922 tid 3 2026-02-20T11:19:51.504 INFO:tasks.rados.rados.0.trial043.stdout:957: done (1 left) 2026-02-20T11:19:51.505 INFO:tasks.rados.rados.0.trial043.stdout:960: snap_remove snap 118 2026-02-20T11:19:51.889 INFO:tasks.rados.rados.0.trial043.stdout:960: done (1 left) 2026-02-20T11:19:51.889 INFO:tasks.rados.rados.0.trial043.stdout:961: delete oid 45 current snap is 119 2026-02-20T11:19:51.889 INFO:tasks.rados.rados.0.trial043.stdout:959: finishing write tid 1 to trial04342147-31 2026-02-20T11:19:51.890 INFO:tasks.rados.rados.0.trial043.stdout:959: finishing write tid 2 to trial04342147-31 2026-02-20T11:19:51.890 INFO:tasks.rados.rados.0.trial043.stdout:959: finishing write tid 3 to trial04342147-31 2026-02-20T11:19:51.890 INFO:tasks.rados.rados.0.trial043.stdout:959: finishing write tid 4 to trial04342147-31 2026-02-20T11:19:51.890 INFO:tasks.rados.rados.0.trial043.stdout:959: finishing write tid 5 to trial04342147-31 2026-02-20T11:19:51.890 INFO:tasks.rados.rados.0.trial043.stdout:959: oid 31 updating version 0 to 87 2026-02-20T11:19:51.890 INFO:tasks.rados.rados.0.trial043.stdout:959: oid 31 updating version 87 to 90 2026-02-20T11:19:51.890 INFO:tasks.rados.rados.0.trial043.stdout:959: oid 31 version 90 is already newer than 90 2026-02-20T11:19:51.890 INFO:tasks.rados.rados.0.trial043.stdout:959: oid 31 version 90 is already newer than 90 2026-02-20T11:19:51.890 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 31 v 90 (ObjNum 446 snap 119 seq_num 446) dirty exists 2026-02-20T11:19:51.890 INFO:tasks.rados.rados.0.trial043.stdout:959: left oid 31 (ObjNum 446 snap 119 seq_num 446) 2026-02-20T11:19:51.903 INFO:tasks.rados.rados.0.trial043.stdout:959: done (1 left) 2026-02-20T11:19:51.903 INFO:tasks.rados.rados.0.trial043.stdout:961: done (0 left) 2026-02-20T11:19:51.903 INFO:tasks.rados.rados.0.trial043.stdout:962: write (excl) oid 27 current snap is 119 2026-02-20T11:19:51.904 INFO:tasks.rados.rados.0.trial043.stdout:962: seq_num 448 ranges {729536=460026,1749580=611149,2853383=452450} 2026-02-20T11:19:51.908 INFO:tasks.rados.rados.0.trial043.stdout:962: writing trial04342147-27 from 729536 to 1189562 tid 1 2026-02-20T11:19:51.912 INFO:tasks.rados.rados.0.trial043.stdout:962: writing trial04342147-27 from 1749580 to 2360729 tid 2 2026-02-20T11:19:51.916 INFO:tasks.rados.rados.0.trial043.stdout:962: writing trial04342147-27 from 2853383 to 3305833 tid 3 2026-02-20T11:19:51.916 INFO:tasks.rados.rados.0.trial043.stdout:963: delete oid 49 current snap is 119 2026-02-20T11:19:51.930 INFO:tasks.rados.rados.0.trial043.stdout:962: finishing write tid 1 to trial04342147-27 2026-02-20T11:19:51.948 INFO:tasks.rados.rados.0.trial043.stdout:962: finishing write tid 2 to trial04342147-27 2026-02-20T11:19:51.952 INFO:tasks.rados.rados.0.trial043.stdout:963: done (1 left) 2026-02-20T11:19:51.952 INFO:tasks.rados.rados.0.trial043.stdout:964: delete oid 21 current snap is 119 2026-02-20T11:19:51.955 INFO:tasks.rados.rados.0.trial043.stdout:964: done (1 left) 2026-02-20T11:19:51.955 INFO:tasks.rados.rados.0.trial043.stdout:965: snap_create 2026-02-20T11:19:51.955 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:19:51.955 INFO:tasks.rados.rados.0.trial043.stdout:962: finishing write tid 3 to trial04342147-27 2026-02-20T11:19:51.960 INFO:tasks.rados.rados.0.trial043.stdout:962: finishing write tid 4 to trial04342147-27 2026-02-20T11:19:51.962 INFO:tasks.rados.rados.0.trial043.stdout:962: finishing write tid 5 to trial04342147-27 2026-02-20T11:19:51.963 INFO:tasks.rados.rados.0.trial043.stdout:962: oid 27 updating version 0 to 272 2026-02-20T11:19:51.963 INFO:tasks.rados.rados.0.trial043.stdout:962: oid 27 updating version 272 to 275 2026-02-20T11:19:51.963 INFO:tasks.rados.rados.0.trial043.stdout:962: oid 27 version 275 is already newer than 275 2026-02-20T11:19:51.963 INFO:tasks.rados.rados.0.trial043.stdout:962: oid 27 version 275 is already newer than 275 2026-02-20T11:19:51.963 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 27 v 275 (ObjNum 448 snap 119 seq_num 448) dirty exists 2026-02-20T11:19:51.963 INFO:tasks.rados.rados.0.trial043.stdout:962: left oid 27 (ObjNum 448 snap 119 seq_num 448) 2026-02-20T11:19:51.963 INFO:tasks.rados.rados.0.trial043.stdout:962: done (0 left) 2026-02-20T11:19:52.241 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:19:52.319 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:52.317+0000 7f1ec1c1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:52.321 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:52.321+0000 7f1ec1c1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:52.346 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:19:52.363 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:19:52.363 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:19:52.895 INFO:tasks.rados.rados.0.trial043.stdout:965: done (0 left) 2026-02-20T11:19:52.895 INFO:tasks.rados.rados.0.trial043.stdout:966: snap_create 2026-02-20T11:19:53.019 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~248s 2026-02-20T11:19:53.896 INFO:tasks.rados.rados.0.trial043.stdout:966: done (0 left) 2026-02-20T11:19:53.896 INFO:tasks.rados.rados.0.trial043.stdout:967: read oid 20 snap -1 2026-02-20T11:19:53.896 INFO:tasks.rados.rados.0.trial043.stdout:967: expect (ObjNum 128 snap 17 seq_num 128) 2026-02-20T11:19:53.896 INFO:tasks.rados.rados.0.trial043.stdout:968: read oid 29 snap -1 2026-02-20T11:19:53.896 INFO:tasks.rados.rados.0.trial043.stdout:968: expect (ObjNum 420 snap 113 seq_num 420) 2026-02-20T11:19:53.896 INFO:tasks.rados.rados.0.trial043.stdout:969: read oid 50 snap -1 2026-02-20T11:19:53.896 INFO:tasks.rados.rados.0.trial043.stdout:969: expect (ObjNum 431 snap 118 seq_num 431) 2026-02-20T11:19:53.896 INFO:tasks.rados.rados.0.trial043.stdout:970: snap_create 2026-02-20T11:19:53.897 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:19:53.928 INFO:tasks.rados.rados.0.trial043.stdout:967: expect (ObjNum 128 snap 17 seq_num 128) 2026-02-20T11:19:53.970 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:53.996 INFO:tasks.rados.rados.0.trial043.stdout:969: expect (ObjNum 431 snap 118 seq_num 431) 2026-02-20T11:19:54.003 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2029646 2026-02-20T11:19:54.014 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:54.055 INFO:tasks.rados.rados.0.trial043.stdout:968: expect (ObjNum 420 snap 113 seq_num 420) 2026-02-20T11:19:54.063 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1083509 2026-02-20T11:19:54.137 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:54.182 INFO:tasks.rados.rados.0.trial043.stdout:967: done (2 left) 2026-02-20T11:19:54.182 INFO:tasks.rados.rados.0.trial043.stdout:968: done (1 left) 2026-02-20T11:19:54.182 INFO:tasks.rados.rados.0.trial043.stdout:969: done (0 left) 2026-02-20T11:19:54.364 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:19:54.442 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:54.441+0000 7f55f8393640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:54.445 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:54.445+0000 7f55f8393640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:54.471 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:19:54.486 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:19:54.486 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:19:54.700 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:19:54.771 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:19:54.774 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:19:54.775 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:19:54.901 INFO:tasks.rados.rados.0.trial043.stdout:970: done (0 left) 2026-02-20T11:19:54.901 INFO:tasks.rados.rados.0.trial043.stdout:971: snap_remove snap 115 2026-02-20T11:19:55.905 INFO:tasks.rados.rados.0.trial043.stdout:971: done (0 left) 2026-02-20T11:19:55.906 INFO:tasks.rados.rados.0.trial043.stdout:972: delete oid 16 current snap is 122 2026-02-20T11:19:55.949 INFO:tasks.rados.rados.0.trial043.stdout:972: done (0 left) 2026-02-20T11:19:55.949 INFO:tasks.rados.rados.0.trial043.stdout:973: read oid 21 snap -1 2026-02-20T11:19:55.950 INFO:tasks.rados.rados.0.trial043.stdout:973: expect deleted 2026-02-20T11:19:55.950 INFO:tasks.rados.rados.0.trial043.stdout:974: read oid 14 snap -1 2026-02-20T11:19:55.950 INFO:tasks.rados.rados.0.trial043.stdout:974: expect (ObjNum 403 snap 105 seq_num 403) 2026-02-20T11:19:55.950 INFO:tasks.rados.rados.0.trial043.stdout:975: write (excl) oid 47 current snap is 122 2026-02-20T11:19:55.950 INFO:tasks.rados.rados.0.trial043.stdout:975: seq_num 452 ranges {796764=477375,1983508=618487,3363075=233689} 2026-02-20T11:19:55.954 INFO:tasks.rados.rados.0.trial043.stdout:975: writing trial04342147-47 from 796764 to 1274139 tid 1 2026-02-20T11:19:55.959 INFO:tasks.rados.rados.0.trial043.stdout:975: writing trial04342147-47 from 1983508 to 2601995 tid 2 2026-02-20T11:19:55.963 INFO:tasks.rados.rados.0.trial043.stdout:975: writing trial04342147-47 from 3363075 to 3596764 tid 3 2026-02-20T11:19:55.963 INFO:tasks.rados.rados.0.trial043.stdout:976: read oid 41 snap -1 2026-02-20T11:19:55.964 INFO:tasks.rados.rados.0.trial043.stdout:976: expect (ObjNum 357 snap 96 seq_num 357) 2026-02-20T11:19:55.964 INFO:tasks.rados.rados.0.trial043.stdout:977: delete oid 4 current snap is 122 2026-02-20T11:19:55.969 INFO:tasks.rados.rados.0.trial043.stdout:973: done (4 left) 2026-02-20T11:19:55.969 INFO:tasks.rados.rados.0.trial043.stdout:977: done (3 left) 2026-02-20T11:19:55.969 INFO:tasks.rados.rados.0.trial043.stdout:978: snap_remove snap 121 2026-02-20T11:19:56.487 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:19:56.564 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:56.563+0000 7f6a785ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:56.566 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:56.566+0000 7f6a785ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:56.592 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:19:56.609 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:19:56.610 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:19:56.909 INFO:tasks.rados.rados.0.trial043.stdout:978: done (3 left) 2026-02-20T11:19:56.909 INFO:tasks.rados.rados.0.trial043.stdout:979: snap_create 2026-02-20T11:19:56.909 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:19:56.909 INFO:tasks.rados.rados.0.trial043.stdout:974: expect (ObjNum 403 snap 105 seq_num 403) 2026-02-20T11:19:56.917 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1771145 2026-02-20T11:19:56.935 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 675989 2026-02-20T11:19:56.977 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2613048 2026-02-20T11:19:56.982 INFO:tasks.rados.rados.0.trial043.stdout:974: done (2 left) 2026-02-20T11:19:56.982 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:19:56.982 INFO:tasks.rados.rados.0.trial043.stdout:975: finishing write tid 1 to trial04342147-47 2026-02-20T11:19:56.982 INFO:tasks.rados.rados.0.trial043.stdout:976: expect (ObjNum 357 snap 96 seq_num 357) 2026-02-20T11:19:57.042 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 442182 2026-02-20T11:19:57.068 INFO:tasks.rados.rados.0.trial043.stdout:975: finishing write tid 2 to trial04342147-47 2026-02-20T11:19:57.068 INFO:tasks.rados.rados.0.trial043.stdout:976: done (1 left) 2026-02-20T11:19:57.068 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:19:57.068 INFO:tasks.rados.rados.0.trial043.stdout:975: finishing write tid 3 to trial04342147-47 2026-02-20T11:19:57.068 INFO:tasks.rados.rados.0.trial043.stdout:975: finishing write tid 4 to trial04342147-47 2026-02-20T11:19:57.068 INFO:tasks.rados.rados.0.trial043.stdout:975: finishing write tid 5 to trial04342147-47 2026-02-20T11:19:57.068 INFO:tasks.rados.rados.0.trial043.stdout:975: oid 47 updating version 0 to 143 2026-02-20T11:19:57.068 INFO:tasks.rados.rados.0.trial043.stdout:975: oid 47 version 143 is already newer than 140 2026-02-20T11:19:57.068 INFO:tasks.rados.rados.0.trial043.stdout:975: oid 47 version 143 is already newer than 143 2026-02-20T11:19:57.069 INFO:tasks.rados.rados.0.trial043.stdout:975: oid 47 version 143 is already newer than 143 2026-02-20T11:19:57.069 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 47 v 143 (ObjNum 452 snap 122 seq_num 452) dirty exists 2026-02-20T11:19:57.069 INFO:tasks.rados.rados.0.trial043.stdout:975: left oid 47 (ObjNum 452 snap 122 seq_num 452) 2026-02-20T11:19:57.069 INFO:tasks.rados.rados.0.trial043.stdout:975: done (0 left) 2026-02-20T11:19:57.914 INFO:tasks.rados.rados.0.trial043.stdout:979: done (0 left) 2026-02-20T11:19:57.914 INFO:tasks.rados.rados.0.trial043.stdout:980: read oid 22 snap -1 2026-02-20T11:19:57.914 INFO:tasks.rados.rados.0.trial043.stdout:980: expect deleted 2026-02-20T11:19:57.914 INFO:tasks.rados.rados.0.trial043.stdout:981: write oid 1 current snap is 123 2026-02-20T11:19:57.914 INFO:tasks.rados.rados.0.trial043.stdout:981: seq_num 454 ranges {430378=627873,1700472=622156,3017921=612457} 2026-02-20T11:19:57.918 INFO:tasks.rados.rados.0.trial043.stdout:981: writing trial04342147-1 from 430378 to 1058251 tid 1 2026-02-20T11:19:57.923 INFO:tasks.rados.rados.0.trial043.stdout:981: writing trial04342147-1 from 1700472 to 2322628 tid 2 2026-02-20T11:19:57.927 INFO:tasks.rados.rados.0.trial043.stdout:981: writing trial04342147-1 from 3017921 to 3630378 tid 3 2026-02-20T11:19:57.928 INFO:tasks.rados.rados.0.trial043.stdout:982: snap_remove snap 120 2026-02-20T11:19:58.523 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~253s 2026-02-20T11:19:58.611 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:19:58.689 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:58.687+0000 7fb054f74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:58.691 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:19:58.691+0000 7fb054f74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:19:58.717 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:19:58.734 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:19:58.735 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:19:58.779 INFO:tasks.rados.rados.0.trial043.stdout:982: done (2 left) 2026-02-20T11:19:58.780 INFO:tasks.rados.rados.0.trial043.stdout:983: write oid 31 current snap is 123 2026-02-20T11:19:58.780 INFO:tasks.rados.rados.0.trial043.stdout:983: seq_num 455 ranges {447185=703122,1642601=765814,2928991=447707} 2026-02-20T11:19:58.784 INFO:tasks.rados.rados.0.trial043.stdout:983: writing trial04342147-31 from 447185 to 1150307 tid 1 2026-02-20T11:19:58.789 INFO:tasks.rados.rados.0.trial043.stdout:983: writing trial04342147-31 from 1642601 to 2408415 tid 2 2026-02-20T11:19:58.793 INFO:tasks.rados.rados.0.trial043.stdout:983: writing trial04342147-31 from 2928991 to 3376698 tid 3 2026-02-20T11:19:58.793 INFO:tasks.rados.rados.0.trial043.stdout:984: snap_remove snap 117 2026-02-20T11:19:59.776 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:19:59.781 INFO:tasks.rados.rados.0.trial043.stdout:984: done (3 left) 2026-02-20T11:19:59.781 INFO:tasks.rados.rados.0.trial043.stdout:985: read oid 44 snap -1 2026-02-20T11:19:59.782 INFO:tasks.rados.rados.0.trial043.stdout:985: expect (ObjNum 427 snap 116 seq_num 427) 2026-02-20T11:19:59.782 INFO:tasks.rados.rados.0.trial043.stdout:986: snap_create 2026-02-20T11:19:59.782 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 4 2026-02-20T11:19:59.782 INFO:tasks.rados.rados.0.trial043.stdout:981: finishing write tid 1 to trial04342147-1 2026-02-20T11:19:59.782 INFO:tasks.rados.rados.0.trial043.stdout:981: finishing write tid 2 to trial04342147-1 2026-02-20T11:19:59.782 INFO:tasks.rados.rados.0.trial043.stdout:981: finishing write tid 3 to trial04342147-1 2026-02-20T11:19:59.782 INFO:tasks.rados.rados.0.trial043.stdout:980: done (3 left) 2026-02-20T11:19:59.782 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:19:59.782 INFO:tasks.rados.rados.0.trial043.stdout:981: finishing write tid 4 to trial04342147-1 2026-02-20T11:19:59.782 INFO:tasks.rados.rados.0.trial043.stdout:981: finishing write tid 5 to trial04342147-1 2026-02-20T11:19:59.782 INFO:tasks.rados.rados.0.trial043.stdout:981: oid 1 updating version 0 to 108 2026-02-20T11:19:59.783 INFO:tasks.rados.rados.0.trial043.stdout:981: oid 1 version 108 is already newer than 108 2026-02-20T11:19:59.783 INFO:tasks.rados.rados.0.trial043.stdout:981: oid 1 version 108 is already newer than 108 2026-02-20T11:19:59.783 INFO:tasks.rados.rados.0.trial043.stdout:981: oid 1 version 108 is already newer than 108 2026-02-20T11:19:59.783 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 1 v 108 (ObjNum 454 snap 123 seq_num 454) dirty exists 2026-02-20T11:19:59.783 INFO:tasks.rados.rados.0.trial043.stdout:981: left oid 1 (ObjNum 454 snap 123 seq_num 454) 2026-02-20T11:19:59.783 INFO:tasks.rados.rados.0.trial043.stdout:981: done (2 left) 2026-02-20T11:19:59.783 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:19:59.783 INFO:tasks.rados.rados.0.trial043.stdout:983: finishing write tid 1 to trial04342147-31 2026-02-20T11:19:59.783 INFO:tasks.rados.rados.0.trial043.stdout:983: finishing write tid 2 to trial04342147-31 2026-02-20T11:19:59.783 INFO:tasks.rados.rados.0.trial043.stdout:983: finishing write tid 3 to trial04342147-31 2026-02-20T11:19:59.783 INFO:tasks.rados.rados.0.trial043.stdout:983: finishing write tid 4 to trial04342147-31 2026-02-20T11:19:59.783 INFO:tasks.rados.rados.0.trial043.stdout:983: finishing write tid 5 to trial04342147-31 2026-02-20T11:19:59.784 INFO:tasks.rados.rados.0.trial043.stdout:983: oid 31 updating version 0 to 97 2026-02-20T11:19:59.784 INFO:tasks.rados.rados.0.trial043.stdout:983: oid 31 version 97 is already newer than 97 2026-02-20T11:19:59.784 INFO:tasks.rados.rados.0.trial043.stdout:983: oid 31 version 97 is already newer than 97 2026-02-20T11:19:59.784 INFO:tasks.rados.rados.0.trial043.stdout:983: oid 31 version 97 is already newer than 97 2026-02-20T11:19:59.784 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 31 v 97 (ObjNum 455 snap 123 seq_num 455) dirty exists 2026-02-20T11:19:59.784 INFO:tasks.rados.rados.0.trial043.stdout:983: left oid 31 (ObjNum 455 snap 123 seq_num 455) 2026-02-20T11:19:59.784 INFO:tasks.rados.rados.0.trial043.stdout:983: done (1 left) 2026-02-20T11:19:59.784 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:19:59.812 INFO:tasks.rados.rados.0.trial043.stdout:985: expect (ObjNum 427 snap 116 seq_num 427) 2026-02-20T11:19:59.849 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:19:59.851 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:19:59.852 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:19:59.860 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1992151 2026-02-20T11:19:59.877 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:19:59.914 INFO:tasks.rados.rados.0.trial043.stdout:985: done (0 left) 2026-02-20T11:20:00.735 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:20:00.782 INFO:tasks.rados.rados.0.trial043.stdout:986: done (0 left) 2026-02-20T11:20:00.783 INFO:tasks.rados.rados.0.trial043.stdout:987: write oid 26 current snap is 124 2026-02-20T11:20:00.783 INFO:tasks.rados.rados.0.trial043.stdout:987: seq_num 456 ranges {463992=778371,1701083=625825,2956414=658979} 2026-02-20T11:20:00.787 INFO:tasks.rados.rados.0.trial043.stdout:987: writing trial04342147-26 from 463992 to 1242363 tid 1 2026-02-20T11:20:00.791 INFO:tasks.rados.rados.0.trial043.stdout:987: writing trial04342147-26 from 1701083 to 2326908 tid 2 2026-02-20T11:20:00.797 INFO:tasks.rados.rados.0.trial043.stdout:987: writing trial04342147-26 from 2956414 to 3615393 tid 3 2026-02-20T11:20:00.797 INFO:tasks.rados.rados.0.trial043.stdout:988: delete oid 15 current snap is 124 2026-02-20T11:20:00.818 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:00.817+0000 7f9c44de4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:00.820 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:00.821+0000 7f9c44de4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:00.835 INFO:tasks.rados.rados.0.trial043.stdout:987: finishing write tid 1 to trial04342147-26 2026-02-20T11:20:00.848 INFO:tasks.rados.rados.0.trial043.stdout:987: finishing write tid 2 to trial04342147-26 2026-02-20T11:20:00.860 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:20:00.861 INFO:tasks.rados.rados.0.trial043.stdout:987: finishing write tid 3 to trial04342147-26 2026-02-20T11:20:00.868 INFO:tasks.rados.rados.0.trial043.stdout:987: finishing write tid 4 to trial04342147-26 2026-02-20T11:20:00.870 INFO:tasks.rados.rados.0.trial043.stdout:987: finishing write tid 5 to trial04342147-26 2026-02-20T11:20:00.870 INFO:tasks.rados.rados.0.trial043.stdout:987: oid 26 updating version 0 to 280 2026-02-20T11:20:00.870 INFO:tasks.rados.rados.0.trial043.stdout:987: oid 26 version 280 is already newer than 280 2026-02-20T11:20:00.871 INFO:tasks.rados.rados.0.trial043.stdout:987: oid 26 version 280 is already newer than 280 2026-02-20T11:20:00.871 INFO:tasks.rados.rados.0.trial043.stdout:987: oid 26 version 280 is already newer than 280 2026-02-20T11:20:00.871 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 26 v 280 (ObjNum 456 snap 124 seq_num 456) dirty exists 2026-02-20T11:20:00.871 INFO:tasks.rados.rados.0.trial043.stdout:987: left oid 26 (ObjNum 456 snap 124 seq_num 456) 2026-02-20T11:20:00.875 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:20:00.876 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:20:00.885 INFO:tasks.rados.rados.0.trial043.stdout:987: done (1 left) 2026-02-20T11:20:00.886 INFO:tasks.rados.rados.0.trial043.stdout:988: done (0 left) 2026-02-20T11:20:00.886 INFO:tasks.rados.rados.0.trial043.stdout:989: delete oid 30 current snap is 124 2026-02-20T11:20:00.921 INFO:tasks.rados.rados.0.trial043.stdout:989: done (0 left) 2026-02-20T11:20:00.921 INFO:tasks.rados.rados.0.trial043.stdout:990: read oid 36 snap -1 2026-02-20T11:20:00.921 INFO:tasks.rados.rados.0.trial043.stdout:990: expect deleted 2026-02-20T11:20:00.921 INFO:tasks.rados.rados.0.trial043.stdout:991: delete oid 16 current snap is 124 2026-02-20T11:20:00.924 INFO:tasks.rados.rados.0.trial043.stdout:991: done (1 left) 2026-02-20T11:20:00.924 INFO:tasks.rados.rados.0.trial043.stdout:992: read oid 49 snap 122 2026-02-20T11:20:00.924 INFO:tasks.rados.rados.0.trial043.stdout:992: expect deleted 2026-02-20T11:20:00.924 INFO:tasks.rados.rados.0.trial043.stdout:993: read oid 39 snap -1 2026-02-20T11:20:00.924 INFO:tasks.rados.rados.0.trial043.stdout:993: expect (ObjNum 425 snap 116 seq_num 425) 2026-02-20T11:20:00.924 INFO:tasks.rados.rados.0.trial043.stdout:994: write (excl) oid 9 current snap is 124 2026-02-20T11:20:00.924 INFO:tasks.rados.rados.0.trial043.stdout:994: seq_num 460 ranges {531220=795720,1935011=516810,3066106=420420} 2026-02-20T11:20:00.929 INFO:tasks.rados.rados.0.trial043.stdout:994: writing trial04342147-9 from 531220 to 1326940 tid 1 2026-02-20T11:20:00.933 INFO:tasks.rados.rados.0.trial043.stdout:994: writing trial04342147-9 from 1935011 to 2451821 tid 2 2026-02-20T11:20:00.937 INFO:tasks.rados.rados.0.trial043.stdout:994: writing trial04342147-9 from 3066106 to 3486526 tid 3 2026-02-20T11:20:00.938 INFO:tasks.rados.rados.0.trial043.stdout:995: read oid 43 snap -1 2026-02-20T11:20:00.938 INFO:tasks.rados.rados.0.trial043.stdout:995: expect deleted 2026-02-20T11:20:00.938 INFO:tasks.rados.rados.0.trial043.stdout:996: snap_remove snap 119 2026-02-20T11:20:01.786 INFO:tasks.rados.rados.0.trial043.stdout:996: done (5 left) 2026-02-20T11:20:01.787 INFO:tasks.rados.rados.0.trial043.stdout:997: read oid 40 snap 123 2026-02-20T11:20:01.787 INFO:tasks.rados.rados.0.trial043.stdout:997: expect (ObjNum 442 snap 119 seq_num 442) 2026-02-20T11:20:01.787 INFO:tasks.rados.rados.0.trial043.stdout:998: read oid 4 snap -1 2026-02-20T11:20:01.787 INFO:tasks.rados.rados.0.trial043.stdout:998: expect deleted 2026-02-20T11:20:01.787 INFO:tasks.rados.rados.0.trial043.stdout:999: read oid 3 snap 123 2026-02-20T11:20:01.787 INFO:tasks.rados.rados.0.trial043.stdout:999: expect deleted 2026-02-20T11:20:01.787 INFO:tasks.rados.rados.0.trial043.stdout:1000: delete oid 5 current snap is 124 2026-02-20T11:20:01.787 INFO:tasks.rados.rados.0.trial043.stdout:993: expect (ObjNum 425 snap 116 seq_num 425) 2026-02-20T11:20:01.804 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1475163 2026-02-20T11:20:01.818 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:20:01.845 INFO:tasks.rados.rados.0.trial043.stdout:994: finishing write tid 1 to trial04342147-9 2026-02-20T11:20:01.845 INFO:tasks.rados.rados.0.trial043.stdout:994: finishing write tid 2 to trial04342147-9 2026-02-20T11:20:01.846 INFO:tasks.rados.rados.0.trial043.stdout:990: done (8 left) 2026-02-20T11:20:01.846 INFO:tasks.rados.rados.0.trial043.stdout:992: done (7 left) 2026-02-20T11:20:01.846 INFO:tasks.rados.rados.0.trial043.stdout:993: done (6 left) 2026-02-20T11:20:01.846 INFO:tasks.rados.rados.0.trial043.stdout:995: done (5 left) 2026-02-20T11:20:01.846 INFO:tasks.rados.rados.0.trial043.stdout:1000: done (4 left) 2026-02-20T11:20:01.846 INFO:tasks.rados.rados.0.trial043.stdout:1001: write oid 44 current snap is 124 2026-02-20T11:20:01.846 INFO:tasks.rados.rados.0.trial043.stdout:994: finishing write tid 3 to trial04342147-9 2026-02-20T11:20:01.846 INFO:tasks.rados.rados.0.trial043.stdout:994: finishing write tid 4 to trial04342147-9 2026-02-20T11:20:01.846 INFO:tasks.rados.rados.0.trial043.stdout:994: finishing write tid 5 to trial04342147-9 2026-02-20T11:20:01.846 INFO:tasks.rados.rados.0.trial043.stdout:994: oid 9 updating version 0 to 233 2026-02-20T11:20:01.846 INFO:tasks.rados.rados.0.trial043.stdout:994: oid 9 version 233 is already newer than 233 2026-02-20T11:20:01.846 INFO:tasks.rados.rados.0.trial043.stdout:994: oid 9 version 233 is already newer than 233 2026-02-20T11:20:01.846 INFO:tasks.rados.rados.0.trial043.stdout:994: oid 9 version 233 is already newer than 230 2026-02-20T11:20:01.846 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 9 v 233 (ObjNum 460 snap 124 seq_num 460) dirty exists 2026-02-20T11:20:01.846 INFO:tasks.rados.rados.0.trial043.stdout:994: left oid 9 (ObjNum 460 snap 124 seq_num 460) 2026-02-20T11:20:01.846 INFO:tasks.rados.rados.0.trial043.stdout:1001: seq_num 462 ranges {564834=546218,1651975=520479,2720952=559317} 2026-02-20T11:20:01.850 INFO:tasks.rados.rados.0.trial043.stdout:1001: writing trial04342147-44 from 564834 to 1111052 tid 1 2026-02-20T11:20:01.854 INFO:tasks.rados.rados.0.trial043.stdout:1001: writing trial04342147-44 from 1651975 to 2172454 tid 2 2026-02-20T11:20:01.858 INFO:tasks.rados.rados.0.trial043.stdout:1001: writing trial04342147-44 from 2720952 to 3280269 tid 3 2026-02-20T11:20:01.859 INFO:tasks.rados.rados.0.trial043.stdout:994: done (4 left) 2026-02-20T11:20:01.859 INFO:tasks.rados.rados.0.trial043.stdout:1002: snap_remove snap 122 2026-02-20T11:20:02.791 INFO:tasks.rados.rados.0.trial043.stdout:1002: done (4 left) 2026-02-20T11:20:02.791 INFO:tasks.rados.rados.0.trial043.stdout:1003: write (excl) oid 14 current snap is 124 2026-02-20T11:20:02.792 INFO:tasks.rados.rados.0.trial043.stdout:1003: seq_num 463 ranges {581641=621467,1994104=780490,3432022=349619} 2026-02-20T11:20:02.795 INFO:tasks.rados.rados.0.trial043.stdout:1003: writing trial04342147-14 from 581641 to 1203108 tid 1 2026-02-20T11:20:02.802 INFO:tasks.rados.rados.0.trial043.stdout:1003: writing trial04342147-14 from 1994104 to 2774594 tid 2 2026-02-20T11:20:02.806 INFO:tasks.rados.rados.0.trial043.stdout:1003: writing trial04342147-14 from 3432022 to 3781641 tid 3 2026-02-20T11:20:02.806 INFO:tasks.rados.rados.0.trial043.stdout:1004: delete oid 9 current snap is 124 2026-02-20T11:20:02.806 INFO:tasks.rados.rados.0.trial043.stdout:997: expect (ObjNum 442 snap 119 seq_num 442) 2026-02-20T11:20:02.877 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:20:02.880 INFO:tasks.rados.rados.0.trial043.stdout:1001: finishing write tid 1 to trial04342147-44 2026-02-20T11:20:02.881 INFO:tasks.rados.rados.0.trial043.stdout:1001: finishing write tid 2 to trial04342147-44 2026-02-20T11:20:02.881 INFO:tasks.rados.rados.0.trial043.stdout:997: done (5 left) 2026-02-20T11:20:02.881 INFO:tasks.rados.rados.0.trial043.stdout:998: done (4 left) 2026-02-20T11:20:02.881 INFO:tasks.rados.rados.0.trial043.stdout:999: done (3 left) 2026-02-20T11:20:02.881 INFO:tasks.rados.rados.0.trial043.stdout:1004: done (2 left) 2026-02-20T11:20:02.881 INFO:tasks.rados.rados.0.trial043.stdout:1005: delete oid 25 current snap is 124 2026-02-20T11:20:02.881 INFO:tasks.rados.rados.0.trial043.stdout:1001: finishing write tid 3 to trial04342147-44 2026-02-20T11:20:02.881 INFO:tasks.rados.rados.0.trial043.stdout:1001: finishing write tid 4 to trial04342147-44 2026-02-20T11:20:02.881 INFO:tasks.rados.rados.0.trial043.stdout:1001: finishing write tid 5 to trial04342147-44 2026-02-20T11:20:02.881 INFO:tasks.rados.rados.0.trial043.stdout:1001: oid 44 updating version 0 to 148 2026-02-20T11:20:02.882 INFO:tasks.rados.rados.0.trial043.stdout:1001: oid 44 version 148 is already newer than 148 2026-02-20T11:20:02.882 INFO:tasks.rados.rados.0.trial043.stdout:1001: oid 44 version 148 is already newer than 148 2026-02-20T11:20:02.882 INFO:tasks.rados.rados.0.trial043.stdout:1001: oid 44 version 148 is already newer than 148 2026-02-20T11:20:02.882 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 44 v 148 (ObjNum 462 snap 124 seq_num 462) dirty exists 2026-02-20T11:20:02.882 INFO:tasks.rados.rados.0.trial043.stdout:1001: left oid 44 (ObjNum 462 snap 124 seq_num 462) 2026-02-20T11:20:02.882 INFO:tasks.rados.rados.0.trial043.stdout:1003: finishing write tid 1 to trial04342147-14 2026-02-20T11:20:02.882 INFO:tasks.rados.rados.0.trial043.stdout:1003: finishing write tid 2 to trial04342147-14 2026-02-20T11:20:02.882 INFO:tasks.rados.rados.0.trial043.stdout:1003: finishing write tid 3 to trial04342147-14 2026-02-20T11:20:02.882 INFO:tasks.rados.rados.0.trial043.stdout:1003: finishing write tid 4 to trial04342147-14 2026-02-20T11:20:02.882 INFO:tasks.rados.rados.0.trial043.stdout:1003: finishing write tid 5 to trial04342147-14 2026-02-20T11:20:02.883 INFO:tasks.rados.rados.0.trial043.stdout:1003: oid 14 updating version 0 to 287 2026-02-20T11:20:02.883 INFO:tasks.rados.rados.0.trial043.stdout:1003: oid 14 version 287 is already newer than 284 2026-02-20T11:20:02.883 INFO:tasks.rados.rados.0.trial043.stdout:1003: oid 14 version 287 is already newer than 287 2026-02-20T11:20:02.883 INFO:tasks.rados.rados.0.trial043.stdout:1003: oid 14 version 287 is already newer than 287 2026-02-20T11:20:02.883 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 14 v 287 (ObjNum 463 snap 124 seq_num 463) dirty exists 2026-02-20T11:20:02.883 INFO:tasks.rados.rados.0.trial043.stdout:1003: left oid 14 (ObjNum 463 snap 124 seq_num 463) 2026-02-20T11:20:02.903 INFO:tasks.rados.rados.0.trial043.stdout:1001: done (2 left) 2026-02-20T11:20:02.903 INFO:tasks.rados.rados.0.trial043.stdout:1003: done (1 left) 2026-02-20T11:20:02.903 INFO:tasks.rados.rados.0.trial043.stdout:1005: done (0 left) 2026-02-20T11:20:02.904 INFO:tasks.rados.rados.0.trial043.stdout:1006: write oid 30 current snap is 124 2026-02-20T11:20:02.904 INFO:tasks.rados.rados.0.trial043.stdout:1006: seq_num 466 ranges {632062=563567,1885903=527817,2830644=720758} 2026-02-20T11:20:02.908 INFO:tasks.rados.rados.0.trial043.stdout:1006: writing trial04342147-30 from 632062 to 1195629 tid 1 2026-02-20T11:20:02.912 INFO:tasks.rados.rados.0.trial043.stdout:1006: writing trial04342147-30 from 1885903 to 2413720 tid 2 2026-02-20T11:20:02.917 INFO:tasks.rados.rados.0.trial043.stdout:1006: writing trial04342147-30 from 2830644 to 3551402 tid 3 2026-02-20T11:20:02.917 INFO:tasks.rados.rados.0.trial043.stdout:1007: snap_create 2026-02-20T11:20:02.917 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:20:02.926 INFO:tasks.rados.rados.0.trial043.stdout:1006: finishing write tid 1 to trial04342147-30 2026-02-20T11:20:02.936 INFO:tasks.rados.rados.0.trial043.stdout:1006: finishing write tid 2 to trial04342147-30 2026-02-20T11:20:02.943 INFO:tasks.rados.rados.0.trial043.stdout:1006: finishing write tid 3 to trial04342147-30 2026-02-20T11:20:02.951 INFO:tasks.rados.rados.0.trial043.stdout:1006: finishing write tid 4 to trial04342147-30 2026-02-20T11:20:02.953 INFO:tasks.rados.rados.0.trial043.stdout:1006: finishing write tid 5 to trial04342147-30 2026-02-20T11:20:02.953 INFO:tasks.rados.rados.0.trial043.stdout:1006: oid 30 updating version 0 to 49 2026-02-20T11:20:02.953 INFO:tasks.rados.rados.0.trial043.stdout:1006: oid 30 version 49 is already newer than 49 2026-02-20T11:20:02.953 INFO:tasks.rados.rados.0.trial043.stdout:1006: oid 30 version 49 is already newer than 49 2026-02-20T11:20:02.953 INFO:tasks.rados.rados.0.trial043.stdout:1006: oid 30 version 49 is already newer than 49 2026-02-20T11:20:02.953 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 30 v 49 (ObjNum 466 snap 124 seq_num 466) dirty exists 2026-02-20T11:20:02.954 INFO:tasks.rados.rados.0.trial043.stdout:1006: left oid 30 (ObjNum 466 snap 124 seq_num 466) 2026-02-20T11:20:02.954 INFO:tasks.rados.rados.0.trial043.stdout:1006: done (0 left) 2026-02-20T11:20:02.955 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:02.954+0000 7fc211d74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:02.957 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:02.958+0000 7fc211d74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:02.983 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:20:02.998 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:20:02.999 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:20:03.792 INFO:tasks.rados.rados.0.trial043.stdout:1007: done (0 left) 2026-02-20T11:20:03.792 INFO:tasks.rados.rados.0.trial043.stdout:1008: delete oid 16 current snap is 125 2026-02-20T11:20:03.821 INFO:tasks.rados.rados.0.trial043.stdout:1008: done (0 left) 2026-02-20T11:20:03.821 INFO:tasks.rados.rados.0.trial043.stdout:1009: snap_create 2026-02-20T11:20:04.027 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~259s 2026-02-20T11:20:04.797 INFO:tasks.rados.rados.0.trial043.stdout:1009: done (0 left) 2026-02-20T11:20:04.797 INFO:tasks.rados.rados.0.trial043.stdout:1010: write (excl) oid 50 current snap is 126 2026-02-20T11:20:04.797 INFO:tasks.rados.rados.0.trial043.stdout:1010: seq_num 468 ranges {665676=714065,1886514=531486,3169137=696539} 2026-02-20T11:20:04.802 INFO:tasks.rados.rados.0.trial043.stdout:1010: writing trial04342147-50 from 665676 to 1379741 tid 1 2026-02-20T11:20:04.806 INFO:tasks.rados.rados.0.trial043.stdout:1010: writing trial04342147-50 from 1886514 to 2418000 tid 2 2026-02-20T11:20:04.812 INFO:tasks.rados.rados.0.trial043.stdout:1010: writing trial04342147-50 from 3169137 to 3865676 tid 3 2026-02-20T11:20:04.812 INFO:tasks.rados.rados.0.trial043.stdout:1011: read oid 37 snap 124 2026-02-20T11:20:04.812 INFO:tasks.rados.rados.0.trial043.stdout:1011: expect deleted 2026-02-20T11:20:04.813 INFO:tasks.rados.rados.0.trial043.stdout:1012: read oid 42 snap -1 2026-02-20T11:20:04.813 INFO:tasks.rados.rados.0.trial043.stdout:1012: expect (ObjNum 440 snap 119 seq_num 440) 2026-02-20T11:20:04.813 INFO:tasks.rados.rados.0.trial043.stdout:1013: read oid 47 snap -1 2026-02-20T11:20:04.813 INFO:tasks.rados.rados.0.trial043.stdout:1013: expect (ObjNum 452 snap 122 seq_num 452) 2026-02-20T11:20:04.813 INFO:tasks.rados.rados.0.trial043.stdout:1014: write oid 5 current snap is 126 2026-02-20T11:20:04.813 INFO:tasks.rados.rados.0.trial043.stdout:1014: seq_num 469 ranges {682483=789314,1944996=791497,3196560=554574} 2026-02-20T11:20:04.817 INFO:tasks.rados.rados.0.trial043.stdout:1014: writing trial04342147-5 from 682483 to 1471797 tid 1 2026-02-20T11:20:04.822 INFO:tasks.rados.rados.0.trial043.stdout:1014: writing trial04342147-5 from 1944996 to 2736493 tid 2 2026-02-20T11:20:04.826 INFO:tasks.rados.rados.0.trial043.stdout:1014: writing trial04342147-5 from 3196560 to 3751134 tid 3 2026-02-20T11:20:04.826 INFO:tasks.rados.rados.0.trial043.stdout:1015: write oid 27 current snap is 126 2026-02-20T11:20:04.826 INFO:tasks.rados.rados.0.trial043.stdout:1015: seq_num 470 ranges {699290=464563,1603478=535155,2823983=765846} 2026-02-20T11:20:04.830 INFO:tasks.rados.rados.0.trial043.stdout:1015: writing trial04342147-27 from 699290 to 1163853 tid 1 2026-02-20T11:20:04.834 INFO:tasks.rados.rados.0.trial043.stdout:1015: writing trial04342147-27 from 1603478 to 2138633 tid 2 2026-02-20T11:20:04.840 INFO:tasks.rados.rados.0.trial043.stdout:1015: writing trial04342147-27 from 2823983 to 3589829 tid 3 2026-02-20T11:20:04.840 INFO:tasks.rados.rados.0.trial043.stdout:1016: snap_remove snap 125 2026-02-20T11:20:04.853 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:20:04.928 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:20:04.931 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:20:04.932 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:20:04.999 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:20:05.073 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:05.072+0000 7f9bd49cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:05.076 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:05.076+0000 7f9bd49cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:05.102 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:20:05.120 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:20:05.120 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:20:05.800 INFO:tasks.rados.rados.0.trial043.stdout:1016: done (6 left) 2026-02-20T11:20:05.800 INFO:tasks.rados.rados.0.trial043.stdout:1017: read oid 15 snap -1 2026-02-20T11:20:05.800 INFO:tasks.rados.rados.0.trial043.stdout:1017: expect deleted 2026-02-20T11:20:05.801 INFO:tasks.rados.rados.0.trial043.stdout:1018: snap_create 2026-02-20T11:20:05.801 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 7 2026-02-20T11:20:05.801 INFO:tasks.rados.rados.0.trial043.stdout:1011: done (6 left) 2026-02-20T11:20:05.801 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 6 2026-02-20T11:20:05.801 INFO:tasks.rados.rados.0.trial043.stdout:1012: expect (ObjNum 440 snap 119 seq_num 440) 2026-02-20T11:20:05.841 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 3214288 2026-02-20T11:20:05.850 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 2171474 2026-02-20T11:20:05.861 INFO:tasks.rados.rados.0.trial043.stdout:1014: finishing write tid 1 to trial04342147-5 2026-02-20T11:20:05.861 INFO:tasks.rados.rados.0.trial043.stdout:1012: done (5 left) 2026-02-20T11:20:05.861 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 5 2026-02-20T11:20:05.861 INFO:tasks.rados.rados.0.trial043.stdout:1014: finishing write tid 2 to trial04342147-5 2026-02-20T11:20:05.861 INFO:tasks.rados.rados.0.trial043.stdout:1010: finishing write tid 1 to trial04342147-50 2026-02-20T11:20:05.861 INFO:tasks.rados.rados.0.trial043.stdout:1014: finishing write tid 3 to trial04342147-5 2026-02-20T11:20:05.861 INFO:tasks.rados.rados.0.trial043.stdout:1010: finishing write tid 2 to trial04342147-50 2026-02-20T11:20:05.862 INFO:tasks.rados.rados.0.trial043.stdout:1014: finishing write tid 4 to trial04342147-5 2026-02-20T11:20:05.862 INFO:tasks.rados.rados.0.trial043.stdout:1014: finishing write tid 5 to trial04342147-5 2026-02-20T11:20:05.862 INFO:tasks.rados.rados.0.trial043.stdout:1014: oid 5 updating version 0 to 111 2026-02-20T11:20:05.862 INFO:tasks.rados.rados.0.trial043.stdout:1014: oid 5 version 111 is already newer than 111 2026-02-20T11:20:05.862 INFO:tasks.rados.rados.0.trial043.stdout:1014: oid 5 version 111 is already newer than 111 2026-02-20T11:20:05.862 INFO:tasks.rados.rados.0.trial043.stdout:1014: oid 5 version 111 is already newer than 111 2026-02-20T11:20:05.862 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 5 v 111 (ObjNum 469 snap 126 seq_num 469) dirty exists 2026-02-20T11:20:05.862 INFO:tasks.rados.rados.0.trial043.stdout:1014: left oid 5 (ObjNum 469 snap 126 seq_num 469) 2026-02-20T11:20:05.862 INFO:tasks.rados.rados.0.trial043.stdout:1013: expect (ObjNum 452 snap 122 seq_num 452) 2026-02-20T11:20:05.862 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:20:05.921 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:20:05.981 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:20:06.042 INFO:tasks.rados.rados.0.trial043.stdout:1010: finishing write tid 3 to trial04342147-50 2026-02-20T11:20:06.042 INFO:tasks.rados.rados.0.trial043.stdout:1013: done (4 left) 2026-02-20T11:20:06.042 INFO:tasks.rados.rados.0.trial043.stdout:1014: done (3 left) 2026-02-20T11:20:06.043 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:20:06.043 INFO:tasks.rados.rados.0.trial043.stdout:1010: finishing write tid 4 to trial04342147-50 2026-02-20T11:20:06.043 INFO:tasks.rados.rados.0.trial043.stdout:1010: finishing write tid 5 to trial04342147-50 2026-02-20T11:20:06.043 INFO:tasks.rados.rados.0.trial043.stdout:1010: oid 50 updating version 0 to 119 2026-02-20T11:20:06.043 INFO:tasks.rados.rados.0.trial043.stdout:1010: oid 50 version 119 is already newer than 119 2026-02-20T11:20:06.043 INFO:tasks.rados.rados.0.trial043.stdout:1010: oid 50 version 119 is already newer than 116 2026-02-20T11:20:06.043 INFO:tasks.rados.rados.0.trial043.stdout:1010: oid 50 version 119 is already newer than 119 2026-02-20T11:20:06.043 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 50 v 119 (ObjNum 468 snap 126 seq_num 468) dirty exists 2026-02-20T11:20:06.043 INFO:tasks.rados.rados.0.trial043.stdout:1010: left oid 50 (ObjNum 468 snap 126 seq_num 468) 2026-02-20T11:20:06.043 INFO:tasks.rados.rados.0.trial043.stdout:1015: finishing write tid 1 to trial04342147-27 2026-02-20T11:20:06.043 INFO:tasks.rados.rados.0.trial043.stdout:1015: finishing write tid 2 to trial04342147-27 2026-02-20T11:20:06.044 INFO:tasks.rados.rados.0.trial043.stdout:1010: done (2 left) 2026-02-20T11:20:06.044 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:20:06.044 INFO:tasks.rados.rados.0.trial043.stdout:1015: finishing write tid 3 to trial04342147-27 2026-02-20T11:20:06.044 INFO:tasks.rados.rados.0.trial043.stdout:1015: finishing write tid 4 to trial04342147-27 2026-02-20T11:20:06.044 INFO:tasks.rados.rados.0.trial043.stdout:1015: finishing write tid 5 to trial04342147-27 2026-02-20T11:20:06.044 INFO:tasks.rados.rados.0.trial043.stdout:1015: oid 27 updating version 0 to 294 2026-02-20T11:20:06.044 INFO:tasks.rados.rados.0.trial043.stdout:1015: oid 27 version 294 is already newer than 294 2026-02-20T11:20:06.044 INFO:tasks.rados.rados.0.trial043.stdout:1015: oid 27 version 294 is already newer than 294 2026-02-20T11:20:06.044 INFO:tasks.rados.rados.0.trial043.stdout:1015: oid 27 version 294 is already newer than 294 2026-02-20T11:20:06.044 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 27 v 294 (ObjNum 470 snap 126 seq_num 470) dirty exists 2026-02-20T11:20:06.044 INFO:tasks.rados.rados.0.trial043.stdout:1015: left oid 27 (ObjNum 470 snap 126 seq_num 470) 2026-02-20T11:20:06.044 INFO:tasks.rados.rados.0.trial043.stdout:1015: done (1 left) 2026-02-20T11:20:06.045 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:20:06.045 INFO:tasks.rados.rados.0.trial043.stdout:1017: done (0 left) 2026-02-20T11:20:06.805 INFO:tasks.rados.rados.0.trial043.stdout:1018: done (0 left) 2026-02-20T11:20:06.805 INFO:tasks.rados.rados.0.trial043.stdout:1019: write (excl) oid 17 current snap is 127 2026-02-20T11:20:06.805 INFO:tasks.rados.rados.0.trial043.stdout:1019: seq_num 471 ranges {716097=539812,1945607=795166,3535053=381044} 2026-02-20T11:20:06.810 INFO:tasks.rados.rados.0.trial043.stdout:1019: writing trial04342147-17 from 716097 to 1255909 tid 1 2026-02-20T11:20:06.816 INFO:tasks.rados.rados.0.trial043.stdout:1019: writing trial04342147-17 from 1945607 to 2740773 tid 2 2026-02-20T11:20:06.820 INFO:tasks.rados.rados.0.trial043.stdout:1019: writing trial04342147-17 from 3535053 to 3916097 tid 3 2026-02-20T11:20:06.820 INFO:tasks.rados.rados.0.trial043.stdout:1020: snap_create 2026-02-20T11:20:06.820 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:20:06.836 INFO:tasks.rados.rados.0.trial043.stdout:1019: finishing write tid 1 to trial04342147-17 2026-02-20T11:20:06.850 INFO:tasks.rados.rados.0.trial043.stdout:1019: finishing write tid 2 to trial04342147-17 2026-02-20T11:20:06.856 INFO:tasks.rados.rados.0.trial043.stdout:1019: finishing write tid 3 to trial04342147-17 2026-02-20T11:20:06.860 INFO:tasks.rados.rados.0.trial043.stdout:1019: finishing write tid 4 to trial04342147-17 2026-02-20T11:20:06.862 INFO:tasks.rados.rados.0.trial043.stdout:1019: finishing write tid 5 to trial04342147-17 2026-02-20T11:20:06.862 INFO:tasks.rados.rados.0.trial043.stdout:1019: oid 17 updating version 0 to 300 2026-02-20T11:20:06.862 INFO:tasks.rados.rados.0.trial043.stdout:1019: oid 17 version 300 is already newer than 300 2026-02-20T11:20:06.862 INFO:tasks.rados.rados.0.trial043.stdout:1019: oid 17 version 300 is already newer than 300 2026-02-20T11:20:06.862 INFO:tasks.rados.rados.0.trial043.stdout:1019: oid 17 version 300 is already newer than 297 2026-02-20T11:20:06.862 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 17 v 300 (ObjNum 471 snap 127 seq_num 471) dirty exists 2026-02-20T11:20:06.862 INFO:tasks.rados.rados.0.trial043.stdout:1019: left oid 17 (ObjNum 471 snap 127 seq_num 471) 2026-02-20T11:20:06.863 INFO:tasks.rados.rados.0.trial043.stdout:1019: done (0 left) 2026-02-20T11:20:07.121 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:20:07.200 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:07.199+0000 7fb23e451640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:07.202 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:07.202+0000 7fb23e451640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:07.228 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:20:07.245 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:20:07.246 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:20:07.808 INFO:tasks.rados.rados.0.trial043.stdout:1020: done (0 left) 2026-02-20T11:20:07.808 INFO:tasks.rados.rados.0.trial043.stdout:1021: delete oid 13 current snap is 128 2026-02-20T11:20:07.855 INFO:tasks.rados.rados.0.trial043.stdout:1021: done (0 left) 2026-02-20T11:20:07.856 INFO:tasks.rados.rados.0.trial043.stdout:1022: snap_create 2026-02-20T11:20:08.782 INFO:tasks.rados.rados.0.trial043.stdout:1022: done (0 left) 2026-02-20T11:20:08.783 INFO:tasks.rados.rados.0.trial043.stdout:1023: snap_remove snap 127 2026-02-20T11:20:09.247 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:20:09.326 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:09.325+0000 7fc6f04cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:09.328 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:09.329+0000 7fc6f04cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:09.354 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:20:09.370 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:20:09.371 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:20:09.531 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~264s 2026-02-20T11:20:09.783 INFO:tasks.rados.rados.0.trial043.stdout:1023: done (0 left) 2026-02-20T11:20:09.784 INFO:tasks.rados.rados.0.trial043.stdout:1024: delete oid 1 current snap is 129 2026-02-20T11:20:09.834 INFO:tasks.rados.rados.0.trial043.stdout:1024: done (0 left) 2026-02-20T11:20:09.835 INFO:tasks.rados.rados.0.trial043.stdout:1025: delete oid 38 current snap is 129 2026-02-20T11:20:09.863 INFO:tasks.rados.rados.0.trial043.stdout:1025: done (0 left) 2026-02-20T11:20:09.863 INFO:tasks.rados.rados.0.trial043.stdout:1026: write (excl) oid 33 current snap is 129 2026-02-20T11:20:09.864 INFO:tasks.rados.rados.0.trial043.stdout:1026: seq_num 475 ranges {783325=557161,1779535=686151,3244745=454912} 2026-02-20T11:20:09.868 INFO:tasks.rados.rados.0.trial043.stdout:1026: writing trial04342147-33 from 783325 to 1340486 tid 1 2026-02-20T11:20:09.873 INFO:tasks.rados.rados.0.trial043.stdout:1026: writing trial04342147-33 from 1779535 to 2465686 tid 2 2026-02-20T11:20:09.878 INFO:tasks.rados.rados.0.trial043.stdout:1026: writing trial04342147-33 from 3244745 to 3699657 tid 3 2026-02-20T11:20:09.879 INFO:tasks.rados.rados.0.trial043.stdout:1027: snap_create 2026-02-20T11:20:09.879 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:20:09.896 INFO:tasks.rados.rados.0.trial043.stdout:1026: finishing write tid 1 to trial04342147-33 2026-02-20T11:20:09.915 INFO:tasks.rados.rados.0.trial043.stdout:1026: finishing write tid 2 to trial04342147-33 2026-02-20T11:20:09.920 INFO:tasks.rados.rados.0.trial043.stdout:1026: finishing write tid 3 to trial04342147-33 2026-02-20T11:20:09.925 INFO:tasks.rados.rados.0.trial043.stdout:1026: finishing write tid 4 to trial04342147-33 2026-02-20T11:20:09.927 INFO:tasks.rados.rados.0.trial043.stdout:1026: finishing write tid 5 to trial04342147-33 2026-02-20T11:20:09.928 INFO:tasks.rados.rados.0.trial043.stdout:1026: oid 33 updating version 0 to 346 2026-02-20T11:20:09.928 INFO:tasks.rados.rados.0.trial043.stdout:1026: oid 33 updating version 346 to 349 2026-02-20T11:20:09.928 INFO:tasks.rados.rados.0.trial043.stdout:1026: oid 33 version 349 is already newer than 349 2026-02-20T11:20:09.928 INFO:tasks.rados.rados.0.trial043.stdout:1026: oid 33 version 349 is already newer than 349 2026-02-20T11:20:09.928 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 33 v 349 (ObjNum 475 snap 129 seq_num 475) dirty exists 2026-02-20T11:20:09.928 INFO:tasks.rados.rados.0.trial043.stdout:1026: left oid 33 (ObjNum 475 snap 129 seq_num 475) 2026-02-20T11:20:09.928 INFO:tasks.rados.rados.0.trial043.stdout:1026: done (0 left) 2026-02-20T11:20:09.933 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:20:10.002 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:20:10.004 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:20:10.005 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:20:10.816 INFO:tasks.rados.rados.0.trial043.stdout:1027: done (0 left) 2026-02-20T11:20:10.817 INFO:tasks.rados.rados.0.trial043.stdout:1028: read oid 17 snap -1 2026-02-20T11:20:10.817 INFO:tasks.rados.rados.0.trial043.stdout:1028: expect (ObjNum 471 snap 127 seq_num 471) 2026-02-20T11:20:10.817 INFO:tasks.rados.rados.0.trial043.stdout:1029: read oid 9 snap -1 2026-02-20T11:20:10.817 INFO:tasks.rados.rados.0.trial043.stdout:1029: expect deleted 2026-02-20T11:20:10.817 INFO:tasks.rados.rados.0.trial043.stdout:1030: read oid 30 snap 128 2026-02-20T11:20:10.817 INFO:tasks.rados.rados.0.trial043.stdout:1030: expect (ObjNum 466 snap 124 seq_num 466) 2026-02-20T11:20:10.817 INFO:tasks.rados.rados.0.trial043.stdout:1031: snap_remove snap 123 2026-02-20T11:20:11.372 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:20:11.448 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:11.447+0000 7fc1e6211640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:11.451 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:11.451+0000 7fc1e6211640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:11.476 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:20:11.492 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:20:11.493 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:20:11.820 INFO:tasks.rados.rados.0.trial043.stdout:1031: done (3 left) 2026-02-20T11:20:11.821 INFO:tasks.rados.rados.0.trial043.stdout:1032: snap_create 2026-02-20T11:20:11.821 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:20:11.821 INFO:tasks.rados.rados.0.trial043.stdout:1029: done (2 left) 2026-02-20T11:20:11.821 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:20:11.821 INFO:tasks.rados.rados.0.trial043.stdout:1030: expect (ObjNum 466 snap 124 seq_num 466) 2026-02-20T11:20:11.855 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:20:11.914 INFO:tasks.rados.rados.0.trial043.stdout:1028: expect (ObjNum 471 snap 127 seq_num 471) 2026-02-20T11:20:11.914 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:20:11.950 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1653672 2026-02-20T11:20:11.973 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:20:12.007 INFO:tasks.rados.rados.0.trial043.stdout:1028: done (1 left) 2026-02-20T11:20:12.007 INFO:tasks.rados.rados.0.trial043.stdout:1030: done (0 left) 2026-02-20T11:20:12.823 INFO:tasks.rados.rados.0.trial043.stdout:1032: done (0 left) 2026-02-20T11:20:12.823 INFO:tasks.rados.rados.0.trial043.stdout:1033: read oid 43 snap -1 2026-02-20T11:20:12.823 INFO:tasks.rados.rados.0.trial043.stdout:1033: expect deleted 2026-02-20T11:20:12.823 INFO:tasks.rados.rados.0.trial043.stdout:1034: snap_remove snap 124 2026-02-20T11:20:13.494 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:20:13.569 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:13.568+0000 7f839fdcc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:13.572 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:13.572+0000 7f839fdcc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:13.598 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:20:13.613 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:20:13.614 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:20:13.826 INFO:tasks.rados.rados.0.trial043.stdout:1034: done (1 left) 2026-02-20T11:20:13.826 INFO:tasks.rados.rados.0.trial043.stdout:1035: read oid 7 snap -1 2026-02-20T11:20:13.826 INFO:tasks.rados.rados.0.trial043.stdout:1035: expect deleted 2026-02-20T11:20:13.826 INFO:tasks.rados.rados.0.trial043.stdout:1036: read oid 16 snap -1 2026-02-20T11:20:13.826 INFO:tasks.rados.rados.0.trial043.stdout:1036: expect deleted 2026-02-20T11:20:13.826 INFO:tasks.rados.rados.0.trial043.stdout:1037: read oid 1 snap -1 2026-02-20T11:20:13.826 INFO:tasks.rados.rados.0.trial043.stdout:1037: expect deleted 2026-02-20T11:20:13.826 INFO:tasks.rados.rados.0.trial043.stdout:1038: snap_remove snap 130 2026-02-20T11:20:14.832 INFO:tasks.rados.rados.0.trial043.stdout:1038: done (4 left) 2026-02-20T11:20:14.832 INFO:tasks.rados.rados.0.trial043.stdout:1039: snap_create 2026-02-20T11:20:14.832 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 4 2026-02-20T11:20:14.832 INFO:tasks.rados.rados.0.trial043.stdout:1033: done (3 left) 2026-02-20T11:20:14.832 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:20:14.832 INFO:tasks.rados.rados.0.trial043.stdout:1036: done (2 left) 2026-02-20T11:20:14.832 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:20:14.832 INFO:tasks.rados.rados.0.trial043.stdout:1035: done (1 left) 2026-02-20T11:20:14.832 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:20:14.833 INFO:tasks.rados.rados.0.trial043.stdout:1037: done (0 left) 2026-02-20T11:20:15.006 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:20:15.035 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~270s 2026-02-20T11:20:15.078 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:20:15.081 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:20:15.081 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:20:15.614 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:20:15.688 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:15.687+0000 7f7a39574640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:15.691 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:15.691+0000 7f7a39574640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:15.718 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:20:15.735 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:20:15.736 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:20:15.831 INFO:tasks.rados.rados.0.trial043.stdout:1039: done (0 left) 2026-02-20T11:20:15.831 INFO:tasks.rados.rados.0.trial043.stdout:1040: snap_create 2026-02-20T11:20:16.838 INFO:tasks.rados.rados.0.trial043.stdout:1040: done (0 left) 2026-02-20T11:20:16.838 INFO:tasks.rados.rados.0.trial043.stdout:1041: snap_create 2026-02-20T11:20:17.737 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:20:17.816 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:17.816+0000 7f2694bb8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:17.819 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:17.819+0000 7f2694bb8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:17.840 INFO:tasks.rados.rados.0.trial043.stdout:1041: done (0 left) 2026-02-20T11:20:17.840 INFO:tasks.rados.rados.0.trial043.stdout:1042: read oid 50 snap -1 2026-02-20T11:20:17.840 INFO:tasks.rados.rados.0.trial043.stdout:1042: expect (ObjNum 468 snap 126 seq_num 468) 2026-02-20T11:20:17.840 INFO:tasks.rados.rados.0.trial043.stdout:1043: read oid 1 snap -1 2026-02-20T11:20:17.840 INFO:tasks.rados.rados.0.trial043.stdout:1043: expect deleted 2026-02-20T11:20:17.840 INFO:tasks.rados.rados.0.trial043.stdout:1044: write oid 40 current snap is 134 2026-02-20T11:20:17.840 INFO:tasks.rados.rados.0.trial043.stdout:1044: seq_num 476 ranges {400132=632410,1438017=546162} 2026-02-20T11:20:17.844 INFO:tasks.rados.rados.0.trial043.stdout:1044: writing trial04342147-40 from 400132 to 1032542 tid 1 2026-02-20T11:20:17.848 INFO:tasks.rados.rados.0.trial043.stdout:1044: writing trial04342147-40 from 1438017 to 1984179 tid 2 2026-02-20T11:20:17.848 INFO:tasks.rados.rados.0.trial043.stdout:1045: snap_remove snap 128 2026-02-20T11:20:17.862 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:20:17.901 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:20:17.902 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:20:18.786 INFO:tasks.rados.rados.0.trial043.stdout:1045: done (3 left) 2026-02-20T11:20:18.786 INFO:tasks.rados.rados.0.trial043.stdout:1046: write oid 13 current snap is 134 2026-02-20T11:20:18.786 INFO:tasks.rados.rados.0.trial043.stdout:1046: seq_num 477 ranges {416939=707659,1896499=120440} 2026-02-20T11:20:18.790 INFO:tasks.rados.rados.0.trial043.stdout:1046: writing trial04342147-13 from 416939 to 1124598 tid 1 2026-02-20T11:20:18.794 INFO:tasks.rados.rados.0.trial043.stdout:1046: writing trial04342147-13 from 1896499 to 2016939 tid 2 2026-02-20T11:20:18.794 INFO:tasks.rados.rados.0.trial043.stdout:1047: read oid 38 snap 132 2026-02-20T11:20:18.794 INFO:tasks.rados.rados.0.trial043.stdout:1047: expect deleted 2026-02-20T11:20:18.794 INFO:tasks.rados.rados.0.trial043.stdout:1048: delete oid 11 current snap is 134 2026-02-20T11:20:18.794 INFO:tasks.rados.rados.0.trial043.stdout:1042: expect (ObjNum 468 snap 126 seq_num 468) 2026-02-20T11:20:18.831 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1685041 2026-02-20T11:20:18.867 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1421495 2026-02-20T11:20:18.900 INFO:tasks.rados.rados.0.trial043.stdout:1044: finishing write tid 1 to trial04342147-40 2026-02-20T11:20:18.901 INFO:tasks.rados.rados.0.trial043.stdout:1044: finishing write tid 2 to trial04342147-40 2026-02-20T11:20:18.901 INFO:tasks.rados.rados.0.trial043.stdout:1042: done (5 left) 2026-02-20T11:20:18.901 INFO:tasks.rados.rados.0.trial043.stdout:1043: done (4 left) 2026-02-20T11:20:18.901 INFO:tasks.rados.rados.0.trial043.stdout:1048: done (3 left) 2026-02-20T11:20:18.901 INFO:tasks.rados.rados.0.trial043.stdout:1049: delete oid 25 current snap is 134 2026-02-20T11:20:18.901 INFO:tasks.rados.rados.0.trial043.stdout:1044: finishing write tid 3 to trial04342147-40 2026-02-20T11:20:18.901 INFO:tasks.rados.rados.0.trial043.stdout:1044: finishing write tid 4 to trial04342147-40 2026-02-20T11:20:18.901 INFO:tasks.rados.rados.0.trial043.stdout:1044: oid 40 updating version 0 to 244 2026-02-20T11:20:18.901 INFO:tasks.rados.rados.0.trial043.stdout:1044: oid 40 version 244 is already newer than 244 2026-02-20T11:20:18.901 INFO:tasks.rados.rados.0.trial043.stdout:1044: oid 40 version 244 is already newer than 244 2026-02-20T11:20:18.901 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 40 v 244 (ObjNum 476 snap 134 seq_num 476) dirty exists 2026-02-20T11:20:18.902 INFO:tasks.rados.rados.0.trial043.stdout:1044: left oid 40 (ObjNum 476 snap 134 seq_num 476) 2026-02-20T11:20:18.902 INFO:tasks.rados.rados.0.trial043.stdout:1046: finishing write tid 1 to trial04342147-13 2026-02-20T11:20:18.902 INFO:tasks.rados.rados.0.trial043.stdout:1046: finishing write tid 2 to trial04342147-13 2026-02-20T11:20:18.902 INFO:tasks.rados.rados.0.trial043.stdout:1046: finishing write tid 3 to trial04342147-13 2026-02-20T11:20:18.902 INFO:tasks.rados.rados.0.trial043.stdout:1046: finishing write tid 4 to trial04342147-13 2026-02-20T11:20:18.902 INFO:tasks.rados.rados.0.trial043.stdout:1046: oid 13 updating version 0 to 240 2026-02-20T11:20:18.902 INFO:tasks.rados.rados.0.trial043.stdout:1046: oid 13 version 240 is already newer than 240 2026-02-20T11:20:18.902 INFO:tasks.rados.rados.0.trial043.stdout:1046: oid 13 version 240 is already newer than 240 2026-02-20T11:20:18.902 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 13 v 240 (ObjNum 477 snap 134 seq_num 477) dirty exists 2026-02-20T11:20:18.902 INFO:tasks.rados.rados.0.trial043.stdout:1046: left oid 13 (ObjNum 477 snap 134 seq_num 477) 2026-02-20T11:20:18.903 INFO:tasks.rados.rados.0.trial043.stdout:1044: done (3 left) 2026-02-20T11:20:18.903 INFO:tasks.rados.rados.0.trial043.stdout:1046: done (2 left) 2026-02-20T11:20:18.903 INFO:tasks.rados.rados.0.trial043.stdout:1047: done (1 left) 2026-02-20T11:20:18.903 INFO:tasks.rados.rados.0.trial043.stdout:1049: done (0 left) 2026-02-20T11:20:18.903 INFO:tasks.rados.rados.0.trial043.stdout:1050: write oid 35 current snap is 134 2026-02-20T11:20:18.904 INFO:tasks.rados.rados.0.trial043.stdout:1050: seq_num 480 ranges {467360=649759,1671945=395415} 2026-02-20T11:20:18.907 INFO:tasks.rados.rados.0.trial043.stdout:1050: writing trial04342147-35 from 467360 to 1117119 tid 1 2026-02-20T11:20:18.911 INFO:tasks.rados.rados.0.trial043.stdout:1050: writing trial04342147-35 from 1671945 to 2067360 tid 2 2026-02-20T11:20:18.911 INFO:tasks.rados.rados.0.trial043.stdout:1051: read oid 12 snap -1 2026-02-20T11:20:18.911 INFO:tasks.rados.rados.0.trial043.stdout:1051: expect (ObjNum 432 snap 118 seq_num 432) 2026-02-20T11:20:18.911 INFO:tasks.rados.rados.0.trial043.stdout:1052: read oid 42 snap -1 2026-02-20T11:20:18.911 INFO:tasks.rados.rados.0.trial043.stdout:1052: expect (ObjNum 440 snap 119 seq_num 440) 2026-02-20T11:20:18.911 INFO:tasks.rados.rados.0.trial043.stdout:1053: write oid 38 current snap is 134 2026-02-20T11:20:18.911 INFO:tasks.rados.rados.0.trial043.stdout:1053: seq_num 481 ranges {484167=725008,1730427=353740} 2026-02-20T11:20:18.915 INFO:tasks.rados.rados.0.trial043.stdout:1053: writing trial04342147-38 from 484167 to 1209175 tid 1 2026-02-20T11:20:18.919 INFO:tasks.rados.rados.0.trial043.stdout:1053: writing trial04342147-38 from 1730427 to 2084167 tid 2 2026-02-20T11:20:18.919 INFO:tasks.rados.rados.0.trial043.stdout:1054: read oid 11 snap -1 2026-02-20T11:20:18.919 INFO:tasks.rados.rados.0.trial043.stdout:1054: expect deleted 2026-02-20T11:20:18.919 INFO:tasks.rados.rados.0.trial043.stdout:1055: snap_create 2026-02-20T11:20:18.919 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 5 2026-02-20T11:20:18.923 INFO:tasks.rados.rados.0.trial043.stdout:1054: done (4 left) 2026-02-20T11:20:18.923 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 4 2026-02-20T11:20:18.937 INFO:tasks.rados.rados.0.trial043.stdout:1053: finishing write tid 1 to trial04342147-38 2026-02-20T11:20:18.939 INFO:tasks.rados.rados.0.trial043.stdout:1051: expect (ObjNum 432 snap 118 seq_num 432) 2026-02-20T11:20:18.939 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:20:18.969 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1385292 2026-02-20T11:20:19.015 INFO:tasks.rados.rados.0.trial043.stdout:1050: finishing write tid 1 to trial04342147-35 2026-02-20T11:20:19.015 INFO:tasks.rados.rados.0.trial043.stdout:1053: finishing write tid 2 to trial04342147-38 2026-02-20T11:20:19.015 INFO:tasks.rados.rados.0.trial043.stdout:1050: finishing write tid 2 to trial04342147-35 2026-02-20T11:20:19.015 INFO:tasks.rados.rados.0.trial043.stdout:1051: done (3 left) 2026-02-20T11:20:19.015 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:20:19.015 INFO:tasks.rados.rados.0.trial043.stdout:1052: expect (ObjNum 440 snap 119 seq_num 440) 2026-02-20T11:20:19.031 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:20:19.089 INFO:tasks.rados.rados.0.trial043.stdout:1053: finishing write tid 3 to trial04342147-38 2026-02-20T11:20:19.089 INFO:tasks.rados.rados.0.trial043.stdout:1053: finishing write tid 4 to trial04342147-38 2026-02-20T11:20:19.089 INFO:tasks.rados.rados.0.trial043.stdout:1053: oid 38 updating version 0 to 42 2026-02-20T11:20:19.090 INFO:tasks.rados.rados.0.trial043.stdout:1053: oid 38 version 42 is already newer than 42 2026-02-20T11:20:19.090 INFO:tasks.rados.rados.0.trial043.stdout:1053: oid 38 version 42 is already newer than 42 2026-02-20T11:20:19.090 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 38 v 42 (ObjNum 481 snap 134 seq_num 481) dirty exists 2026-02-20T11:20:19.090 INFO:tasks.rados.rados.0.trial043.stdout:1053: left oid 38 (ObjNum 481 snap 134 seq_num 481) 2026-02-20T11:20:19.090 INFO:tasks.rados.rados.0.trial043.stdout:1050: finishing write tid 3 to trial04342147-35 2026-02-20T11:20:19.090 INFO:tasks.rados.rados.0.trial043.stdout:1052: done (2 left) 2026-02-20T11:20:19.090 INFO:tasks.rados.rados.0.trial043.stdout:1053: done (1 left) 2026-02-20T11:20:19.090 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:20:19.090 INFO:tasks.rados.rados.0.trial043.stdout:1050: finishing write tid 4 to trial04342147-35 2026-02-20T11:20:19.090 INFO:tasks.rados.rados.0.trial043.stdout:1050: oid 35 updating version 0 to 244 2026-02-20T11:20:19.090 INFO:tasks.rados.rados.0.trial043.stdout:1050: oid 35 version 244 is already newer than 244 2026-02-20T11:20:19.090 INFO:tasks.rados.rados.0.trial043.stdout:1050: oid 35 version 244 is already newer than 244 2026-02-20T11:20:19.090 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 35 v 244 (ObjNum 480 snap 134 seq_num 480) dirty exists 2026-02-20T11:20:19.090 INFO:tasks.rados.rados.0.trial043.stdout:1050: left oid 35 (ObjNum 480 snap 134 seq_num 480) 2026-02-20T11:20:19.090 INFO:tasks.rados.rados.0.trial043.stdout:1050: done (0 left) 2026-02-20T11:20:19.844 INFO:tasks.rados.rados.0.trial043.stdout:1055: done (0 left) 2026-02-20T11:20:19.844 INFO:tasks.rados.rados.0.trial043.stdout:1056: delete oid 50 current snap is 135 2026-02-20T11:20:19.888 INFO:tasks.rados.rados.0.trial043.stdout:1056: done (0 left) 2026-02-20T11:20:19.889 INFO:tasks.rados.rados.0.trial043.stdout:1057: delete oid 28 current snap is 135 2026-02-20T11:20:19.902 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:20:19.924 INFO:tasks.rados.rados.0.trial043.stdout:1057: done (0 left) 2026-02-20T11:20:19.924 INFO:tasks.rados.rados.0.trial043.stdout:1058: write (excl) oid 42 current snap is 135 2026-02-20T11:20:19.924 INFO:tasks.rados.rados.0.trial043.stdout:1058: seq_num 484 ranges {534588=550755,1789520=345068} 2026-02-20T11:20:19.928 INFO:tasks.rados.rados.0.trial043.stdout:1058: writing trial04342147-42 from 534588 to 1085343 tid 1 2026-02-20T11:20:19.932 INFO:tasks.rados.rados.0.trial043.stdout:1058: writing trial04342147-42 from 1789520 to 2134588 tid 2 2026-02-20T11:20:19.932 INFO:tasks.rados.rados.0.trial043.stdout:1059: snap_remove snap 133 2026-02-20T11:20:19.978 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:19.976+0000 7f67e385a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:19.980 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:19.980+0000 7f67e385a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:20.011 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:20:20.026 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:20:20.027 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:20:20.082 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:20:20.156 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:20:20.158 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:20:20.159 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:20:20.539 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~275s 2026-02-20T11:20:20.848 INFO:tasks.rados.rados.0.trial043.stdout:1059: done (1 left) 2026-02-20T11:20:20.848 INFO:tasks.rados.rados.0.trial043.stdout:1060: write oid 48 current snap is 135 2026-02-20T11:20:20.848 INFO:tasks.rados.rados.0.trial043.stdout:1058: finishing write tid 1 to trial04342147-42 2026-02-20T11:20:20.848 INFO:tasks.rados.rados.0.trial043.stdout:1060: seq_num 485 ranges {551395=626004,1848002=303393} 2026-02-20T11:20:20.853 INFO:tasks.rados.rados.0.trial043.stdout:1060: writing trial04342147-48 from 551395 to 1177399 tid 1 2026-02-20T11:20:20.858 INFO:tasks.rados.rados.0.trial043.stdout:1060: writing trial04342147-48 from 1848002 to 2151395 tid 2 2026-02-20T11:20:20.858 INFO:tasks.rados.rados.0.trial043.stdout:1061: read oid 20 snap -1 2026-02-20T11:20:20.858 INFO:tasks.rados.rados.0.trial043.stdout:1061: expect (ObjNum 128 snap 17 seq_num 128) 2026-02-20T11:20:20.858 INFO:tasks.rados.rados.0.trial043.stdout:1062: write (excl) oid 24 current snap is 135 2026-02-20T11:20:20.858 INFO:tasks.rados.rados.0.trial043.stdout:1058: finishing write tid 2 to trial04342147-42 2026-02-20T11:20:20.858 INFO:tasks.rados.rados.0.trial043.stdout:1062: seq_num 486 ranges {568202=701253,1906484=261718} 2026-02-20T11:20:20.862 INFO:tasks.rados.rados.0.trial043.stdout:1062: writing trial04342147-24 from 568202 to 1269455 tid 1 2026-02-20T11:20:20.866 INFO:tasks.rados.rados.0.trial043.stdout:1062: writing trial04342147-24 from 1906484 to 2168202 tid 2 2026-02-20T11:20:20.866 INFO:tasks.rados.rados.0.trial043.stdout:1063: delete oid 1 current snap is 135 2026-02-20T11:20:20.866 INFO:tasks.rados.rados.0.trial043.stdout:1058: finishing write tid 3 to trial04342147-42 2026-02-20T11:20:20.866 INFO:tasks.rados.rados.0.trial043.stdout:1058: finishing write tid 4 to trial04342147-42 2026-02-20T11:20:20.866 INFO:tasks.rados.rados.0.trial043.stdout:1058: oid 42 updating version 0 to 152 2026-02-20T11:20:20.866 INFO:tasks.rados.rados.0.trial043.stdout:1058: oid 42 version 152 is already newer than 152 2026-02-20T11:20:20.866 INFO:tasks.rados.rados.0.trial043.stdout:1058: oid 42 version 152 is already newer than 150 2026-02-20T11:20:20.866 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 42 v 152 (ObjNum 484 snap 135 seq_num 484) dirty exists 2026-02-20T11:20:20.866 INFO:tasks.rados.rados.0.trial043.stdout:1058: left oid 42 (ObjNum 484 snap 135 seq_num 484) 2026-02-20T11:20:20.869 INFO:tasks.rados.rados.0.trial043.stdout:1058: done (4 left) 2026-02-20T11:20:20.869 INFO:tasks.rados.rados.0.trial043.stdout:1063: done (3 left) 2026-02-20T11:20:20.869 INFO:tasks.rados.rados.0.trial043.stdout:1064: write (excl) oid 34 current snap is 135 2026-02-20T11:20:20.869 INFO:tasks.rados.rados.0.trial043.stdout:1064: seq_num 488 ranges {601816=568104,1623448=451823} 2026-02-20T11:20:20.873 INFO:tasks.rados.rados.0.trial043.stdout:1064: writing trial04342147-34 from 601816 to 1169920 tid 1 2026-02-20T11:20:20.877 INFO:tasks.rados.rados.0.trial043.stdout:1064: writing trial04342147-34 from 1623448 to 2075271 tid 2 2026-02-20T11:20:20.877 INFO:tasks.rados.rados.0.trial043.stdout:1065: write (excl) oid 37 current snap is 135 2026-02-20T11:20:20.877 INFO:tasks.rados.rados.0.trial043.stdout:1065: seq_num 489 ranges {618623=643353,1681930=536693} 2026-02-20T11:20:20.881 INFO:tasks.rados.rados.0.trial043.stdout:1065: writing trial04342147-37 from 618623 to 1261976 tid 1 2026-02-20T11:20:20.885 INFO:tasks.rados.rados.0.trial043.stdout:1065: writing trial04342147-37 from 1681930 to 2218623 tid 2 2026-02-20T11:20:20.885 INFO:tasks.rados.rados.0.trial043.stdout:1066: write oid 46 current snap is 135 2026-02-20T11:20:20.885 INFO:tasks.rados.rados.0.trial043.stdout:1066: seq_num 490 ranges {635430=718602,2140412=95018} 2026-02-20T11:20:20.890 INFO:tasks.rados.rados.0.trial043.stdout:1066: writing trial04342147-46 from 635430 to 1354032 tid 1 2026-02-20T11:20:20.894 INFO:tasks.rados.rados.0.trial043.stdout:1066: writing trial04342147-46 from 2140412 to 2235430 tid 2 2026-02-20T11:20:20.894 INFO:tasks.rados.rados.0.trial043.stdout:1067: write (excl) oid 1 current snap is 135 2026-02-20T11:20:20.894 INFO:tasks.rados.rados.0.trial043.stdout:1067: seq_num 491 ranges {652237=793851,2198894=53343} 2026-02-20T11:20:20.899 INFO:tasks.rados.rados.0.trial043.stdout:1067: writing trial04342147-1 from 652237 to 1446088 tid 1 2026-02-20T11:20:20.902 INFO:tasks.rados.rados.0.trial043.stdout:1067: writing trial04342147-1 from 2198894 to 2252237 tid 2 2026-02-20T11:20:20.903 INFO:tasks.rados.rados.0.trial043.stdout:1068: snap_create 2026-02-20T11:20:20.903 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 7 2026-02-20T11:20:20.903 INFO:tasks.rados.rados.0.trial043.stdout:1061: expect (ObjNum 128 snap 17 seq_num 128) 2026-02-20T11:20:20.903 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:20:20.980 INFO:tasks.rados.rados.0.trial043.stdout:1060: finishing write tid 1 to trial04342147-48 2026-02-20T11:20:20.980 INFO:tasks.rados.rados.0.trial043.stdout:1062: finishing write tid 1 to trial04342147-24 2026-02-20T11:20:20.981 INFO:tasks.rados.rados.0.trial043.stdout:1060: finishing write tid 2 to trial04342147-48 2026-02-20T11:20:20.981 INFO:tasks.rados.rados.0.trial043.stdout:1065: finishing write tid 1 to trial04342147-37 2026-02-20T11:20:20.981 INFO:tasks.rados.rados.0.trial043.stdout:1061: done (6 left) 2026-02-20T11:20:20.981 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 6 2026-02-20T11:20:20.981 INFO:tasks.rados.rados.0.trial043.stdout:1060: finishing write tid 3 to trial04342147-48 2026-02-20T11:20:20.981 INFO:tasks.rados.rados.0.trial043.stdout:1064: finishing write tid 1 to trial04342147-34 2026-02-20T11:20:20.981 INFO:tasks.rados.rados.0.trial043.stdout:1060: finishing write tid 4 to trial04342147-48 2026-02-20T11:20:20.981 INFO:tasks.rados.rados.0.trial043.stdout:1060: oid 48 updating version 0 to 101 2026-02-20T11:20:20.981 INFO:tasks.rados.rados.0.trial043.stdout:1060: oid 48 version 101 is already newer than 101 2026-02-20T11:20:20.981 INFO:tasks.rados.rados.0.trial043.stdout:1060: oid 48 version 101 is already newer than 101 2026-02-20T11:20:20.981 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 48 v 101 (ObjNum 485 snap 135 seq_num 485) dirty exists 2026-02-20T11:20:20.981 INFO:tasks.rados.rados.0.trial043.stdout:1060: left oid 48 (ObjNum 485 snap 135 seq_num 485) 2026-02-20T11:20:20.981 INFO:tasks.rados.rados.0.trial043.stdout:1062: finishing write tid 2 to trial04342147-24 2026-02-20T11:20:20.982 INFO:tasks.rados.rados.0.trial043.stdout:1062: finishing write tid 3 to trial04342147-24 2026-02-20T11:20:20.982 INFO:tasks.rados.rados.0.trial043.stdout:1067: finishing write tid 1 to trial04342147-1 2026-02-20T11:20:20.982 INFO:tasks.rados.rados.0.trial043.stdout:1060: done (5 left) 2026-02-20T11:20:20.982 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 5 2026-02-20T11:20:20.982 INFO:tasks.rados.rados.0.trial043.stdout:1065: finishing write tid 2 to trial04342147-37 2026-02-20T11:20:20.982 INFO:tasks.rados.rados.0.trial043.stdout:1062: finishing write tid 4 to trial04342147-24 2026-02-20T11:20:20.982 INFO:tasks.rados.rados.0.trial043.stdout:1062: oid 24 updating version 0 to 79 2026-02-20T11:20:20.982 INFO:tasks.rados.rados.0.trial043.stdout:1062: oid 24 version 79 is already newer than 79 2026-02-20T11:20:20.982 INFO:tasks.rados.rados.0.trial043.stdout:1062: oid 24 version 79 is already newer than 77 2026-02-20T11:20:20.982 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 24 v 79 (ObjNum 486 snap 135 seq_num 486) dirty exists 2026-02-20T11:20:20.982 INFO:tasks.rados.rados.0.trial043.stdout:1062: left oid 24 (ObjNum 486 snap 135 seq_num 486) 2026-02-20T11:20:20.982 INFO:tasks.rados.rados.0.trial043.stdout:1066: finishing write tid 1 to trial04342147-46 2026-02-20T11:20:20.982 INFO:tasks.rados.rados.0.trial043.stdout:1065: finishing write tid 3 to trial04342147-37 2026-02-20T11:20:20.982 INFO:tasks.rados.rados.0.trial043.stdout:1067: finishing write tid 2 to trial04342147-1 2026-02-20T11:20:20.982 INFO:tasks.rados.rados.0.trial043.stdout:1062: done (4 left) 2026-02-20T11:20:20.983 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 4 2026-02-20T11:20:20.983 INFO:tasks.rados.rados.0.trial043.stdout:1067: finishing write tid 3 to trial04342147-1 2026-02-20T11:20:20.983 INFO:tasks.rados.rados.0.trial043.stdout:1067: finishing write tid 4 to trial04342147-1 2026-02-20T11:20:20.983 INFO:tasks.rados.rados.0.trial043.stdout:1067: oid 1 updating version 0 to 117 2026-02-20T11:20:20.983 INFO:tasks.rados.rados.0.trial043.stdout:1067: oid 1 version 117 is already newer than 115 2026-02-20T11:20:20.983 INFO:tasks.rados.rados.0.trial043.stdout:1067: oid 1 version 117 is already newer than 117 2026-02-20T11:20:20.983 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 1 v 117 (ObjNum 491 snap 135 seq_num 491) dirty exists 2026-02-20T11:20:20.983 INFO:tasks.rados.rados.0.trial043.stdout:1067: left oid 1 (ObjNum 491 snap 135 seq_num 491) 2026-02-20T11:20:20.983 INFO:tasks.rados.rados.0.trial043.stdout:1065: finishing write tid 4 to trial04342147-37 2026-02-20T11:20:20.983 INFO:tasks.rados.rados.0.trial043.stdout:1065: oid 37 updating version 0 to 247 2026-02-20T11:20:20.983 INFO:tasks.rados.rados.0.trial043.stdout:1065: oid 37 version 247 is already newer than 247 2026-02-20T11:20:20.983 INFO:tasks.rados.rados.0.trial043.stdout:1065: oid 37 version 247 is already newer than 245 2026-02-20T11:20:20.983 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 37 v 247 (ObjNum 489 snap 135 seq_num 489) dirty exists 2026-02-20T11:20:20.983 INFO:tasks.rados.rados.0.trial043.stdout:1065: left oid 37 (ObjNum 489 snap 135 seq_num 489) 2026-02-20T11:20:20.983 INFO:tasks.rados.rados.0.trial043.stdout:1066: finishing write tid 2 to trial04342147-46 2026-02-20T11:20:20.983 INFO:tasks.rados.rados.0.trial043.stdout:1064: finishing write tid 2 to trial04342147-34 2026-02-20T11:20:20.984 INFO:tasks.rados.rados.0.trial043.stdout:1064: finishing write tid 3 to trial04342147-34 2026-02-20T11:20:20.984 INFO:tasks.rados.rados.0.trial043.stdout:1064: finishing write tid 4 to trial04342147-34 2026-02-20T11:20:20.984 INFO:tasks.rados.rados.0.trial043.stdout:1064: oid 34 updating version 0 to 66 2026-02-20T11:20:20.984 INFO:tasks.rados.rados.0.trial043.stdout:1064: oid 34 version 66 is already newer than 64 2026-02-20T11:20:20.984 INFO:tasks.rados.rados.0.trial043.stdout:1064: oid 34 version 66 is already newer than 66 2026-02-20T11:20:20.984 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 34 v 66 (ObjNum 488 snap 135 seq_num 488) dirty exists 2026-02-20T11:20:20.984 INFO:tasks.rados.rados.0.trial043.stdout:1064: left oid 34 (ObjNum 488 snap 135 seq_num 488) 2026-02-20T11:20:20.984 INFO:tasks.rados.rados.0.trial043.stdout:1064: done (3 left) 2026-02-20T11:20:20.984 INFO:tasks.rados.rados.0.trial043.stdout:1065: done (2 left) 2026-02-20T11:20:20.984 INFO:tasks.rados.rados.0.trial043.stdout:1067: done (1 left) 2026-02-20T11:20:20.984 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:20:20.984 INFO:tasks.rados.rados.0.trial043.stdout:1066: finishing write tid 3 to trial04342147-46 2026-02-20T11:20:20.985 INFO:tasks.rados.rados.0.trial043.stdout:1066: finishing write tid 4 to trial04342147-46 2026-02-20T11:20:20.985 INFO:tasks.rados.rados.0.trial043.stdout:1066: oid 46 updating version 0 to 31 2026-02-20T11:20:20.985 INFO:tasks.rados.rados.0.trial043.stdout:1066: oid 46 version 31 is already newer than 31 2026-02-20T11:20:20.985 INFO:tasks.rados.rados.0.trial043.stdout:1066: oid 46 version 31 is already newer than 31 2026-02-20T11:20:20.985 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 46 v 31 (ObjNum 490 snap 135 seq_num 490) dirty exists 2026-02-20T11:20:20.985 INFO:tasks.rados.rados.0.trial043.stdout:1066: left oid 46 (ObjNum 490 snap 135 seq_num 490) 2026-02-20T11:20:20.985 INFO:tasks.rados.rados.0.trial043.stdout:1066: done (0 left) 2026-02-20T11:20:21.852 INFO:tasks.rados.rados.0.trial043.stdout:1068: done (0 left) 2026-02-20T11:20:21.852 INFO:tasks.rados.rados.0.trial043.stdout:1069: write oid 43 current snap is 136 2026-02-20T11:20:21.852 INFO:tasks.rados.rados.0.trial043.stdout:1069: seq_num 492 ranges {669044=469100,1741023=459161} 2026-02-20T11:20:21.856 INFO:tasks.rados.rados.0.trial043.stdout:1069: writing trial04342147-43 from 669044 to 1138144 tid 1 2026-02-20T11:20:21.860 INFO:tasks.rados.rados.0.trial043.stdout:1069: writing trial04342147-43 from 1741023 to 2200184 tid 2 2026-02-20T11:20:21.861 INFO:tasks.rados.rados.0.trial043.stdout:1070: snap_create 2026-02-20T11:20:21.861 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:20:21.880 INFO:tasks.rados.rados.0.trial043.stdout:1069: finishing write tid 1 to trial04342147-43 2026-02-20T11:20:21.884 INFO:tasks.rados.rados.0.trial043.stdout:1069: finishing write tid 2 to trial04342147-43 2026-02-20T11:20:21.890 INFO:tasks.rados.rados.0.trial043.stdout:1069: finishing write tid 3 to trial04342147-43 2026-02-20T11:20:21.892 INFO:tasks.rados.rados.0.trial043.stdout:1069: finishing write tid 4 to trial04342147-43 2026-02-20T11:20:21.892 INFO:tasks.rados.rados.0.trial043.stdout:1069: oid 43 updating version 0 to 362 2026-02-20T11:20:21.892 INFO:tasks.rados.rados.0.trial043.stdout:1069: oid 43 version 362 is already newer than 362 2026-02-20T11:20:21.892 INFO:tasks.rados.rados.0.trial043.stdout:1069: oid 43 version 362 is already newer than 362 2026-02-20T11:20:21.892 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 43 v 362 (ObjNum 492 snap 136 seq_num 492) dirty exists 2026-02-20T11:20:21.892 INFO:tasks.rados.rados.0.trial043.stdout:1069: left oid 43 (ObjNum 492 snap 136 seq_num 492) 2026-02-20T11:20:21.892 INFO:tasks.rados.rados.0.trial043.stdout:1069: done (0 left) 2026-02-20T11:20:22.028 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:20:22.103 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:22.102+0000 7f7a446cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:22.105 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:22.106+0000 7f7a446cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:22.131 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:20:22.148 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:20:22.148 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:20:22.856 INFO:tasks.rados.rados.0.trial043.stdout:1070: done (0 left) 2026-02-20T11:20:22.856 INFO:tasks.rados.rados.0.trial043.stdout:1071: read oid 22 snap -1 2026-02-20T11:20:22.856 INFO:tasks.rados.rados.0.trial043.stdout:1071: expect deleted 2026-02-20T11:20:22.856 INFO:tasks.rados.rados.0.trial043.stdout:1072: read oid 12 snap -1 2026-02-20T11:20:22.857 INFO:tasks.rados.rados.0.trial043.stdout:1072: expect (ObjNum 432 snap 118 seq_num 432) 2026-02-20T11:20:22.857 INFO:tasks.rados.rados.0.trial043.stdout:1073: snap_create 2026-02-20T11:20:22.857 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:20:22.875 INFO:tasks.rados.rados.0.trial043.stdout:1071: done (1 left) 2026-02-20T11:20:22.875 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:20:22.885 INFO:tasks.rados.rados.0.trial043.stdout:1072: expect (ObjNum 432 snap 118 seq_num 432) 2026-02-20T11:20:22.934 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:20:22.963 INFO:tasks.rados.rados.0.trial043.stdout:1072: done (0 left) 2026-02-20T11:20:23.859 INFO:tasks.rados.rados.0.trial043.stdout:1073: done (0 left) 2026-02-20T11:20:23.859 INFO:tasks.rados.rados.0.trial043.stdout:1074: delete oid 47 current snap is 138 2026-02-20T11:20:23.905 INFO:tasks.rados.rados.0.trial043.stdout:1074: done (0 left) 2026-02-20T11:20:23.905 INFO:tasks.rados.rados.0.trial043.stdout:1075: write oid 27 current snap is 138 2026-02-20T11:20:23.905 INFO:tasks.rados.rados.0.trial043.stdout:1075: seq_num 494 ranges {702658=619598,1857987=444671} 2026-02-20T11:20:23.910 INFO:tasks.rados.rados.0.trial043.stdout:1075: writing trial04342147-27 from 702658 to 1322256 tid 1 2026-02-20T11:20:23.914 INFO:tasks.rados.rados.0.trial043.stdout:1075: writing trial04342147-27 from 1857987 to 2302658 tid 2 2026-02-20T11:20:23.914 INFO:tasks.rados.rados.0.trial043.stdout:1076: delete oid 38 current snap is 138 2026-02-20T11:20:23.945 INFO:tasks.rados.rados.0.trial043.stdout:1075: finishing write tid 1 to trial04342147-27 2026-02-20T11:20:23.946 INFO:tasks.rados.rados.0.trial043.stdout:1076: done (1 left) 2026-02-20T11:20:23.946 INFO:tasks.rados.rados.0.trial043.stdout:1077: snap_remove snap 134 2026-02-20T11:20:24.149 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:20:24.225 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:24.224+0000 7fd3bde11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:24.228 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:24.228+0000 7fd3bde11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:24.254 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:20:24.271 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:20:24.271 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:20:24.864 INFO:tasks.rados.rados.0.trial043.stdout:1077: done (1 left) 2026-02-20T11:20:24.864 INFO:tasks.rados.rados.0.trial043.stdout:1078: write (excl) oid 42 current snap is 138 2026-02-20T11:20:24.864 INFO:tasks.rados.rados.0.trial043.stdout:1075: finishing write tid 2 to trial04342147-27 2026-02-20T11:20:24.864 INFO:tasks.rados.rados.0.trial043.stdout:1078: seq_num 496 ranges {736272=486449,1974951=361321} 2026-02-20T11:20:24.868 INFO:tasks.rados.rados.0.trial043.stdout:1078: writing trial04342147-42 from 736272 to 1222721 tid 1 2026-02-20T11:20:24.873 INFO:tasks.rados.rados.0.trial043.stdout:1078: writing trial04342147-42 from 1974951 to 2336272 tid 2 2026-02-20T11:20:24.873 INFO:tasks.rados.rados.0.trial043.stdout:1079: snap_remove snap 137 2026-02-20T11:20:25.160 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:20:25.231 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:20:25.234 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:20:25.234 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:20:25.865 INFO:tasks.rados.rados.0.trial043.stdout:1079: done (2 left) 2026-02-20T11:20:25.865 INFO:tasks.rados.rados.0.trial043.stdout:1080: write (excl) oid 44 current snap is 138 2026-02-20T11:20:25.866 INFO:tasks.rados.rados.0.trial043.stdout:1075: finishing write tid 3 to trial04342147-27 2026-02-20T11:20:25.866 INFO:tasks.rados.rados.0.trial043.stdout:1080: seq_num 497 ranges {753079=561698,2033433=319646} 2026-02-20T11:20:25.870 INFO:tasks.rados.rados.0.trial043.stdout:1080: writing trial04342147-44 from 753079 to 1314777 tid 1 2026-02-20T11:20:25.876 INFO:tasks.rados.rados.0.trial043.stdout:1080: writing trial04342147-44 from 2033433 to 2353079 tid 2 2026-02-20T11:20:25.876 INFO:tasks.rados.rados.0.trial043.stdout:1081: snap_create 2026-02-20T11:20:25.876 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:20:25.876 INFO:tasks.rados.rados.0.trial043.stdout:1075: finishing write tid 4 to trial04342147-27 2026-02-20T11:20:25.876 INFO:tasks.rados.rados.0.trial043.stdout:1075: oid 27 updating version 0 to 310 2026-02-20T11:20:25.876 INFO:tasks.rados.rados.0.trial043.stdout:1075: oid 27 version 310 is already newer than 310 2026-02-20T11:20:25.876 INFO:tasks.rados.rados.0.trial043.stdout:1075: oid 27 version 310 is already newer than 310 2026-02-20T11:20:25.876 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 27 v 310 (ObjNum 494 snap 138 seq_num 494) dirty exists 2026-02-20T11:20:25.876 INFO:tasks.rados.rados.0.trial043.stdout:1075: left oid 27 (ObjNum 494 snap 138 seq_num 494) 2026-02-20T11:20:25.876 INFO:tasks.rados.rados.0.trial043.stdout:1075: done (2 left) 2026-02-20T11:20:25.876 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:20:25.877 INFO:tasks.rados.rados.0.trial043.stdout:1078: finishing write tid 1 to trial04342147-42 2026-02-20T11:20:25.877 INFO:tasks.rados.rados.0.trial043.stdout:1078: finishing write tid 2 to trial04342147-42 2026-02-20T11:20:25.877 INFO:tasks.rados.rados.0.trial043.stdout:1078: finishing write tid 3 to trial04342147-42 2026-02-20T11:20:25.877 INFO:tasks.rados.rados.0.trial043.stdout:1078: finishing write tid 4 to trial04342147-42 2026-02-20T11:20:25.877 INFO:tasks.rados.rados.0.trial043.stdout:1078: oid 42 updating version 0 to 158 2026-02-20T11:20:25.877 INFO:tasks.rados.rados.0.trial043.stdout:1078: oid 42 version 158 is already newer than 156 2026-02-20T11:20:25.877 INFO:tasks.rados.rados.0.trial043.stdout:1078: oid 42 version 158 is already newer than 158 2026-02-20T11:20:25.877 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 42 v 158 (ObjNum 496 snap 138 seq_num 496) dirty exists 2026-02-20T11:20:25.877 INFO:tasks.rados.rados.0.trial043.stdout:1078: left oid 42 (ObjNum 496 snap 138 seq_num 496) 2026-02-20T11:20:25.877 INFO:tasks.rados.rados.0.trial043.stdout:1078: done (1 left) 2026-02-20T11:20:25.877 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:20:25.911 INFO:tasks.rados.rados.0.trial043.stdout:1080: finishing write tid 1 to trial04342147-44 2026-02-20T11:20:25.931 INFO:tasks.rados.rados.0.trial043.stdout:1080: finishing write tid 2 to trial04342147-44 2026-02-20T11:20:25.937 INFO:tasks.rados.rados.0.trial043.stdout:1080: finishing write tid 3 to trial04342147-44 2026-02-20T11:20:25.939 INFO:tasks.rados.rados.0.trial043.stdout:1080: finishing write tid 4 to trial04342147-44 2026-02-20T11:20:25.939 INFO:tasks.rados.rados.0.trial043.stdout:1080: oid 44 updating version 0 to 160 2026-02-20T11:20:25.939 INFO:tasks.rados.rados.0.trial043.stdout:1080: oid 44 updating version 160 to 162 2026-02-20T11:20:25.939 INFO:tasks.rados.rados.0.trial043.stdout:1080: oid 44 version 162 is already newer than 162 2026-02-20T11:20:25.939 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 44 v 162 (ObjNum 497 snap 138 seq_num 497) dirty exists 2026-02-20T11:20:25.939 INFO:tasks.rados.rados.0.trial043.stdout:1080: left oid 44 (ObjNum 497 snap 138 seq_num 497) 2026-02-20T11:20:25.939 INFO:tasks.rados.rados.0.trial043.stdout:1080: done (0 left) 2026-02-20T11:20:26.044 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~281s 2026-02-20T11:20:26.272 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:20:26.347 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:26.346+0000 7f642675b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:26.350 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:26.350+0000 7f642675b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:26.376 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:20:26.392 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:20:26.392 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:20:26.870 INFO:tasks.rados.rados.0.trial043.stdout:1081: done (0 left) 2026-02-20T11:20:26.870 INFO:tasks.rados.rados.0.trial043.stdout:1082: read oid 26 snap 126 2026-02-20T11:20:26.871 INFO:tasks.rados.rados.0.trial043.stdout:1082: expect (ObjNum 456 snap 124 seq_num 456) 2026-02-20T11:20:26.871 INFO:tasks.rados.rados.0.trial043.stdout:1083: delete oid 32 current snap is 139 2026-02-20T11:20:26.913 INFO:tasks.rados.rados.0.trial043.stdout:1083: done (1 left) 2026-02-20T11:20:26.913 INFO:tasks.rados.rados.0.trial043.stdout:1084: snap_create 2026-02-20T11:20:26.913 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:20:26.923 INFO:tasks.rados.rados.0.trial043.stdout:1082: expect (ObjNum 456 snap 124 seq_num 456) 2026-02-20T11:20:27.084 INFO:tasks.rados.rados.0.trial043.stdout:1082: done (0 left) 2026-02-20T11:20:27.876 INFO:tasks.rados.rados.0.trial043.stdout:1084: done (0 left) 2026-02-20T11:20:27.876 INFO:tasks.rados.rados.0.trial043.stdout:1085: snap_remove snap 138 2026-02-20T11:20:28.394 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:20:28.470 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:28.469+0000 7f6cf6b83640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:28.473 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:28.473+0000 7f6cf6b83640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:28.499 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:20:28.515 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:20:28.516 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:20:28.787 INFO:tasks.rados.rados.0.trial043.stdout:1085: done (0 left) 2026-02-20T11:20:28.788 INFO:tasks.rados.rados.0.trial043.stdout:1086: read oid 36 snap -1 2026-02-20T11:20:28.788 INFO:tasks.rados.rados.0.trial043.stdout:1086: expect deleted 2026-02-20T11:20:28.788 INFO:tasks.rados.rados.0.trial043.stdout:1087: delete oid 12 current snap is 140 2026-02-20T11:20:28.827 INFO:tasks.rados.rados.0.trial043.stdout:1086: done (1 left) 2026-02-20T11:20:28.827 INFO:tasks.rados.rados.0.trial043.stdout:1087: done (0 left) 2026-02-20T11:20:28.827 INFO:tasks.rados.rados.0.trial043.stdout:1088: write oid 49 current snap is 140 2026-02-20T11:20:28.827 INFO:tasks.rados.rados.0.trial043.stdout:1088: seq_num 500 ranges {403500=787445,1692526=473837} 2026-02-20T11:20:28.830 INFO:tasks.rados.rados.0.trial043.stdout:1088: writing trial04342147-49 from 403500 to 1190945 tid 1 2026-02-20T11:20:28.834 INFO:tasks.rados.rados.0.trial043.stdout:1088: writing trial04342147-49 from 1692526 to 2166363 tid 2 2026-02-20T11:20:28.835 INFO:tasks.rados.rados.0.trial043.stdout:1089: read oid 2 snap -1 2026-02-20T11:20:28.835 INFO:tasks.rados.rados.0.trial043.stdout:1089: expect (ObjNum 388 snap 102 seq_num 388) 2026-02-20T11:20:28.835 INFO:tasks.rados.rados.0.trial043.stdout:1090: snap_create 2026-02-20T11:20:28.835 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:20:28.849 INFO:tasks.rados.rados.0.trial043.stdout:1088: finishing write tid 1 to trial04342147-49 2026-02-20T11:20:28.853 INFO:tasks.rados.rados.0.trial043.stdout:1089: expect (ObjNum 388 snap 102 seq_num 388) 2026-02-20T11:20:28.853 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 408043 2026-02-20T11:20:28.891 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 622504 2026-02-20T11:20:28.906 INFO:tasks.rados.rados.0.trial043.stdout:1089: done (1 left) 2026-02-20T11:20:28.906 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:20:28.906 INFO:tasks.rados.rados.0.trial043.stdout:1088: finishing write tid 2 to trial04342147-49 2026-02-20T11:20:28.906 INFO:tasks.rados.rados.0.trial043.stdout:1088: finishing write tid 3 to trial04342147-49 2026-02-20T11:20:28.906 INFO:tasks.rados.rados.0.trial043.stdout:1088: finishing write tid 4 to trial04342147-49 2026-02-20T11:20:28.906 INFO:tasks.rados.rados.0.trial043.stdout:1088: oid 49 updating version 0 to 104 2026-02-20T11:20:28.906 INFO:tasks.rados.rados.0.trial043.stdout:1088: oid 49 version 104 is already newer than 104 2026-02-20T11:20:28.906 INFO:tasks.rados.rados.0.trial043.stdout:1088: oid 49 version 104 is already newer than 104 2026-02-20T11:20:28.907 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 49 v 104 (ObjNum 500 snap 140 seq_num 500) dirty exists 2026-02-20T11:20:28.907 INFO:tasks.rados.rados.0.trial043.stdout:1088: left oid 49 (ObjNum 500 snap 140 seq_num 500) 2026-02-20T11:20:28.907 INFO:tasks.rados.rados.0.trial043.stdout:1088: done (0 left) 2026-02-20T11:20:29.883 INFO:tasks.rados.rados.0.trial043.stdout:1090: done (0 left) 2026-02-20T11:20:29.883 INFO:tasks.rados.rados.0.trial043.stdout:1091: read oid 43 snap 135 2026-02-20T11:20:29.883 INFO:tasks.rados.rados.0.trial043.stdout:1091: expect deleted 2026-02-20T11:20:29.883 INFO:tasks.rados.rados.0.trial043.stdout:1092: read oid 42 snap 139 2026-02-20T11:20:29.883 INFO:tasks.rados.rados.0.trial043.stdout:1092: expect (ObjNum 496 snap 138 seq_num 496) 2026-02-20T11:20:29.883 INFO:tasks.rados.rados.0.trial043.stdout:1093: write (excl) oid 32 current snap is 141 2026-02-20T11:20:29.883 INFO:tasks.rados.rados.0.trial043.stdout:1093: seq_num 501 ranges {420307=462694,1351008=617495} 2026-02-20T11:20:29.887 INFO:tasks.rados.rados.0.trial043.stdout:1093: writing trial04342147-32 from 420307 to 883001 tid 1 2026-02-20T11:20:29.891 INFO:tasks.rados.rados.0.trial043.stdout:1093: writing trial04342147-32 from 1351008 to 1968503 tid 2 2026-02-20T11:20:29.891 INFO:tasks.rados.rados.0.trial043.stdout:1094: read oid 1 snap 132 2026-02-20T11:20:29.891 INFO:tasks.rados.rados.0.trial043.stdout:1094: expect deleted 2026-02-20T11:20:29.891 INFO:tasks.rados.rados.0.trial043.stdout:1095: read oid 24 snap -1 2026-02-20T11:20:29.891 INFO:tasks.rados.rados.0.trial043.stdout:1095: expect (ObjNum 486 snap 135 seq_num 486) 2026-02-20T11:20:29.891 INFO:tasks.rados.rados.0.trial043.stdout:1096: read oid 12 snap -1 2026-02-20T11:20:29.891 INFO:tasks.rados.rados.0.trial043.stdout:1096: expect deleted 2026-02-20T11:20:29.891 INFO:tasks.rados.rados.0.trial043.stdout:1097: write (excl) oid 17 current snap is 141 2026-02-20T11:20:29.892 INFO:tasks.rados.rados.0.trial043.stdout:1097: seq_num 502 ranges {437114=654296,1525843=477506} 2026-02-20T11:20:29.896 INFO:tasks.rados.rados.0.trial043.stdout:1097: writing trial04342147-17 from 437114 to 1091410 tid 1 2026-02-20T11:20:29.899 INFO:tasks.rados.rados.0.trial043.stdout:1097: writing trial04342147-17 from 1525843 to 2003349 tid 2 2026-02-20T11:20:29.899 INFO:tasks.rados.rados.0.trial043.stdout:1098: write oid 3 current snap is 141 2026-02-20T11:20:29.899 INFO:tasks.rados.rados.0.trial043.stdout:1098: seq_num 503 ranges {453921=729545,1584325=621164} 2026-02-20T11:20:29.904 INFO:tasks.rados.rados.0.trial043.stdout:1098: writing trial04342147-3 from 453921 to 1183466 tid 1 2026-02-20T11:20:29.909 INFO:tasks.rados.rados.0.trial043.stdout:1098: writing trial04342147-3 from 1584325 to 2205489 tid 2 2026-02-20T11:20:29.909 INFO:tasks.rados.rados.0.trial043.stdout:1099: write oid 47 current snap is 141 2026-02-20T11:20:29.909 INFO:tasks.rados.rados.0.trial043.stdout:1099: seq_num 504 ranges {470728=404794,1526454=481175} 2026-02-20T11:20:29.912 INFO:tasks.rados.rados.0.trial043.stdout:1099: writing trial04342147-47 from 470728 to 875522 tid 1 2026-02-20T11:20:29.916 INFO:tasks.rados.rados.0.trial043.stdout:1099: writing trial04342147-47 from 1526454 to 2007629 tid 2 2026-02-20T11:20:29.917 INFO:tasks.rados.rados.0.trial043.stdout:1100: delete oid 36 current snap is 141 2026-02-20T11:20:29.920 INFO:tasks.rados.rados.0.trial043.stdout:1091: done (9 left) 2026-02-20T11:20:29.920 INFO:tasks.rados.rados.0.trial043.stdout:1094: done (8 left) 2026-02-20T11:20:29.920 INFO:tasks.rados.rados.0.trial043.stdout:1100: done (7 left) 2026-02-20T11:20:29.920 INFO:tasks.rados.rados.0.trial043.stdout:1101: snap_create 2026-02-20T11:20:29.920 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 7 2026-02-20T11:20:29.932 INFO:tasks.rados.rados.0.trial043.stdout:1098: finishing write tid 1 to trial04342147-3 2026-02-20T11:20:29.934 INFO:tasks.rados.rados.0.trial043.stdout:1093: finishing write tid 1 to trial04342147-32 2026-02-20T11:20:29.939 INFO:tasks.rados.rados.0.trial043.stdout:1096: done (6 left) 2026-02-20T11:20:29.939 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 6 2026-02-20T11:20:29.950 INFO:tasks.rados.rados.0.trial043.stdout:1098: finishing write tid 2 to trial04342147-3 2026-02-20T11:20:29.958 INFO:tasks.rados.rados.0.trial043.stdout:1092: expect (ObjNum 496 snap 138 seq_num 496) 2026-02-20T11:20:29.963 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1225998 2026-02-20T11:20:29.981 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:20:30.007 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:20:30.034 INFO:tasks.rados.rados.0.trial043.stdout:1093: finishing write tid 2 to trial04342147-32 2026-02-20T11:20:30.034 INFO:tasks.rados.rados.0.trial043.stdout:1098: finishing write tid 3 to trial04342147-3 2026-02-20T11:20:30.034 INFO:tasks.rados.rados.0.trial043.stdout:1098: finishing write tid 4 to trial04342147-3 2026-02-20T11:20:30.034 INFO:tasks.rados.rados.0.trial043.stdout:1098: oid 3 updating version 0 to 250 2026-02-20T11:20:30.034 INFO:tasks.rados.rados.0.trial043.stdout:1098: oid 3 version 250 is already newer than 250 2026-02-20T11:20:30.034 INFO:tasks.rados.rados.0.trial043.stdout:1098: oid 3 version 250 is already newer than 250 2026-02-20T11:20:30.034 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 3 v 250 (ObjNum 503 snap 141 seq_num 503) dirty exists 2026-02-20T11:20:30.034 INFO:tasks.rados.rados.0.trial043.stdout:1098: left oid 3 (ObjNum 503 snap 141 seq_num 503) 2026-02-20T11:20:30.034 INFO:tasks.rados.rados.0.trial043.stdout:1097: finishing write tid 1 to trial04342147-17 2026-02-20T11:20:30.034 INFO:tasks.rados.rados.0.trial043.stdout:1095: expect (ObjNum 486 snap 135 seq_num 486) 2026-02-20T11:20:30.086 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:20:30.138 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:20:30.191 INFO:tasks.rados.rados.0.trial043.stdout:1093: finishing write tid 3 to trial04342147-32 2026-02-20T11:20:30.192 INFO:tasks.rados.rados.0.trial043.stdout:1092: done (5 left) 2026-02-20T11:20:30.192 INFO:tasks.rados.rados.0.trial043.stdout:1095: done (4 left) 2026-02-20T11:20:30.192 INFO:tasks.rados.rados.0.trial043.stdout:1098: done (3 left) 2026-02-20T11:20:30.192 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:20:30.192 INFO:tasks.rados.rados.0.trial043.stdout:1093: finishing write tid 4 to trial04342147-32 2026-02-20T11:20:30.192 INFO:tasks.rados.rados.0.trial043.stdout:1093: oid 32 updating version 0 to 25 2026-02-20T11:20:30.192 INFO:tasks.rados.rados.0.trial043.stdout:1093: oid 32 updating version 25 to 27 2026-02-20T11:20:30.192 INFO:tasks.rados.rados.0.trial043.stdout:1093: oid 32 version 27 is already newer than 27 2026-02-20T11:20:30.192 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 32 v 27 (ObjNum 501 snap 141 seq_num 501) dirty exists 2026-02-20T11:20:30.192 INFO:tasks.rados.rados.0.trial043.stdout:1093: left oid 32 (ObjNum 501 snap 141 seq_num 501) 2026-02-20T11:20:30.192 INFO:tasks.rados.rados.0.trial043.stdout:1099: finishing write tid 1 to trial04342147-47 2026-02-20T11:20:30.192 INFO:tasks.rados.rados.0.trial043.stdout:1099: finishing write tid 2 to trial04342147-47 2026-02-20T11:20:30.192 INFO:tasks.rados.rados.0.trial043.stdout:1099: finishing write tid 3 to trial04342147-47 2026-02-20T11:20:30.192 INFO:tasks.rados.rados.0.trial043.stdout:1093: done (2 left) 2026-02-20T11:20:30.192 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:20:30.192 INFO:tasks.rados.rados.0.trial043.stdout:1099: finishing write tid 4 to trial04342147-47 2026-02-20T11:20:30.193 INFO:tasks.rados.rados.0.trial043.stdout:1099: oid 47 updating version 0 to 167 2026-02-20T11:20:30.193 INFO:tasks.rados.rados.0.trial043.stdout:1099: oid 47 version 167 is already newer than 167 2026-02-20T11:20:30.193 INFO:tasks.rados.rados.0.trial043.stdout:1099: oid 47 version 167 is already newer than 167 2026-02-20T11:20:30.193 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 47 v 167 (ObjNum 504 snap 141 seq_num 504) dirty exists 2026-02-20T11:20:30.193 INFO:tasks.rados.rados.0.trial043.stdout:1099: left oid 47 (ObjNum 504 snap 141 seq_num 504) 2026-02-20T11:20:30.193 INFO:tasks.rados.rados.0.trial043.stdout:1097: finishing write tid 2 to trial04342147-17 2026-02-20T11:20:30.193 INFO:tasks.rados.rados.0.trial043.stdout:1097: finishing write tid 3 to trial04342147-17 2026-02-20T11:20:30.193 INFO:tasks.rados.rados.0.trial043.stdout:1097: finishing write tid 4 to trial04342147-17 2026-02-20T11:20:30.193 INFO:tasks.rados.rados.0.trial043.stdout:1097: oid 17 updating version 0 to 318 2026-02-20T11:20:30.193 INFO:tasks.rados.rados.0.trial043.stdout:1097: oid 17 version 318 is already newer than 318 2026-02-20T11:20:30.193 INFO:tasks.rados.rados.0.trial043.stdout:1097: oid 17 version 318 is already newer than 316 2026-02-20T11:20:30.193 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 17 v 318 (ObjNum 502 snap 141 seq_num 502) dirty exists 2026-02-20T11:20:30.193 INFO:tasks.rados.rados.0.trial043.stdout:1097: left oid 17 (ObjNum 502 snap 141 seq_num 502) 2026-02-20T11:20:30.193 INFO:tasks.rados.rados.0.trial043.stdout:1097: done (1 left) 2026-02-20T11:20:30.194 INFO:tasks.rados.rados.0.trial043.stdout:1099: done (0 left) 2026-02-20T11:20:30.235 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:20:30.307 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:20:30.310 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:20:30.310 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:20:30.517 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:20:30.592 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:30.590+0000 7f4c8efa4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:30.595 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:30.594+0000 7f4c8efa4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:30.620 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:20:30.637 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:20:30.638 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:20:30.885 INFO:tasks.rados.rados.0.trial043.stdout:1101: done (0 left) 2026-02-20T11:20:30.885 INFO:tasks.rados.rados.0.trial043.stdout:1102: write (excl) oid 45 current snap is 142 2026-02-20T11:20:30.885 INFO:tasks.rados.rados.0.trial043.stdout:1102: seq_num 506 ranges {504342=555292,1643418=484844} 2026-02-20T11:20:30.889 INFO:tasks.rados.rados.0.trial043.stdout:1102: writing trial04342147-45 from 504342 to 1059634 tid 1 2026-02-20T11:20:30.893 INFO:tasks.rados.rados.0.trial043.stdout:1102: writing trial04342147-45 from 1643418 to 2128262 tid 2 2026-02-20T11:20:30.894 INFO:tasks.rados.rados.0.trial043.stdout:1103: snap_remove snap 139 2026-02-20T11:20:31.549 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~286s 2026-02-20T11:20:31.889 INFO:tasks.rados.rados.0.trial043.stdout:1103: done (1 left) 2026-02-20T11:20:31.889 INFO:tasks.rados.rados.0.trial043.stdout:1104: write oid 31 current snap is 142 2026-02-20T11:20:31.889 INFO:tasks.rados.rados.0.trial043.stdout:1102: finishing write tid 1 to trial04342147-45 2026-02-20T11:20:31.889 INFO:tasks.rados.rados.0.trial043.stdout:1102: finishing write tid 2 to trial04342147-45 2026-02-20T11:20:31.889 INFO:tasks.rados.rados.0.trial043.stdout:1102: finishing write tid 3 to trial04342147-45 2026-02-20T11:20:31.889 INFO:tasks.rados.rados.0.trial043.stdout:1102: finishing write tid 4 to trial04342147-45 2026-02-20T11:20:31.889 INFO:tasks.rados.rados.0.trial043.stdout:1102: oid 45 updating version 0 to 253 2026-02-20T11:20:31.890 INFO:tasks.rados.rados.0.trial043.stdout:1102: oid 45 version 253 is already newer than 251 2026-02-20T11:20:31.890 INFO:tasks.rados.rados.0.trial043.stdout:1102: oid 45 version 253 is already newer than 253 2026-02-20T11:20:31.890 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 45 v 253 (ObjNum 506 snap 142 seq_num 506) dirty exists 2026-02-20T11:20:31.890 INFO:tasks.rados.rados.0.trial043.stdout:1102: left oid 45 (ObjNum 506 snap 142 seq_num 506) 2026-02-20T11:20:31.890 INFO:tasks.rados.rados.0.trial043.stdout:1104: seq_num 507 ranges {521149=630541,1701900=628502} 2026-02-20T11:20:31.894 INFO:tasks.rados.rados.0.trial043.stdout:1104: writing trial04342147-31 from 521149 to 1151690 tid 1 2026-02-20T11:20:31.899 INFO:tasks.rados.rados.0.trial043.stdout:1104: writing trial04342147-31 from 1701900 to 2330402 tid 2 2026-02-20T11:20:31.899 INFO:tasks.rados.rados.0.trial043.stdout:1102: done (1 left) 2026-02-20T11:20:31.899 INFO:tasks.rados.rados.0.trial043.stdout:1105: read oid 4 snap -1 2026-02-20T11:20:31.899 INFO:tasks.rados.rados.0.trial043.stdout:1105: expect deleted 2026-02-20T11:20:31.899 INFO:tasks.rados.rados.0.trial043.stdout:1106: snap_remove snap 126 2026-02-20T11:20:32.639 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:20:32.715 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:32.713+0000 7fe701da4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:32.718 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:32.717+0000 7fe701da4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:32.746 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:20:32.763 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:20:32.763 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:20:32.893 INFO:tasks.rados.rados.0.trial043.stdout:1106: done (2 left) 2026-02-20T11:20:32.894 INFO:tasks.rados.rados.0.trial043.stdout:1107: snap_remove snap 131 2026-02-20T11:20:33.894 INFO:tasks.rados.rados.0.trial043.stdout:1107: done (2 left) 2026-02-20T11:20:33.894 INFO:tasks.rados.rados.0.trial043.stdout:1108: snap_remove snap 135 2026-02-20T11:20:34.764 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:20:34.841 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:34.839+0000 7f31f365a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:34.843 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:34.842+0000 7f31f365a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:34.869 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:20:34.886 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:20:34.887 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:20:34.897 INFO:tasks.rados.rados.0.trial043.stdout:1108: done (2 left) 2026-02-20T11:20:34.898 INFO:tasks.rados.rados.0.trial043.stdout:1109: delete oid 37 current snap is 142 2026-02-20T11:20:34.898 INFO:tasks.rados.rados.0.trial043.stdout:1104: finishing write tid 1 to trial04342147-31 2026-02-20T11:20:34.898 INFO:tasks.rados.rados.0.trial043.stdout:1104: finishing write tid 2 to trial04342147-31 2026-02-20T11:20:34.898 INFO:tasks.rados.rados.0.trial043.stdout:1104: finishing write tid 3 to trial04342147-31 2026-02-20T11:20:34.898 INFO:tasks.rados.rados.0.trial043.stdout:1104: finishing write tid 4 to trial04342147-31 2026-02-20T11:20:34.898 INFO:tasks.rados.rados.0.trial043.stdout:1104: oid 31 updating version 0 to 108 2026-02-20T11:20:34.898 INFO:tasks.rados.rados.0.trial043.stdout:1104: oid 31 version 108 is already newer than 108 2026-02-20T11:20:34.898 INFO:tasks.rados.rados.0.trial043.stdout:1104: oid 31 version 108 is already newer than 108 2026-02-20T11:20:34.899 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 31 v 108 (ObjNum 507 snap 142 seq_num 507) dirty exists 2026-02-20T11:20:34.899 INFO:tasks.rados.rados.0.trial043.stdout:1104: left oid 31 (ObjNum 507 snap 142 seq_num 507) 2026-02-20T11:20:34.939 INFO:tasks.rados.rados.0.trial043.stdout:1104: done (2 left) 2026-02-20T11:20:34.939 INFO:tasks.rados.rados.0.trial043.stdout:1105: done (1 left) 2026-02-20T11:20:34.939 INFO:tasks.rados.rados.0.trial043.stdout:1109: done (0 left) 2026-02-20T11:20:34.940 INFO:tasks.rados.rados.0.trial043.stdout:1110: snap_remove snap 129 2026-02-20T11:20:35.311 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:20:35.382 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:20:35.384 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:20:35.385 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:20:35.902 INFO:tasks.rados.rados.0.trial043.stdout:1110: done (0 left) 2026-02-20T11:20:35.903 INFO:tasks.rados.rados.0.trial043.stdout:1111: snap_create 2026-02-20T11:20:36.888 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:20:36.904 INFO:tasks.rados.rados.0.trial043.stdout:1111: done (0 left) 2026-02-20T11:20:36.904 INFO:tasks.rados.rados.0.trial043.stdout:1112: write (excl) oid 23 current snap is 143 2026-02-20T11:20:36.905 INFO:tasks.rados.rados.0.trial043.stdout:1112: seq_num 509 ranges {554763=781039,2102511=452252} 2026-02-20T11:20:36.909 INFO:tasks.rados.rados.0.trial043.stdout:1112: writing trial04342147-23 from 554763 to 1335802 tid 1 2026-02-20T11:20:36.914 INFO:tasks.rados.rados.0.trial043.stdout:1112: writing trial04342147-23 from 2102511 to 2554763 tid 2 2026-02-20T11:20:36.914 INFO:tasks.rados.rados.0.trial043.stdout:1113: read oid 33 snap 132 2026-02-20T11:20:36.915 INFO:tasks.rados.rados.0.trial043.stdout:1113: expect (ObjNum 475 snap 129 seq_num 475) 2026-02-20T11:20:36.915 INFO:tasks.rados.rados.0.trial043.stdout:1114: snap_create 2026-02-20T11:20:36.915 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:20:36.952 INFO:tasks.rados.rados.0.trial043.stdout:1112: finishing write tid 1 to trial04342147-23 2026-02-20T11:20:36.958 INFO:tasks.rados.rados.0.trial043.stdout:1113: expect (ObjNum 475 snap 129 seq_num 475) 2026-02-20T11:20:36.969 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:36.967+0000 7f8b70193640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:36.971 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:36.971+0000 7f8b70193640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:36.998 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:20:37.012 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:20:37.013 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:20:37.052 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~292s 2026-02-20T11:20:37.081 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:20:37.143 INFO:tasks.rados.rados.0.trial043.stdout:1112: finishing write tid 2 to trial04342147-23 2026-02-20T11:20:37.143 INFO:tasks.rados.rados.0.trial043.stdout:1112: finishing write tid 3 to trial04342147-23 2026-02-20T11:20:37.144 INFO:tasks.rados.rados.0.trial043.stdout:1113: done (1 left) 2026-02-20T11:20:37.144 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:20:37.144 INFO:tasks.rados.rados.0.trial043.stdout:1112: finishing write tid 4 to trial04342147-23 2026-02-20T11:20:37.144 INFO:tasks.rados.rados.0.trial043.stdout:1112: oid 23 updating version 0 to 115 2026-02-20T11:20:37.144 INFO:tasks.rados.rados.0.trial043.stdout:1112: oid 23 version 115 is already newer than 115 2026-02-20T11:20:37.144 INFO:tasks.rados.rados.0.trial043.stdout:1112: oid 23 version 115 is already newer than 113 2026-02-20T11:20:37.144 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 23 v 115 (ObjNum 509 snap 143 seq_num 509) dirty exists 2026-02-20T11:20:37.144 INFO:tasks.rados.rados.0.trial043.stdout:1112: left oid 23 (ObjNum 509 snap 143 seq_num 509) 2026-02-20T11:20:37.144 INFO:tasks.rados.rados.0.trial043.stdout:1112: done (0 left) 2026-02-20T11:20:37.910 INFO:tasks.rados.rados.0.trial043.stdout:1114: done (0 left) 2026-02-20T11:20:37.910 INFO:tasks.rados.rados.0.trial043.stdout:1115: write oid 49 current snap is 144 2026-02-20T11:20:37.910 INFO:tasks.rados.rados.0.trial043.stdout:1115: seq_num 510 ranges {571570=572641,1877346=694224} 2026-02-20T11:20:37.914 INFO:tasks.rados.rados.0.trial043.stdout:1115: writing trial04342147-49 from 571570 to 1144211 tid 1 2026-02-20T11:20:37.921 INFO:tasks.rados.rados.0.trial043.stdout:1115: writing trial04342147-49 from 1877346 to 2571570 tid 2 2026-02-20T11:20:37.921 INFO:tasks.rados.rados.0.trial043.stdout:1116: write oid 48 current snap is 144 2026-02-20T11:20:37.921 INFO:tasks.rados.rados.0.trial043.stdout:1116: seq_num 511 ranges {588377=647890,1935828=635840} 2026-02-20T11:20:37.925 INFO:tasks.rados.rados.0.trial043.stdout:1116: writing trial04342147-48 from 588377 to 1236267 tid 1 2026-02-20T11:20:37.931 INFO:tasks.rados.rados.0.trial043.stdout:1116: writing trial04342147-48 from 1935828 to 2571668 tid 2 2026-02-20T11:20:37.931 INFO:tasks.rados.rados.0.trial043.stdout:1117: write (excl) oid 22 current snap is 144 2026-02-20T11:20:37.931 INFO:tasks.rados.rados.0.trial043.stdout:1117: seq_num 512 ranges {605184=723139,1877957=727227} 2026-02-20T11:20:37.936 INFO:tasks.rados.rados.0.trial043.stdout:1117: writing trial04342147-22 from 605184 to 1328323 tid 1 2026-02-20T11:20:37.941 INFO:tasks.rados.rados.0.trial043.stdout:1117: writing trial04342147-22 from 1877957 to 2605184 tid 2 2026-02-20T11:20:37.941 INFO:tasks.rados.rados.0.trial043.stdout:1118: snap_remove snap 136 2026-02-20T11:20:38.790 INFO:tasks.rados.rados.0.trial043.stdout:1118: done (3 left) 2026-02-20T11:20:38.791 INFO:tasks.rados.rados.0.trial043.stdout:1119: delete oid 32 current snap is 144 2026-02-20T11:20:38.791 INFO:tasks.rados.rados.0.trial043.stdout:1117: finishing write tid 1 to trial04342147-22 2026-02-20T11:20:38.791 INFO:tasks.rados.rados.0.trial043.stdout:1115: finishing write tid 1 to trial04342147-49 2026-02-20T11:20:38.791 INFO:tasks.rados.rados.0.trial043.stdout:1115: finishing write tid 2 to trial04342147-49 2026-02-20T11:20:38.791 INFO:tasks.rados.rados.0.trial043.stdout:1117: finishing write tid 2 to trial04342147-22 2026-02-20T11:20:38.791 INFO:tasks.rados.rados.0.trial043.stdout:1115: finishing write tid 3 to trial04342147-49 2026-02-20T11:20:38.791 INFO:tasks.rados.rados.0.trial043.stdout:1115: finishing write tid 4 to trial04342147-49 2026-02-20T11:20:38.791 INFO:tasks.rados.rados.0.trial043.stdout:1115: oid 49 updating version 0 to 112 2026-02-20T11:20:38.791 INFO:tasks.rados.rados.0.trial043.stdout:1115: oid 49 version 112 is already newer than 112 2026-02-20T11:20:38.791 INFO:tasks.rados.rados.0.trial043.stdout:1115: oid 49 version 112 is already newer than 112 2026-02-20T11:20:38.792 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 49 v 112 (ObjNum 510 snap 144 seq_num 510) dirty exists 2026-02-20T11:20:38.792 INFO:tasks.rados.rados.0.trial043.stdout:1115: left oid 49 (ObjNum 510 snap 144 seq_num 510) 2026-02-20T11:20:38.792 INFO:tasks.rados.rados.0.trial043.stdout:1117: finishing write tid 3 to trial04342147-22 2026-02-20T11:20:38.792 INFO:tasks.rados.rados.0.trial043.stdout:1117: finishing write tid 4 to trial04342147-22 2026-02-20T11:20:38.792 INFO:tasks.rados.rados.0.trial043.stdout:1117: oid 22 updating version 0 to 124 2026-02-20T11:20:38.792 INFO:tasks.rados.rados.0.trial043.stdout:1117: oid 22 version 124 is already newer than 124 2026-02-20T11:20:38.792 INFO:tasks.rados.rados.0.trial043.stdout:1117: oid 22 version 124 is already newer than 122 2026-02-20T11:20:38.792 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 22 v 124 (ObjNum 512 snap 144 seq_num 512) dirty exists 2026-02-20T11:20:38.792 INFO:tasks.rados.rados.0.trial043.stdout:1117: left oid 22 (ObjNum 512 snap 144 seq_num 512) 2026-02-20T11:20:38.792 INFO:tasks.rados.rados.0.trial043.stdout:1116: finishing write tid 1 to trial04342147-48 2026-02-20T11:20:38.793 INFO:tasks.rados.rados.0.trial043.stdout:1116: finishing write tid 2 to trial04342147-48 2026-02-20T11:20:38.793 INFO:tasks.rados.rados.0.trial043.stdout:1116: finishing write tid 3 to trial04342147-48 2026-02-20T11:20:38.793 INFO:tasks.rados.rados.0.trial043.stdout:1116: finishing write tid 4 to trial04342147-48 2026-02-20T11:20:38.793 INFO:tasks.rados.rados.0.trial043.stdout:1116: oid 48 updating version 0 to 116 2026-02-20T11:20:38.793 INFO:tasks.rados.rados.0.trial043.stdout:1116: oid 48 version 116 is already newer than 116 2026-02-20T11:20:38.793 INFO:tasks.rados.rados.0.trial043.stdout:1116: oid 48 version 116 is already newer than 116 2026-02-20T11:20:38.793 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 48 v 116 (ObjNum 511 snap 144 seq_num 511) dirty exists 2026-02-20T11:20:38.793 INFO:tasks.rados.rados.0.trial043.stdout:1116: left oid 48 (ObjNum 511 snap 144 seq_num 511) 2026-02-20T11:20:38.830 INFO:tasks.rados.rados.0.trial043.stdout:1115: done (3 left) 2026-02-20T11:20:38.830 INFO:tasks.rados.rados.0.trial043.stdout:1116: done (2 left) 2026-02-20T11:20:38.831 INFO:tasks.rados.rados.0.trial043.stdout:1117: done (1 left) 2026-02-20T11:20:38.831 INFO:tasks.rados.rados.0.trial043.stdout:1119: done (0 left) 2026-02-20T11:20:38.831 INFO:tasks.rados.rados.0.trial043.stdout:1120: read oid 19 snap -1 2026-02-20T11:20:38.831 INFO:tasks.rados.rados.0.trial043.stdout:1120: expect deleted 2026-02-20T11:20:38.831 INFO:tasks.rados.rados.0.trial043.stdout:1121: delete oid 16 current snap is 144 2026-02-20T11:20:38.844 INFO:tasks.rados.rados.0.trial043.stdout:1120: done (1 left) 2026-02-20T11:20:38.844 INFO:tasks.rados.rados.0.trial043.stdout:1121: done (0 left) 2026-02-20T11:20:38.844 INFO:tasks.rados.rados.0.trial043.stdout:1122: snap_create 2026-02-20T11:20:39.013 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:20:39.089 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:39.087+0000 7f3005e1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:39.092 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:39.091+0000 7f3005e1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:39.116 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:20:39.133 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:20:39.134 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:20:39.790 INFO:tasks.rados.rados.0.trial043.stdout:1122: done (0 left) 2026-02-20T11:20:39.791 INFO:tasks.rados.rados.0.trial043.stdout:1123: snap_remove snap 144 2026-02-20T11:20:40.387 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:20:40.462 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:20:40.465 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:20:40.466 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:20:40.791 INFO:tasks.rados.rados.0.trial043.stdout:1123: done (0 left) 2026-02-20T11:20:40.791 INFO:tasks.rados.rados.0.trial043.stdout:1124: snap_remove snap 143 2026-02-20T11:20:41.135 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:20:41.212 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:41.210+0000 7f5acbe5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:41.214 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:41.213+0000 7f5acbe5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:41.240 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:20:41.257 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:20:41.257 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:20:41.792 INFO:tasks.rados.rados.0.trial043.stdout:1124: done (0 left) 2026-02-20T11:20:41.793 INFO:tasks.rados.rados.0.trial043.stdout:1125: read oid 4 snap 140 2026-02-20T11:20:41.793 INFO:tasks.rados.rados.0.trial043.stdout:1125: expect deleted 2026-02-20T11:20:41.793 INFO:tasks.rados.rados.0.trial043.stdout:1126: write oid 11 current snap is 145 2026-02-20T11:20:41.793 INFO:tasks.rados.rados.0.trial043.stdout:1126: seq_num 515 ranges {655605=548886,1653403=643178} 2026-02-20T11:20:41.797 INFO:tasks.rados.rados.0.trial043.stdout:1126: writing trial04342147-11 from 655605 to 1204491 tid 1 2026-02-20T11:20:41.802 INFO:tasks.rados.rados.0.trial043.stdout:1126: writing trial04342147-11 from 1653403 to 2296581 tid 2 2026-02-20T11:20:41.802 INFO:tasks.rados.rados.0.trial043.stdout:1127: read oid 6 snap -1 2026-02-20T11:20:41.802 INFO:tasks.rados.rados.0.trial043.stdout:1127: expect (ObjNum 424 snap 116 seq_num 424) 2026-02-20T11:20:41.802 INFO:tasks.rados.rados.0.trial043.stdout:1128: snap_remove snap 141 2026-02-20T11:20:42.555 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~297s 2026-02-20T11:20:42.794 INFO:tasks.rados.rados.0.trial043.stdout:1128: done (3 left) 2026-02-20T11:20:42.794 INFO:tasks.rados.rados.0.trial043.stdout:1129: write (excl) oid 30 current snap is 145 2026-02-20T11:20:42.794 INFO:tasks.rados.rados.0.trial043.stdout:1129: seq_num 516 ranges {672412=624135,1995532=676880} 2026-02-20T11:20:42.798 INFO:tasks.rados.rados.0.trial043.stdout:1129: writing trial04342147-30 from 672412 to 1296547 tid 1 2026-02-20T11:20:42.805 INFO:tasks.rados.rados.0.trial043.stdout:1129: writing trial04342147-30 from 1995532 to 2672412 tid 2 2026-02-20T11:20:42.805 INFO:tasks.rados.rados.0.trial043.stdout:1130: write (excl) oid 34 current snap is 145 2026-02-20T11:20:42.805 INFO:tasks.rados.rados.0.trial043.stdout:1130: seq_num 517 ranges {689219=415737,1770367=646847} 2026-02-20T11:20:42.809 INFO:tasks.rados.rados.0.trial043.stdout:1130: writing trial04342147-34 from 689219 to 1104956 tid 1 2026-02-20T11:20:42.814 INFO:tasks.rados.rados.0.trial043.stdout:1130: writing trial04342147-34 from 1770367 to 2417214 tid 2 2026-02-20T11:20:42.814 INFO:tasks.rados.rados.0.trial043.stdout:1131: write (excl) oid 15 current snap is 145 2026-02-20T11:20:42.815 INFO:tasks.rados.rados.0.trial043.stdout:1131: seq_num 518 ranges {706026=490986,1828849=790505} 2026-02-20T11:20:42.819 INFO:tasks.rados.rados.0.trial043.stdout:1131: writing trial04342147-15 from 706026 to 1197012 tid 1 2026-02-20T11:20:42.825 INFO:tasks.rados.rados.0.trial043.stdout:1131: writing trial04342147-15 from 1828849 to 2619354 tid 2 2026-02-20T11:20:42.825 INFO:tasks.rados.rados.0.trial043.stdout:1132: read oid 33 snap -1 2026-02-20T11:20:42.825 INFO:tasks.rados.rados.0.trial043.stdout:1132: expect (ObjNum 475 snap 129 seq_num 475) 2026-02-20T11:20:42.825 INFO:tasks.rados.rados.0.trial043.stdout:1133: read oid 40 snap -1 2026-02-20T11:20:42.825 INFO:tasks.rados.rados.0.trial043.stdout:1133: expect (ObjNum 476 snap 134 seq_num 476) 2026-02-20T11:20:42.825 INFO:tasks.rados.rados.0.trial043.stdout:1134: write (excl) oid 13 current snap is 145 2026-02-20T11:20:42.825 INFO:tasks.rados.rados.0.trial043.stdout:1134: seq_num 519 ranges {722833=566235,1887331=650516} 2026-02-20T11:20:42.829 INFO:tasks.rados.rados.0.trial043.stdout:1134: writing trial04342147-13 from 722833 to 1289068 tid 1 2026-02-20T11:20:42.834 INFO:tasks.rados.rados.0.trial043.stdout:1134: writing trial04342147-13 from 1887331 to 2537847 tid 2 2026-02-20T11:20:42.834 INFO:tasks.rados.rados.0.trial043.stdout:1135: write oid 20 current snap is 145 2026-02-20T11:20:42.834 INFO:tasks.rados.rados.0.trial043.stdout:1135: seq_num 520 ranges {739640=641484,1829460=794174} 2026-02-20T11:20:42.839 INFO:tasks.rados.rados.0.trial043.stdout:1135: writing trial04342147-20 from 739640 to 1381124 tid 1 2026-02-20T11:20:42.844 INFO:tasks.rados.rados.0.trial043.stdout:1135: writing trial04342147-20 from 1829460 to 2623634 tid 2 2026-02-20T11:20:42.844 INFO:tasks.rados.rados.0.trial043.stdout:1136: write (excl) oid 44 current snap is 145 2026-02-20T11:20:42.844 INFO:tasks.rados.rados.0.trial043.stdout:1127: expect (ObjNum 424 snap 116 seq_num 424) 2026-02-20T11:20:42.844 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:20:42.890 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:20:42.980 INFO:tasks.rados.rados.0.trial043.stdout:1126: finishing write tid 1 to trial04342147-11 2026-02-20T11:20:42.980 INFO:tasks.rados.rados.0.trial043.stdout:1126: finishing write tid 2 to trial04342147-11 2026-02-20T11:20:42.980 INFO:tasks.rados.rados.0.trial043.stdout:1136: seq_num 521 ranges {756447=716733,1887942=537832} 2026-02-20T11:20:42.986 INFO:tasks.rados.rados.0.trial043.stdout:1136: writing trial04342147-44 from 756447 to 1473180 tid 1 2026-02-20T11:20:42.991 INFO:tasks.rados.rados.0.trial043.stdout:1136: writing trial04342147-44 from 1887942 to 2425774 tid 2 2026-02-20T11:20:42.991 INFO:tasks.rados.rados.0.trial043.stdout:1125: done (10 left) 2026-02-20T11:20:42.991 INFO:tasks.rados.rados.0.trial043.stdout:1127: done (9 left) 2026-02-20T11:20:42.991 INFO:tasks.rados.rados.0.trial043.stdout:1137: read oid 49 snap -1 2026-02-20T11:20:42.991 INFO:tasks.rados.rados.0.trial043.stdout:1137: expect (ObjNum 510 snap 144 seq_num 510) 2026-02-20T11:20:42.992 INFO:tasks.rados.rados.0.trial043.stdout:1138: read oid 36 snap -1 2026-02-20T11:20:42.992 INFO:tasks.rados.rados.0.trial043.stdout:1138: expect deleted 2026-02-20T11:20:42.992 INFO:tasks.rados.rados.0.trial043.stdout:1139: delete oid 12 current snap is 145 2026-02-20T11:20:42.992 INFO:tasks.rados.rados.0.trial043.stdout:1126: finishing write tid 3 to trial04342147-11 2026-02-20T11:20:42.992 INFO:tasks.rados.rados.0.trial043.stdout:1126: finishing write tid 4 to trial04342147-11 2026-02-20T11:20:42.992 INFO:tasks.rados.rados.0.trial043.stdout:1126: oid 11 updating version 0 to 374 2026-02-20T11:20:42.992 INFO:tasks.rados.rados.0.trial043.stdout:1126: oid 11 version 374 is already newer than 374 2026-02-20T11:20:42.992 INFO:tasks.rados.rados.0.trial043.stdout:1126: oid 11 version 374 is already newer than 374 2026-02-20T11:20:42.992 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 11 v 374 (ObjNum 515 snap 145 seq_num 515) dirty exists 2026-02-20T11:20:42.992 INFO:tasks.rados.rados.0.trial043.stdout:1126: left oid 11 (ObjNum 515 snap 145 seq_num 515) 2026-02-20T11:20:42.992 INFO:tasks.rados.rados.0.trial043.stdout:1129: finishing write tid 1 to trial04342147-30 2026-02-20T11:20:42.992 INFO:tasks.rados.rados.0.trial043.stdout:1130: finishing write tid 1 to trial04342147-34 2026-02-20T11:20:42.993 INFO:tasks.rados.rados.0.trial043.stdout:1131: finishing write tid 1 to trial04342147-15 2026-02-20T11:20:42.993 INFO:tasks.rados.rados.0.trial043.stdout:1129: finishing write tid 2 to trial04342147-30 2026-02-20T11:20:42.993 INFO:tasks.rados.rados.0.trial043.stdout:1134: finishing write tid 1 to trial04342147-13 2026-02-20T11:20:42.993 INFO:tasks.rados.rados.0.trial043.stdout:1129: finishing write tid 3 to trial04342147-30 2026-02-20T11:20:42.993 INFO:tasks.rados.rados.0.trial043.stdout:1129: finishing write tid 4 to trial04342147-30 2026-02-20T11:20:42.993 INFO:tasks.rados.rados.0.trial043.stdout:1129: oid 30 updating version 0 to 53 2026-02-20T11:20:42.993 INFO:tasks.rados.rados.0.trial043.stdout:1129: oid 30 version 53 is already newer than 53 2026-02-20T11:20:42.993 INFO:tasks.rados.rados.0.trial043.stdout:1129: oid 30 version 53 is already newer than 51 2026-02-20T11:20:42.993 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 30 v 53 (ObjNum 516 snap 145 seq_num 516) dirty exists 2026-02-20T11:20:42.993 INFO:tasks.rados.rados.0.trial043.stdout:1129: left oid 30 (ObjNum 516 snap 145 seq_num 516) 2026-02-20T11:20:42.993 INFO:tasks.rados.rados.0.trial043.stdout:1132: expect (ObjNum 475 snap 129 seq_num 475) 2026-02-20T11:20:43.001 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1293450 2026-02-20T11:20:43.045 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:20:43.116 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 1408310 2026-02-20T11:20:43.162 INFO:tasks.rados.rados.0.trial043.stdout:1131: finishing write tid 2 to trial04342147-15 2026-02-20T11:20:43.162 INFO:tasks.rados.rados.0.trial043.stdout:1126: done (11 left) 2026-02-20T11:20:43.163 INFO:tasks.rados.rados.0.trial043.stdout:1129: done (10 left) 2026-02-20T11:20:43.163 INFO:tasks.rados.rados.0.trial043.stdout:1132: done (9 left) 2026-02-20T11:20:43.163 INFO:tasks.rados.rados.0.trial043.stdout:1139: done (8 left) 2026-02-20T11:20:43.163 INFO:tasks.rados.rados.0.trial043.stdout:1140: write oid 37 current snap is 145 2026-02-20T11:20:43.163 INFO:tasks.rados.rados.0.trial043.stdout:1130: finishing write tid 2 to trial04342147-34 2026-02-20T11:20:43.163 INFO:tasks.rados.rados.0.trial043.stdout:1140: seq_num 523 ranges {790061=467231,2004906=541501} 2026-02-20T11:20:43.167 INFO:tasks.rados.rados.0.trial043.stdout:1140: writing trial04342147-37 from 790061 to 1257292 tid 1 2026-02-20T11:20:43.172 INFO:tasks.rados.rados.0.trial043.stdout:1140: writing trial04342147-37 from 2004906 to 2546407 tid 2 2026-02-20T11:20:43.172 INFO:tasks.rados.rados.0.trial043.stdout:1141: write (excl) oid 45 current snap is 145 2026-02-20T11:20:43.172 INFO:tasks.rados.rados.0.trial043.stdout:1131: finishing write tid 3 to trial04342147-15 2026-02-20T11:20:43.172 INFO:tasks.rados.rados.0.trial043.stdout:1141: seq_num 524 ranges {406868=542480,1547035=401512,2690354=116514} 2026-02-20T11:20:43.175 INFO:tasks.rados.rados.0.trial043.stdout:1141: writing trial04342147-45 from 406868 to 949348 tid 1 2026-02-20T11:20:43.179 INFO:tasks.rados.rados.0.trial043.stdout:1141: writing trial04342147-45 from 1547035 to 1948547 tid 2 2026-02-20T11:20:43.183 INFO:tasks.rados.rados.0.trial043.stdout:1141: writing trial04342147-45 from 2690354 to 2806868 tid 3 2026-02-20T11:20:43.184 INFO:tasks.rados.rados.0.trial043.stdout:1142: read oid 47 snap 142 2026-02-20T11:20:43.184 INFO:tasks.rados.rados.0.trial043.stdout:1142: expect (ObjNum 504 snap 141 seq_num 504) 2026-02-20T11:20:43.184 INFO:tasks.rados.rados.0.trial043.stdout:1143: snap_create 2026-02-20T11:20:43.184 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 11 2026-02-20T11:20:43.184 INFO:tasks.rados.rados.0.trial043.stdout:1131: finishing write tid 4 to trial04342147-15 2026-02-20T11:20:43.184 INFO:tasks.rados.rados.0.trial043.stdout:1131: oid 15 updating version 0 to 82 2026-02-20T11:20:43.184 INFO:tasks.rados.rados.0.trial043.stdout:1131: oid 15 version 82 is already newer than 80 2026-02-20T11:20:43.184 INFO:tasks.rados.rados.0.trial043.stdout:1131: oid 15 version 82 is already newer than 82 2026-02-20T11:20:43.184 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 15 v 82 (ObjNum 518 snap 145 seq_num 518) dirty exists 2026-02-20T11:20:43.184 INFO:tasks.rados.rados.0.trial043.stdout:1131: left oid 15 (ObjNum 518 snap 145 seq_num 518) 2026-02-20T11:20:43.184 INFO:tasks.rados.rados.0.trial043.stdout:1131: done (10 left) 2026-02-20T11:20:43.185 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 10 2026-02-20T11:20:43.185 INFO:tasks.rados.rados.0.trial043.stdout:1130: finishing write tid 3 to trial04342147-34 2026-02-20T11:20:43.185 INFO:tasks.rados.rados.0.trial043.stdout:1134: finishing write tid 2 to trial04342147-13 2026-02-20T11:20:43.185 INFO:tasks.rados.rados.0.trial043.stdout:1133: expect (ObjNum 476 snap 134 seq_num 476) 2026-02-20T11:20:43.198 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:20:43.238 INFO:tasks.rados.rados.0.trial043.stdout:1130: finishing write tid 4 to trial04342147-34 2026-02-20T11:20:43.238 INFO:tasks.rados.rados.0.trial043.stdout:1130: oid 34 updating version 0 to 70 2026-02-20T11:20:43.238 INFO:tasks.rados.rados.0.trial043.stdout:1130: oid 34 version 70 is already newer than 68 2026-02-20T11:20:43.238 INFO:tasks.rados.rados.0.trial043.stdout:1130: oid 34 version 70 is already newer than 70 2026-02-20T11:20:43.238 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 34 v 70 (ObjNum 517 snap 145 seq_num 517) dirty exists 2026-02-20T11:20:43.239 INFO:tasks.rados.rados.0.trial043.stdout:1130: left oid 34 (ObjNum 517 snap 145 seq_num 517) 2026-02-20T11:20:43.239 INFO:tasks.rados.rados.0.trial043.stdout:1134: finishing write tid 3 to trial04342147-13 2026-02-20T11:20:43.239 INFO:tasks.rados.rados.0.trial043.stdout:1130: done (9 left) 2026-02-20T11:20:43.239 INFO:tasks.rados.rados.0.trial043.stdout:1133: done (8 left) 2026-02-20T11:20:43.239 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 8 2026-02-20T11:20:43.239 INFO:tasks.rados.rados.0.trial043.stdout:1134: finishing write tid 4 to trial04342147-13 2026-02-20T11:20:43.239 INFO:tasks.rados.rados.0.trial043.stdout:1134: oid 13 updating version 0 to 257 2026-02-20T11:20:43.239 INFO:tasks.rados.rados.0.trial043.stdout:1134: oid 13 updating version 257 to 259 2026-02-20T11:20:43.239 INFO:tasks.rados.rados.0.trial043.stdout:1134: oid 13 version 259 is already newer than 259 2026-02-20T11:20:43.239 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 13 v 259 (ObjNum 519 snap 145 seq_num 519) dirty exists 2026-02-20T11:20:43.239 INFO:tasks.rados.rados.0.trial043.stdout:1134: left oid 13 (ObjNum 519 snap 145 seq_num 519) 2026-02-20T11:20:43.239 INFO:tasks.rados.rados.0.trial043.stdout:1135: finishing write tid 1 to trial04342147-20 2026-02-20T11:20:43.240 INFO:tasks.rados.rados.0.trial043.stdout:1134: done (7 left) 2026-02-20T11:20:43.240 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 7 2026-02-20T11:20:43.240 INFO:tasks.rados.rados.0.trial043.stdout:1135: finishing write tid 2 to trial04342147-20 2026-02-20T11:20:43.240 INFO:tasks.rados.rados.0.trial043.stdout:1135: finishing write tid 3 to trial04342147-20 2026-02-20T11:20:43.240 INFO:tasks.rados.rados.0.trial043.stdout:1135: finishing write tid 4 to trial04342147-20 2026-02-20T11:20:43.240 INFO:tasks.rados.rados.0.trial043.stdout:1135: oid 20 updating version 0 to 263 2026-02-20T11:20:43.240 INFO:tasks.rados.rados.0.trial043.stdout:1135: oid 20 version 263 is already newer than 263 2026-02-20T11:20:43.240 INFO:tasks.rados.rados.0.trial043.stdout:1135: oid 20 version 263 is already newer than 263 2026-02-20T11:20:43.240 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 20 v 263 (ObjNum 520 snap 145 seq_num 520) dirty exists 2026-02-20T11:20:43.240 INFO:tasks.rados.rados.0.trial043.stdout:1135: left oid 20 (ObjNum 520 snap 145 seq_num 520) 2026-02-20T11:20:43.240 INFO:tasks.rados.rados.0.trial043.stdout:1135: done (6 left) 2026-02-20T11:20:43.240 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 6 2026-02-20T11:20:43.241 INFO:tasks.rados.rados.0.trial043.stdout:1137: expect (ObjNum 510 snap 144 seq_num 510) 2026-02-20T11:20:43.258 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:20:43.265 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 314420 2026-02-20T11:20:43.291 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:20:43.319 INFO:tasks.rados.rados.0.trial043.stdout:1136: finishing write tid 1 to trial04342147-44 2026-02-20T11:20:43.319 INFO:tasks.rados.rados.0.trial043.stdout:1136: finishing write tid 2 to trial04342147-44 2026-02-20T11:20:43.319 INFO:tasks.rados.rados.0.trial043.stdout:1136: finishing write tid 3 to trial04342147-44 2026-02-20T11:20:43.319 INFO:tasks.rados.rados.0.trial043.stdout:1136: finishing write tid 4 to trial04342147-44 2026-02-20T11:20:43.319 INFO:tasks.rados.rados.0.trial043.stdout:1136: oid 44 updating version 0 to 169 2026-02-20T11:20:43.319 INFO:tasks.rados.rados.0.trial043.stdout:1136: oid 44 updating version 169 to 171 2026-02-20T11:20:43.319 INFO:tasks.rados.rados.0.trial043.stdout:1136: oid 44 version 171 is already newer than 171 2026-02-20T11:20:43.320 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 44 v 171 (ObjNum 521 snap 145 seq_num 521) dirty exists 2026-02-20T11:20:43.320 INFO:tasks.rados.rados.0.trial043.stdout:1136: left oid 44 (ObjNum 521 snap 145 seq_num 521) 2026-02-20T11:20:43.320 INFO:tasks.rados.rados.0.trial043.stdout:1140: finishing write tid 1 to trial04342147-37 2026-02-20T11:20:43.320 INFO:tasks.rados.rados.0.trial043.stdout:1136: done (5 left) 2026-02-20T11:20:43.320 INFO:tasks.rados.rados.0.trial043.stdout:1137: done (4 left) 2026-02-20T11:20:43.320 INFO:tasks.rados.rados.0.trial043.stdout:1138: done (3 left) 2026-02-20T11:20:43.320 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 3 2026-02-20T11:20:43.320 INFO:tasks.rados.rados.0.trial043.stdout:1140: finishing write tid 2 to trial04342147-37 2026-02-20T11:20:43.320 INFO:tasks.rados.rados.0.trial043.stdout:1142: expect (ObjNum 504 snap 141 seq_num 504) 2026-02-20T11:20:43.336 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:43.334+0000 7f8565d8c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:43.338 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:43.338+0000 7f8565d8c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:43.340 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:20:43.365 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:20:43.374 INFO:tasks.rados.rados.0.trial043.stdout:1140: finishing write tid 3 to trial04342147-37 2026-02-20T11:20:43.374 INFO:tasks.rados.rados.0.trial043.stdout:1142: done (2 left) 2026-02-20T11:20:43.374 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:20:43.374 INFO:tasks.rados.rados.0.trial043.stdout:1140: finishing write tid 4 to trial04342147-37 2026-02-20T11:20:43.374 INFO:tasks.rados.rados.0.trial043.stdout:1140: oid 37 updating version 0 to 266 2026-02-20T11:20:43.375 INFO:tasks.rados.rados.0.trial043.stdout:1140: oid 37 version 266 is already newer than 266 2026-02-20T11:20:43.375 INFO:tasks.rados.rados.0.trial043.stdout:1140: oid 37 version 266 is already newer than 266 2026-02-20T11:20:43.375 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 37 v 266 (ObjNum 523 snap 145 seq_num 523) dirty exists 2026-02-20T11:20:43.375 INFO:tasks.rados.rados.0.trial043.stdout:1140: left oid 37 (ObjNum 523 snap 145 seq_num 523) 2026-02-20T11:20:43.375 INFO:tasks.rados.rados.0.trial043.stdout:1141: finishing write tid 1 to trial04342147-45 2026-02-20T11:20:43.375 INFO:tasks.rados.rados.0.trial043.stdout:1141: finishing write tid 2 to trial04342147-45 2026-02-20T11:20:43.375 INFO:tasks.rados.rados.0.trial043.stdout:1140: done (1 left) 2026-02-20T11:20:43.375 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:20:43.375 INFO:tasks.rados.rados.0.trial043.stdout:1141: finishing write tid 3 to trial04342147-45 2026-02-20T11:20:43.375 INFO:tasks.rados.rados.0.trial043.stdout:1141: finishing write tid 4 to trial04342147-45 2026-02-20T11:20:43.375 INFO:tasks.rados.rados.0.trial043.stdout:1141: finishing write tid 5 to trial04342147-45 2026-02-20T11:20:43.375 INFO:tasks.rados.rados.0.trial043.stdout:1141: oid 45 updating version 0 to 268 2026-02-20T11:20:43.376 INFO:tasks.rados.rados.0.trial043.stdout:1141: oid 45 updating version 268 to 271 2026-02-20T11:20:43.376 INFO:tasks.rados.rados.0.trial043.stdout:1141: oid 45 version 271 is already newer than 271 2026-02-20T11:20:43.376 INFO:tasks.rados.rados.0.trial043.stdout:1141: oid 45 version 271 is already newer than 271 2026-02-20T11:20:43.376 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 45 v 271 (ObjNum 524 snap 145 seq_num 524) dirty exists 2026-02-20T11:20:43.376 INFO:tasks.rados.rados.0.trial043.stdout:1141: left oid 45 (ObjNum 524 snap 145 seq_num 524) 2026-02-20T11:20:43.376 INFO:tasks.rados.rados.0.trial043.stdout:1141: done (0 left) 2026-02-20T11:20:43.381 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:20:43.381 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:20:43.926 INFO:tasks.rados.rados.0.trial043.stdout:1143: done (0 left) 2026-02-20T11:20:43.926 INFO:tasks.rados.rados.0.trial043.stdout:1144: write oid 34 current snap is 146 2026-02-20T11:20:43.926 INFO:tasks.rados.rados.0.trial043.stdout:1144: seq_num 525 ranges {423675=734082,1721870=545170,2717777=105898} 2026-02-20T11:20:43.931 INFO:tasks.rados.rados.0.trial043.stdout:1144: writing trial04342147-34 from 423675 to 1157757 tid 1 2026-02-20T11:20:43.935 INFO:tasks.rados.rados.0.trial043.stdout:1144: writing trial04342147-34 from 1721870 to 2267040 tid 2 2026-02-20T11:20:43.938 INFO:tasks.rados.rados.0.trial043.stdout:1144: writing trial04342147-34 from 2717777 to 2823675 tid 3 2026-02-20T11:20:43.938 INFO:tasks.rados.rados.0.trial043.stdout:1145: write (excl) oid 29 current snap is 146 2026-02-20T11:20:43.938 INFO:tasks.rados.rados.0.trial043.stdout:1145: seq_num 526 ranges {440482=409331,1380352=405181,2461553=378929} 2026-02-20T11:20:43.941 INFO:tasks.rados.rados.0.trial043.stdout:1145: writing trial04342147-29 from 440482 to 849813 tid 1 2026-02-20T11:20:43.945 INFO:tasks.rados.rados.0.trial043.stdout:1145: writing trial04342147-29 from 1380352 to 1785533 tid 2 2026-02-20T11:20:43.949 INFO:tasks.rados.rados.0.trial043.stdout:1145: writing trial04342147-29 from 2461553 to 2840482 tid 3 2026-02-20T11:20:43.949 INFO:tasks.rados.rados.0.trial043.stdout:1146: snap_create 2026-02-20T11:20:43.949 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:20:43.985 INFO:tasks.rados.rados.0.trial043.stdout:1144: finishing write tid 1 to trial04342147-34 2026-02-20T11:20:43.995 INFO:tasks.rados.rados.0.trial043.stdout:1144: finishing write tid 2 to trial04342147-34 2026-02-20T11:20:44.004 INFO:tasks.rados.rados.0.trial043.stdout:1144: finishing write tid 3 to trial04342147-34 2026-02-20T11:20:44.009 INFO:tasks.rados.rados.0.trial043.stdout:1144: finishing write tid 4 to trial04342147-34 2026-02-20T11:20:44.011 INFO:tasks.rados.rados.0.trial043.stdout:1144: finishing write tid 5 to trial04342147-34 2026-02-20T11:20:44.011 INFO:tasks.rados.rados.0.trial043.stdout:1144: oid 34 updating version 0 to 75 2026-02-20T11:20:44.012 INFO:tasks.rados.rados.0.trial043.stdout:1144: oid 34 version 75 is already newer than 75 2026-02-20T11:20:44.012 INFO:tasks.rados.rados.0.trial043.stdout:1144: oid 34 version 75 is already newer than 75 2026-02-20T11:20:44.012 INFO:tasks.rados.rados.0.trial043.stdout:1144: oid 34 version 75 is already newer than 75 2026-02-20T11:20:44.012 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 34 v 75 (ObjNum 525 snap 146 seq_num 525) dirty exists 2026-02-20T11:20:44.012 INFO:tasks.rados.rados.0.trial043.stdout:1144: left oid 34 (ObjNum 525 snap 146 seq_num 525) 2026-02-20T11:20:44.012 INFO:tasks.rados.rados.0.trial043.stdout:1144: done (1 left) 2026-02-20T11:20:44.012 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:20:44.022 INFO:tasks.rados.rados.0.trial043.stdout:1145: finishing write tid 1 to trial04342147-29 2026-02-20T11:20:44.037 INFO:tasks.rados.rados.0.trial043.stdout:1145: finishing write tid 2 to trial04342147-29 2026-02-20T11:20:44.049 INFO:tasks.rados.rados.0.trial043.stdout:1145: finishing write tid 3 to trial04342147-29 2026-02-20T11:20:44.057 INFO:tasks.rados.rados.0.trial043.stdout:1145: finishing write tid 4 to trial04342147-29 2026-02-20T11:20:44.059 INFO:tasks.rados.rados.0.trial043.stdout:1145: finishing write tid 5 to trial04342147-29 2026-02-20T11:20:44.059 INFO:tasks.rados.rados.0.trial043.stdout:1145: oid 29 updating version 0 to 376 2026-02-20T11:20:44.059 INFO:tasks.rados.rados.0.trial043.stdout:1145: oid 29 updating version 376 to 379 2026-02-20T11:20:44.059 INFO:tasks.rados.rados.0.trial043.stdout:1145: oid 29 version 379 is already newer than 379 2026-02-20T11:20:44.059 INFO:tasks.rados.rados.0.trial043.stdout:1145: oid 29 version 379 is already newer than 379 2026-02-20T11:20:44.059 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 29 v 379 (ObjNum 526 snap 146 seq_num 526) dirty exists 2026-02-20T11:20:44.059 INFO:tasks.rados.rados.0.trial043.stdout:1145: left oid 29 (ObjNum 526 snap 146 seq_num 526) 2026-02-20T11:20:44.060 INFO:tasks.rados.rados.0.trial043.stdout:1145: done (0 left) 2026-02-20T11:20:44.928 INFO:tasks.rados.rados.0.trial043.stdout:1146: done (0 left) 2026-02-20T11:20:44.929 INFO:tasks.rados.rados.0.trial043.stdout:1147: write oid 14 current snap is 147 2026-02-20T11:20:44.929 INFO:tasks.rados.rados.0.trial043.stdout:1147: seq_num 527 ranges {457289=484580,1438834=548839,2772623=84666} 2026-02-20T11:20:44.932 INFO:tasks.rados.rados.0.trial043.stdout:1147: writing trial04342147-14 from 457289 to 941869 tid 1 2026-02-20T11:20:44.936 INFO:tasks.rados.rados.0.trial043.stdout:1147: writing trial04342147-14 from 1438834 to 1987673 tid 2 2026-02-20T11:20:44.939 INFO:tasks.rados.rados.0.trial043.stdout:1147: writing trial04342147-14 from 2772623 to 2857289 tid 3 2026-02-20T11:20:44.940 INFO:tasks.rados.rados.0.trial043.stdout:1148: snap_create 2026-02-20T11:20:44.940 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:20:44.981 INFO:tasks.rados.rados.0.trial043.stdout:1147: finishing write tid 1 to trial04342147-14 2026-02-20T11:20:44.988 INFO:tasks.rados.rados.0.trial043.stdout:1147: finishing write tid 2 to trial04342147-14 2026-02-20T11:20:44.993 INFO:tasks.rados.rados.0.trial043.stdout:1147: finishing write tid 3 to trial04342147-14 2026-02-20T11:20:44.998 INFO:tasks.rados.rados.0.trial043.stdout:1147: finishing write tid 4 to trial04342147-14 2026-02-20T11:20:45.000 INFO:tasks.rados.rados.0.trial043.stdout:1147: finishing write tid 5 to trial04342147-14 2026-02-20T11:20:45.001 INFO:tasks.rados.rados.0.trial043.stdout:1147: oid 14 updating version 0 to 335 2026-02-20T11:20:45.001 INFO:tasks.rados.rados.0.trial043.stdout:1147: oid 14 version 335 is already newer than 335 2026-02-20T11:20:45.001 INFO:tasks.rados.rados.0.trial043.stdout:1147: oid 14 version 335 is already newer than 335 2026-02-20T11:20:45.001 INFO:tasks.rados.rados.0.trial043.stdout:1147: oid 14 version 335 is already newer than 335 2026-02-20T11:20:45.001 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 14 v 335 (ObjNum 527 snap 147 seq_num 527) dirty exists 2026-02-20T11:20:45.001 INFO:tasks.rados.rados.0.trial043.stdout:1147: left oid 14 (ObjNum 527 snap 147 seq_num 527) 2026-02-20T11:20:45.001 INFO:tasks.rados.rados.0.trial043.stdout:1147: done (0 left) 2026-02-20T11:20:45.382 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:20:45.459 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:45.457+0000 7f60a0cb7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:45.462 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:45.462+0000 7f60a0cb7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:45.467 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:20:45.488 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:20:45.504 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:20:45.504 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:20:45.537 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:20:45.539 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:20:45.540 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:20:45.933 INFO:tasks.rados.rados.0.trial043.stdout:1148: done (0 left) 2026-02-20T11:20:45.933 INFO:tasks.rados.rados.0.trial043.stdout:1149: write oid 42 current snap is 148 2026-02-20T11:20:45.933 INFO:tasks.rados.rados.0.trial043.stdout:1149: seq_num 528 ranges {474096=559829,1780963=408850,2800046=74050} 2026-02-20T11:20:45.936 INFO:tasks.rados.rados.0.trial043.stdout:1149: writing trial04342147-42 from 474096 to 1033925 tid 1 2026-02-20T11:20:45.941 INFO:tasks.rados.rados.0.trial043.stdout:1149: writing trial04342147-42 from 1780963 to 2189813 tid 2 2026-02-20T11:20:45.944 INFO:tasks.rados.rados.0.trial043.stdout:1149: writing trial04342147-42 from 2800046 to 2874096 tid 3 2026-02-20T11:20:45.944 INFO:tasks.rados.rados.0.trial043.stdout:1150: snap_remove snap 146 2026-02-20T11:20:46.938 INFO:tasks.rados.rados.0.trial043.stdout:1150: done (1 left) 2026-02-20T11:20:46.938 INFO:tasks.rados.rados.0.trial043.stdout:1151: read oid 39 snap -1 2026-02-20T11:20:46.938 INFO:tasks.rados.rados.0.trial043.stdout:1151: expect (ObjNum 425 snap 116 seq_num 425) 2026-02-20T11:20:46.938 INFO:tasks.rados.rados.0.trial043.stdout:1152: snap_create 2026-02-20T11:20:46.938 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 2 2026-02-20T11:20:46.938 INFO:tasks.rados.rados.0.trial043.stdout:1149: finishing write tid 1 to trial04342147-42 2026-02-20T11:20:46.939 INFO:tasks.rados.rados.0.trial043.stdout:1149: finishing write tid 2 to trial04342147-42 2026-02-20T11:20:46.939 INFO:tasks.rados.rados.0.trial043.stdout:1149: finishing write tid 3 to trial04342147-42 2026-02-20T11:20:46.939 INFO:tasks.rados.rados.0.trial043.stdout:1149: finishing write tid 4 to trial04342147-42 2026-02-20T11:20:46.939 INFO:tasks.rados.rados.0.trial043.stdout:1149: finishing write tid 5 to trial04342147-42 2026-02-20T11:20:46.939 INFO:tasks.rados.rados.0.trial043.stdout:1149: oid 42 updating version 0 to 176 2026-02-20T11:20:46.939 INFO:tasks.rados.rados.0.trial043.stdout:1149: oid 42 version 176 is already newer than 176 2026-02-20T11:20:46.939 INFO:tasks.rados.rados.0.trial043.stdout:1149: oid 42 version 176 is already newer than 176 2026-02-20T11:20:46.939 INFO:tasks.rados.rados.0.trial043.stdout:1149: oid 42 version 176 is already newer than 176 2026-02-20T11:20:46.939 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 42 v 176 (ObjNum 528 snap 148 seq_num 528) dirty exists 2026-02-20T11:20:46.939 INFO:tasks.rados.rados.0.trial043.stdout:1149: left oid 42 (ObjNum 528 snap 148 seq_num 528) 2026-02-20T11:20:46.939 INFO:tasks.rados.rados.0.trial043.stdout:1149: done (1 left) 2026-02-20T11:20:46.940 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:20:46.979 INFO:tasks.rados.rados.0.trial043.stdout:1151: expect (ObjNum 425 snap 116 seq_num 425) 2026-02-20T11:20:47.032 INFO:tasks.rados.rados.0.trial043.stdout:seeking to 0 2026-02-20T11:20:47.059 INFO:tasks.rados.rados.0.trial043.stdout:1151: done (0 left) 2026-02-20T11:20:47.506 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:20:47.583 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:47.581+0000 7f11a2a35640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:47.585 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:20:47.584+0000 7f11a2a35640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:20:47.610 INFO:teuthology.orchestra.run.trial043.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:20:47.627 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:20:47.627 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:20:47.938 INFO:tasks.rados.rados.0.trial043.stdout:1152: done (0 left) 2026-02-20T11:20:47.938 INFO:tasks.rados.rados.0.trial043.stdout:1153: write oid 37 current snap is 149 2026-02-20T11:20:47.938 INFO:tasks.rados.rados.0.trial043.stdout:1153: seq_num 529 ranges {490903=635078,1839445=552508} 2026-02-20T11:20:47.941 INFO:tasks.rados.rados.0.trial043.stdout:1153: writing trial04342147-37 from 490903 to 1125981 tid 1 2026-02-20T11:20:47.947 INFO:tasks.rados.rados.0.trial043.stdout:1153: writing trial04342147-37 from 1839445 to 2391953 tid 2 2026-02-20T11:20:47.947 INFO:tasks.rados.rados.0.trial043.stdout:1154: snap_create 2026-02-20T11:20:47.947 INFO:tasks.rados.rados.0.trial043.stdout: waiting on 1 2026-02-20T11:20:47.989 INFO:tasks.rados.rados.0.trial043.stdout:1153: finishing write tid 1 to trial04342147-37 2026-02-20T11:20:47.996 INFO:tasks.rados.rados.0.trial043.stdout:1153: finishing write tid 2 to trial04342147-37 2026-02-20T11:20:48.001 INFO:tasks.rados.rados.0.trial043.stdout:1153: finishing write tid 3 to trial04342147-37 2026-02-20T11:20:48.003 INFO:tasks.rados.rados.0.trial043.stdout:1153: finishing write tid 4 to trial04342147-37 2026-02-20T11:20:48.004 INFO:tasks.rados.rados.0.trial043.stdout:1153: oid 37 updating version 0 to 275 2026-02-20T11:20:48.004 INFO:tasks.rados.rados.0.trial043.stdout:1153: oid 37 version 275 is already newer than 275 2026-02-20T11:20:48.004 INFO:tasks.rados.rados.0.trial043.stdout:1153: oid 37 version 275 is already newer than 275 2026-02-20T11:20:48.004 INFO:tasks.rados.rados.0.trial043.stdout:update_object_version oid 37 v 275 (ObjNum 529 snap 149 seq_num 529) dirty exists 2026-02-20T11:20:48.004 INFO:tasks.rados.rados.0.trial043.stdout:1153: left oid 37 (ObjNum 529 snap 149 seq_num 529) 2026-02-20T11:20:48.004 INFO:tasks.rados.rados.0.trial043.stdout:1153: done (0 left) 2026-02-20T11:20:48.060 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~303s 2026-02-20T11:20:48.060 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2026-02-20T11:20:48.564 INFO:tasks.ceph.osd.0:Sent signal 15 2026-02-20T11:20:48.564 INFO:tasks.ceph.osd.1:Sent signal 15 2026-02-20T11:20:48.565 INFO:tasks.ceph.osd.3:Sent signal 15 2026-02-20T11:20:48.565 INFO:tasks.ceph.mon.a:Sent signal 15 2026-02-20T11:20:48.565 INFO:tasks.ceph.mgr.x:Sent signal 15 2026-02-20T11:20:48.565 INFO:tasks.daemonwatchdog.daemon_watchdog:Killing thrasher OSDThrasher 2026-02-20T11:20:48.566 INFO:tasks.ceph.mon.a.trial043.stderr:2026-02-20T11:20:48.565+0000 7fd0d51fc640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 40033) UID: 0 2026-02-20T11:20:48.566 INFO:tasks.ceph.mon.a.trial043.stderr:2026-02-20T11:20:48.565+0000 7fd0d51fc640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2026-02-20T11:20:48.567 INFO:tasks.ceph.mgr.x.trial109.stderr:2026-02-20T11:20:48.564+0000 7f6de05f8640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster ceph -i x (PID: 39117) UID: 0 2026-02-20T11:20:48.567 INFO:tasks.ceph.mgr.x.trial109.stderr:2026-02-20T11:20:48.564+0000 7f6de05f8640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2026-02-20T11:20:50.541 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:20:50.614 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:20:50.617 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:20:50.617 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:20:55.619 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:20:55.690 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:20:55.692 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:20:55.693 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:21:00.694 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:21:00.766 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:21:00.768 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:21:00.769 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:21:05.770 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:21:05.843 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:21:05.846 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:21:05.847 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:21:10.847 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:21:10.919 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:21:10.922 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:21:10.923 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:21:15.923 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:21:15.995 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:21:15.998 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:21:15.999 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:21:21.000 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:21:21.072 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:21:21.075 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:21:21.076 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:21:26.077 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:21:26.150 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:21:26.153 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:21:26.153 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:21:31.155 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:21:31.227 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:21:31.230 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:21:31.231 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:21:36.232 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:21:36.304 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:21:36.307 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:21:36.308 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:21:41.308 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:21:41.379 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:21:41.382 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:21:41.382 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:21:46.383 DEBUG:teuthology.orchestra.run.trial109:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:21:46.454 INFO:teuthology.orchestra.run.trial109.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:21:46.456 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:21:46.458 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 192, in wrapper return func(self) ^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 1513, in _do_thrash self.choose_action()() File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 562, in revive_osd self.ceph_manager.revive_osd( File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 3236, in revive_osd self.wait_run_admin_socket('osd', osd, File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 2085, in wait_run_admin_socket raise Exception('timed out waiting for admin_socket ' Exception: timed out waiting for admin_socket to appear after osd.2 restart 2026-02-20T11:21:46.458 ERROR:tasks.thrashosds.thrasher:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 1384, in do_thrash self._do_thrash() File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 192, in wrapper return func(self) ^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 1513, in _do_thrash self.choose_action()() File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 562, in revive_osd self.ceph_manager.revive_osd( File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 3236, in revive_osd self.wait_run_admin_socket('osd', osd, File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 2085, in wait_run_admin_socket raise Exception('timed out waiting for admin_socket ' Exception: timed out waiting for admin_socket to appear after osd.2 restart 2026-02-20T11:21:46.458 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2026-02-20T11:21:46.458 INFO:tasks.daemonwatchdog.daemon_watchdog:Killing remote process ceph-test-rados-ceph 2026-02-20T11:21:46.458 DEBUG:tasks.rados:Stopping ceph-test-rados-ceph due to exception The WatchDog barked due to Daemon ceph.osd.2 has failed 2026-02-20T11:21:46.458 INFO:tasks.rados:Stopping instance 0 2026-02-20T11:21:46.471 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/rados.py", line 352, in thread raise watched_process.exception tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.2 has failed 2026-02-20T11:21:46.472 ERROR:teuthology.run_tasks:Manager failed: rados Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.2 has failed 2026-02-20T11:21:46.473 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2026-02-20T11:21:46.483 INFO:tasks.thrashosds:joining thrashosds 2026-02-20T11:21:46.483 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2026-02-20T11:21:46.483 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-20T11:21:46.483 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-20T11:21:46.590 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:21:46.586+0000 7fce97211640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:23:46.546 DEBUG:teuthology.orchestra.run:got remote process result: 124 2026-02-20T11:23:46.553 ERROR:teuthology.run_tasks:Manager failed: thrashosds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/thrashosds.py", line 214, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.2 has failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 158, in __exit__ self.gen.throw(value) File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 2970, in wait_for_all_osds_up while not self.are_all_osds_up(): ^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 2960, in are_all_osds_up x = self.get_osd_dump() ^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 2670, in get_osd_dump return self.get_osd_dump_json()['osds'] ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 2662, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 1713, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 1704, in run_cluster_cmd return self.controller.run(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/remote.py", line 596, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/run.py", line 461, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial043 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2026-02-20T11:23:46.555 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-20T11:23:46.566 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-20T11:23:46.566 DEBUG:teuthology.orchestra.run.trial043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-20T11:23:46.643 INFO:teuthology.orchestra.run.trial043.stderr:2026-02-20T11:23:46.639+0000 7f2534acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:25:46.598 DEBUG:teuthology.orchestra.run:got remote process result: 124 2026-02-20T11:25:46.599 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph.py", line 2000, in task yield File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/thrashosds.py", line 214, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.2 has failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/contextutil.py", line 34, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph.py", line 2010, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 2936, in wait_for_clean num_active_clean = self.get_num_active_clean() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 2715, in get_num_active_clean pgs = self.get_pg_stats() ^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 2481, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 1713, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 1704, in run_cluster_cmd return self.controller.run(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/remote.py", line 596, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/run.py", line 461, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial043 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2026-02-20T11:25:46.604 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-20T11:25:46.604 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-20T11:25:46.604 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-20T11:25:46.604 INFO:teuthology.orchestra.run:waiting for 300 2026-02-20T11:25:46.668 INFO:tasks.ceph.osd.0:Stopped 2026-02-20T11:25:46.668 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-20T11:25:46.668 INFO:teuthology.orchestra.run:waiting for 300 2026-02-20T11:25:46.745 INFO:tasks.ceph.osd.1:Stopped 2026-02-20T11:25:46.745 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-20T11:25:46.745 INFO:teuthology.orchestra.run:waiting for 300 2026-02-20T11:25:46.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-20T11:25:46.745 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph.py", line 2000, in task yield File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/thrashosds.py", line 214, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.2 has failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph.py", line 1525, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/contextutil.py", line 34, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph.py", line 2010, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 2936, in wait_for_clean num_active_clean = self.get_num_active_clean() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 2715, in get_num_active_clean pgs = self.get_pg_stats() ^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 2481, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 1713, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 1704, in run_cluster_cmd return self.controller.run(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/remote.py", line 596, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/run.py", line 461, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial043 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/run.py", line 485, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial109 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log' 2026-02-20T11:25:46.751 INFO:tasks.ceph.osd.2:Stopped 2026-02-20T11:25:46.751 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2026-02-20T11:25:46.751 INFO:teuthology.orchestra.run:waiting for 300 2026-02-20T11:25:46.812 INFO:tasks.ceph.osd.3:Stopped 2026-02-20T11:25:46.812 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-20T11:25:46.812 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-20T11:25:46.812 INFO:teuthology.orchestra.run:waiting for 300 2026-02-20T11:25:46.812 INFO:tasks.ceph.mgr.x:Stopped 2026-02-20T11:25:46.812 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-20T11:25:46.812 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-20T11:25:46.812 INFO:teuthology.orchestra.run:waiting for 300 2026-02-20T11:25:46.812 INFO:tasks.ceph.mon.a:Stopped 2026-02-20T11:25:46.812 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-20T11:25:46.812 DEBUG:teuthology.orchestra.run.trial043:> sudo grep -E '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | grep -E -v '\(MDS_ALL_DOWN\)' | grep -E -v '\(MDS_UP_LESS_THAN_MAX\)' | grep -E -v '\(POOL_APP_NOT_ENABLED\)' | grep -E -v 'but it is still running' | grep -E -v 'objects unfound and apparently lost' | grep -E -v 'overall HEALTH_' | grep -E -v '\(OSDMAP_FLAGS\)' | grep -E -v '\(OSD_' | grep -E -v '\(PG_' | grep -E -v '\(POOL_' | grep -E -v '\(CACHE_POOL_' | grep -E -v '\(SMALLER_PGP_NUM\)' | grep -E -v '\(OBJECT_' | grep -E -v SLOW_OPS | grep -E -v '\(REQUEST_SLOW\)' | grep -E -v '\(TOO_FEW_PGS\)' | grep -E -v 'slow request' | grep -E -v 'timeout on replica' | grep -E -v 'late reservation from' | grep -E -v MON_DOWN | grep -E -v OSDMAP_FLAGS | grep -E -v OSD_DOWN | grep -E -v PG_DEGRADED | grep -E -v PG_AVAILABILITY | grep -E -v POOL_APP_NOT_ENABLED | grep -E -v 'mons down' | grep -E -v 'mon down' | grep -E -v 'out of quorum' | grep -E -v noscrub | grep -E -v nodeep-scrub | grep -E -v 'Degraded data redundancy' | grep -E -v 'is down' | grep -E -v 'osds down' | grep -E -v 'pg .*? is .*?degraded.*?, acting' | grep -E -v 'pg .*? is stuck' | grep -E -v 'pg degraded' | grep -E -v PG_BACKFILL_FULL | grep -E -v 'Low space hindering backfill .*? backfill_toofull' | grep -E -v OSD_ROOT_DOWN | grep -E -v 'pgs degraded' | grep -E -v 'pgs undersized' | grep -E -v 'is active.*backfill_toofull.*' | head -n 1 2026-02-20T11:25:46.841 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial043.front.sepia.ceph.com 2026-02-20T11:25:46.842 DEBUG:teuthology.orchestra.run.trial043:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-20T11:25:46.907 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial043.front.sepia.ceph.com 2026-02-20T11:25:46.907 DEBUG:teuthology.orchestra.run.trial043:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-20T11:25:46.974 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial109.front.sepia.ceph.com 2026-02-20T11:25:46.974 DEBUG:teuthology.orchestra.run.trial109:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-20T11:25:47.005 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@trial109.front.sepia.ceph.com 2026-02-20T11:25:47.006 DEBUG:teuthology.orchestra.run.trial109:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2026-02-20T11:25:47.072 INFO:tasks.ceph:Archiving mon data... 2026-02-20T11:25:47.074 DEBUG:teuthology.misc:Transferring archived files from trial043:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/jjperez-2026-02-20_10:21:31-crimson-rados-wip-perezjos-crimson-only-19-02-2026-PR64439-distro-debug-trial/60477/data/mon.a.tgz 2026-02-20T11:25:47.074 DEBUG:teuthology.orchestra.run.trial043:> mktemp 2026-02-20T11:25:47.092 INFO:teuthology.orchestra.run.trial043.stdout:/tmp/tmp.mPinL2fE1u 2026-02-20T11:25:47.093 DEBUG:teuthology.orchestra.run.trial043:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.mPinL2fE1u 2026-02-20T11:25:47.207 DEBUG:teuthology.orchestra.run.trial043:> sudo chmod 0666 /tmp/tmp.mPinL2fE1u 2026-02-20T11:25:47.290 DEBUG:teuthology.orchestra.remote:trial043:/tmp/tmp.mPinL2fE1u is 420KB 2026-02-20T11:25:47.386 DEBUG:teuthology.orchestra.run.trial043:> rm -fr /tmp/tmp.mPinL2fE1u 2026-02-20T11:25:47.403 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-20T11:25:47.404 DEBUG:teuthology.orchestra.run.trial043:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-20T11:25:47.446 DEBUG:teuthology.orchestra.run.trial109:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-20T11:25:47.525 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-20T11:25:47.541 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-20T11:25:47.541 INFO:tasks.ceph:Archiving crash dumps... 2026-02-20T11:25:47.545 DEBUG:teuthology.misc:Transferring archived files from trial043:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/jjperez-2026-02-20_10:21:31-crimson-rados-wip-perezjos-crimson-only-19-02-2026-PR64439-distro-debug-trial/60477/remote/trial043/crash 2026-02-20T11:25:47.546 DEBUG:teuthology.orchestra.run.trial043:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-20T11:25:47.589 DEBUG:teuthology.misc:Transferring archived files from trial109:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/jjperez-2026-02-20_10:21:31-crimson-rados-wip-perezjos-crimson-only-19-02-2026-PR64439-distro-debug-trial/60477/remote/trial109/crash 2026-02-20T11:25:47.590 DEBUG:teuthology.orchestra.run.trial109:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-20T11:25:47.615 INFO:tasks.ceph:Compressing logs... 2026-02-20T11:25:47.616 DEBUG:teuthology.orchestra.run.trial043:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-20T11:25:47.635 DEBUG:teuthology.orchestra.run.trial109:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-20T11:25:47.657 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49380.log 2026-02-20T11:25:47.657 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44299.log 2026-02-20T11:25:47.658 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.49380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41106.log 2026-02-20T11:25:47.658 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49380.log.gz 2026-02-20T11:25:47.658 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.44299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43811.log 2026-02-20T11:25:47.658 INFO:teuthology.orchestra.run.trial043.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.44299.log.gz 2026-02-20T11:25:47.658 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.41106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42367.log 2026-02-20T11:25:47.658 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41106.log.gz 2026-02-20T11:25:47.658 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.43811.log: gzip -5 50.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.43811.log.gz /var/log/ceph/ceph-client.admin.43217.log 2026-02-20T11:25:47.658 INFO:teuthology.orchestra.run.trial043.stderr: 2026-02-20T11:25:47.658 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.42367.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-20T11:25:47.658 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42367.log.gz 2026-02-20T11:25:47.658 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.43217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44840.log 2026-02-20T11:25:47.658 INFO:teuthology.orchestra.run.trial043.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.43217.log.gz 2026-02-20T11:25:47.659 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49488.log 2026-02-20T11:25:47.687 INFO:teuthology.orchestra.run.trial109.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-20T11:25:47.687 INFO:teuthology.orchestra.run.trial109.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2026-02-20T11:25:47.687 INFO:teuthology.orchestra.run.trial109.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39264.log 2026-02-20T11:25:47.687 INFO:teuthology.orchestra.run.trial109.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39369.log 2026-02-20T11:25:47.688 INFO:teuthology.orchestra.run.trial109.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39217.log 2026-02-20T11:25:47.688 INFO:teuthology.orchestra.run.trial109.stderr:/var/log/ceph/ceph-client.admin.39264.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39264.log.gz 2026-02-20T11:25:47.688 INFO:teuthology.orchestra.run.trial109.stderr:/var/log/ceph/ceph-client.admin.39369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39316.log 2026-02-20T11:25:47.688 INFO:teuthology.orchestra.run.trial109.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39369.log.gz 2026-02-20T11:25:47.688 INFO:teuthology.orchestra.run.trial109.stderr:/var/log/ceph/ceph-client.admin.39217.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-20T11:25:47.688 INFO:teuthology.orchestra.run.trial109.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39217.log.gz 2026-02-20T11:25:47.688 INFO:teuthology.orchestra.run.trial109.stderr:/var/log/ceph/ceph-client.admin.39316.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39316.log.gz 2026-02-20T11:25:47.705 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.44840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46517.log 2026-02-20T11:25:47.706 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44840.log.gz 2026-02-20T11:25:47.706 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.49488.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.49488.log.gz 2026-02-20T11:25:47.706 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40377.log 2026-02-20T11:25:47.706 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.46517.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-20T11:25:47.706 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46517.log.gz 2026-02-20T11:25:47.706 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46949.log 2026-02-20T11:25:47.706 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.40377.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40377.log.gz 2026-02-20T11:25:47.706 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49866.log 2026-02-20T11:25:47.706 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.46949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46895.log 2026-02-20T11:25:47.706 INFO:teuthology.orchestra.run.trial043.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.46949.log.gz 2026-02-20T11:25:47.706 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.49866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42289.log 2026-02-20T11:25:47.706 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49866.log.gz 2026-02-20T11:25:47.706 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46841.log 2026-02-20T11:25:47.706 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.46895.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46895.log.gz 2026-02-20T11:25:47.706 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.42289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48137.log 2026-02-20T11:25:47.706 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42289.log.gz 2026-02-20T11:25:47.706 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.46841.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46841.log.gz 2026-02-20T11:25:47.706 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42017.log 2026-02-20T11:25:47.706 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45003.log 2026-02-20T11:25:47.706 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.48137.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48137.log.gz 2026-02-20T11:25:47.706 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.42017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46085.log 2026-02-20T11:25:47.707 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42017.log.gz 2026-02-20T11:25:47.707 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.45003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42729.log 2026-02-20T11:25:47.707 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45003.log.gz 2026-02-20T11:25:47.707 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.46085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46679.log 2026-02-20T11:25:47.707 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46085.log.gz 2026-02-20T11:25:47.707 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.42729.log: 48.0% -- replaced with /var/log/ceph/ceph-client.admin.42729.log.gz 2026-02-20T11:25:47.707 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44191.log 2026-02-20T11:25:47.707 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45274.log 2026-02-20T11:25:47.707 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.46679.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46679.log.gz 2026-02-20T11:25:47.707 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.44191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47705.log 2026-02-20T11:25:47.707 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44191.log.gz 2026-02-20T11:25:47.707 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.45274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45706.log 2026-02-20T11:25:47.707 INFO:teuthology.orchestra.run.trial043.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.45274.log.gz 2026-02-20T11:25:47.707 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48461.log 2026-02-20T11:25:47.707 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.47705.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47705.log.gz 2026-02-20T11:25:47.707 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.45706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41814.log 2026-02-20T11:25:47.707 INFO:teuthology.orchestra.run.trial043.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.45706.log.gz 2026-02-20T11:25:47.707 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.48461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46733.log 2026-02-20T11:25:47.707 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48461.log.gz 2026-02-20T11:25:47.707 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.41814.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41814.log.gz 2026-02-20T11:25:47.707 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43271.log 2026-02-20T11:25:47.707 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44894.log 2026-02-20T11:25:47.707 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.46733.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46733.log.gz 2026-02-20T11:25:47.707 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.43271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45328.log 2026-02-20T11:25:47.708 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43271.log.gz 2026-02-20T11:25:47.708 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.44894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.42147.log 2026-02-20T11:25:47.708 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44894.log.gz 2026-02-20T11:25:47.708 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.45328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40403.log 2026-02-20T11:25:47.708 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45328.log.gz 2026-02-20T11:25:47.708 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.0.42147.log: 50.0% -- replaced with /var/log/ceph/ceph-client.0.42147.log.gz 2026-02-20T11:25:47.708 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45220.log 2026-02-20T11:25:47.708 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48894.log 2026-02-20T11:25:47.708 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.40403.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40403.log.gz 2026-02-20T11:25:47.708 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.45220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46139.log 2026-02-20T11:25:47.708 INFO:teuthology.orchestra.run.trial043.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.45220.log.gz 2026-02-20T11:25:47.708 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.48894.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48894.log.gz 2026-02-20T11:25:47.708 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40908.log 2026-02-20T11:25:47.708 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46193.log 2026-02-20T11:25:47.708 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.46139.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46139.log.gz 2026-02-20T11:25:47.708 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.40908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43379.log 2026-02-20T11:25:47.708 INFO:teuthology.orchestra.run.trial043.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40908.log.gz 2026-02-20T11:25:47.708 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42622.log 2026-02-20T11:25:47.708 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.46193.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.46193.log.gz 2026-02-20T11:25:47.708 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44461.log 2026-02-20T11:25:47.708 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.43379.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43379.log.gz 2026-02-20T11:25:47.708 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.42622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47327.log 2026-02-20T11:25:47.708 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42622.log.gz 2026-02-20T11:25:47.708 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.44461.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44461.log.gz 2026-02-20T11:25:47.708 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44137.log 2026-02-20T11:25:47.709 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47057.log 2026-02-20T11:25:47.709 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.47327.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47327.log.gz 2026-02-20T11:25:47.709 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.44137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43433.log 2026-02-20T11:25:47.709 INFO:teuthology.orchestra.run.trial043.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.44137.log.gz 2026-02-20T11:25:47.709 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.47057.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47057.log.gz 2026-02-20T11:25:47.709 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43163.log 2026-02-20T11:25:47.709 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.43433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49434.log 2026-02-20T11:25:47.709 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43433.log.gz 2026-02-20T11:25:47.709 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.43163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43865.log 2026-02-20T11:25:47.709 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43163.log.gz 2026-02-20T11:25:47.709 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.49434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48299.log 2026-02-20T11:25:47.709 INFO:teuthology.orchestra.run.trial043.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.49434.log.gz 2026-02-20T11:25:47.709 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48083.log 2026-02-20T11:25:47.709 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.43865.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43865.log.gz 2026-02-20T11:25:47.709 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.48299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49650.log 2026-02-20T11:25:47.709 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48299.log.gz 2026-02-20T11:25:47.709 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.48083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48840.log 2026-02-20T11:25:47.709 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48083.log.gz 2026-02-20T11:25:47.709 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.49650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41613.log 2026-02-20T11:25:47.709 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49650.log.gz 2026-02-20T11:25:47.709 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.48840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47219.log 2026-02-20T11:25:47.709 INFO:teuthology.orchestra.run.trial043.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.48840.log.gz 2026-02-20T11:25:47.709 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.41613.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41613.log.gz 2026-02-20T11:25:47.709 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48353.log 2026-02-20T11:25:47.709 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.47219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45652.log 2026-02-20T11:25:47.710 INFO:teuthology.orchestra.run.trial043.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.47219.log.gz 2026-02-20T11:25:47.710 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.48353.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48353.log.gz 2026-02-20T11:25:47.710 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43108.log 2026-02-20T11:25:47.710 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.45652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49218.log 2026-02-20T11:25:47.710 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45652.log.gz 2026-02-20T11:25:47.710 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.43108.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43108.log.gz 2026-02-20T11:25:47.710 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48677.log 2026-02-20T11:25:47.710 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.49218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44515.log 2026-02-20T11:25:47.710 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49218.log.gz 2026-02-20T11:25:47.710 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.48677.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48677.log.gz 2026-02-20T11:25:47.710 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48732.log 2026-02-20T11:25:47.710 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.44515.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44515.log.gz 2026-02-20T11:25:47.710 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46571.log 2026-02-20T11:25:47.710 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.48732.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48732.log.gz 2026-02-20T11:25:47.710 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41689.log 2026-02-20T11:25:47.710 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.46571.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.46571.log.gz 2026-02-20T11:25:47.710 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45166.log 2026-02-20T11:25:47.710 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.41689.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41689.log.gz 2026-02-20T11:25:47.710 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49110.log 2026-02-20T11:25:47.710 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.45166.log: 49.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.45166.log.gz 2026-02-20T11:25:47.710 INFO:teuthology.orchestra.run.trial043.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.49758.log 2026-02-20T11:25:47.710 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.49110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43054.log 2026-02-20T11:25:47.710 INFO:teuthology.orchestra.run.trial043.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.49110.log.gz 2026-02-20T11:25:47.710 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.49758.log: gzip 50.4% -5 -- replaced with /var/log/ceph/ceph-client.admin.49758.log.gz --verbose 2026-02-20T11:25:47.711 INFO:teuthology.orchestra.run.trial043.stderr: -- /var/log/ceph/ceph-client.admin.42044.log 2026-02-20T11:25:47.711 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.43054.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.43054.log.gz 2026-02-20T11:25:47.711 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42892.log 2026-02-20T11:25:47.711 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.42044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46409.log 2026-02-20T11:25:47.711 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42044.log.gz 2026-02-20T11:25:47.711 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.42892.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42892.log.gz 2026-02-20T11:25:47.711 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45977.log 2026-02-20T11:25:47.711 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.46409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49812.log 2026-02-20T11:25:47.711 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46409.log.gz 2026-02-20T11:25:47.711 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.45977.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.45977.log.gz 2026-02-20T11:25:47.711 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42288.log 2026-02-20T11:25:47.711 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.49812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46247.log 2026-02-20T11:25:47.711 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49812.log.gz 2026-02-20T11:25:47.711 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.42288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44027.log 2026-02-20T11:25:47.711 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42288.log.gz 2026-02-20T11:25:47.711 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.46247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41368.log 2026-02-20T11:25:47.711 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46247.log.gz 2026-02-20T11:25:47.711 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.44027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45057.log 2026-02-20T11:25:47.711 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44027.log.gz 2026-02-20T11:25:47.711 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.41368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42783.log 2026-02-20T11:25:47.711 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41368.log.gz 2026-02-20T11:25:47.711 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.45057.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45057.log.gzgzip 2026-02-20T11:25:47.711 INFO:teuthology.orchestra.run.trial043.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41221.log 2026-02-20T11:25:47.711 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.42783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48786.log 2026-02-20T11:25:47.711 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42783.log.gz 2026-02-20T11:25:47.711 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.41221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40560.log 2026-02-20T11:25:47.712 INFO:teuthology.orchestra.run.trial043.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41221.log.gz 2026-02-20T11:25:47.712 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.48786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48515.log 2026-02-20T11:25:47.712 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48786.log.gz 2026-02-20T11:25:47.712 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.40560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42146.log 2026-02-20T11:25:47.712 INFO:teuthology.orchestra.run.trial043.stderr: 66.9% -- replaced with /var/log/ceph/ceph-client.admin.40560.log.gz 2026-02-20T11:25:47.712 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.48515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47489.log 2026-02-20T11:25:47.712 INFO:teuthology.orchestra.run.trial043.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.48515.log.gz 2026-02-20T11:25:47.712 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.42146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43541.log 2026-02-20T11:25:47.712 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42146.log.gz 2026-02-20T11:25:47.712 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.47489.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47489.log.gz 2026-02-20T11:25:47.712 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49596.log 2026-02-20T11:25:47.712 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.43541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40780.log 2026-02-20T11:25:47.712 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43541.log.gz 2026-02-20T11:25:47.712 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.49596.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49596.log.gz 2026-02-20T11:25:47.712 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49164.log 2026-02-20T11:25:47.712 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40169.log 2026-02-20T11:25:47.712 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.40780.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40780.log.gz 2026-02-20T11:25:47.712 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.49164.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49164.log.gz 2026-02-20T11:25:47.712 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50029.log 2026-02-20T11:25:47.712 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.40169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47651.log 2026-02-20T11:25:47.712 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40169.log.gz 2026-02-20T11:25:47.712 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.50029.log: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.50029.log.gz 2026-02-20T11:25:47.712 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46355.log 2026-02-20T11:25:47.712 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.47651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44082.log 2026-02-20T11:25:47.712 INFO:teuthology.orchestra.run.trial043.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47651.log.gz 2026-02-20T11:25:47.713 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.46355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48245.log 2026-02-20T11:25:47.713 INFO:teuthology.orchestra.run.trial043.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.46355.log.gz 2026-02-20T11:25:47.713 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.44082.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.44082.log.gz 2026-02-20T11:25:47.713 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45544.log 2026-02-20T11:25:47.713 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.48245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40910.log 2026-02-20T11:25:47.713 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48245.log.gz 2026-02-20T11:25:47.713 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.45544.log: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.45544.log.gz 2026-02-20T11:25:47.713 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40907.log 2026-02-20T11:25:47.713 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.40910.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40910.log.gz 2026-02-20T11:25:47.713 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44949.log 2026-02-20T11:25:47.713 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.40907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48948.log 2026-02-20T11:25:47.713 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40907.log.gz 2026-02-20T11:25:47.713 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.44949.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44949.log.gz 2026-02-20T11:25:47.713 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47165.log 2026-02-20T11:25:47.713 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.48948.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.48948.log.gz 2026-02-20T11:25:47.713 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.38963.log 2026-02-20T11:25:47.713 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.47165.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47165.log.gz 2026-02-20T11:25:47.713 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40463.log 2026-02-20T11:25:47.713 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph.tmp-client.admin.38963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43973.log 2026-02-20T11:25:47.713 INFO:teuthology.orchestra.run.trial043.stderr: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.38963.log.gz 2026-02-20T11:25:47.713 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.40463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43325.log 2026-02-20T11:25:47.713 INFO:teuthology.orchestra.run.trial043.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40463.log.gz 2026-02-20T11:25:47.713 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.43973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47273.log 2026-02-20T11:25:47.713 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43973.log.gz 2026-02-20T11:25:47.713 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.43325.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43325.log.gz 2026-02-20T11:25:47.713 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43919.log 2026-02-20T11:25:47.713 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.47273.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.47273.log.gz 2026-02-20T11:25:47.714 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45815.log 2026-02-20T11:25:47.714 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.43919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40682.log 2026-02-20T11:25:47.714 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43919.log.gz 2026-02-20T11:25:47.714 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.45815.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45815.log.gz 2026-02-20T11:25:47.714 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44245.log 2026-02-20T11:25:47.714 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.40682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41466.log 2026-02-20T11:25:47.714 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40682.log.gz 2026-02-20T11:25:47.714 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.44245.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-20T11:25:47.714 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44245.log.gz 2026-02-20T11:25:47.714 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.41466.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41466.log.gz 2026-02-20T11:25:47.714 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45923.log 2026-02-20T11:25:47.714 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45436.log 2026-02-20T11:25:47.714 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.45923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41270.log 2026-02-20T11:25:47.714 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45923.log.gz 2026-02-20T11:25:47.714 INFO:teuthology.orchestra.run.trial043.stderr: 87.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-20T11:25:47.714 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.45436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48623.log 2026-02-20T11:25:47.714 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45436.log.gz 2026-02-20T11:25:47.714 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.41270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47003.log 2026-02-20T11:25:47.714 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41270.log.gz 2026-02-20T11:25:47.714 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.48623.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48623.log.gz 2026-02-20T11:25:47.714 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49542.log 2026-02-20T11:25:47.714 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.47003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44786.log 2026-02-20T11:25:47.714 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47003.log.gz 2026-02-20T11:25:47.714 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.49542.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.49542.log.gz 2026-02-20T11:25:47.714 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41417.log 2026-02-20T11:25:47.714 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.44786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47543.log 2026-02-20T11:25:47.714 INFO:teuthology.orchestra.run.trial043.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.44786.log.gz 2026-02-20T11:25:47.714 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.41417.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41417.log.gz 2026-02-20T11:25:47.714 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44569.log 2026-02-20T11:25:47.715 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.47543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40731.log 2026-02-20T11:25:47.715 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47543.log.gz 2026-02-20T11:25:47.715 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.44569.log: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.44569.log.gz 2026-02-20T11:25:47.715 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41815.log 2026-02-20T11:25:47.715 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.40731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44677.log 2026-02-20T11:25:47.715 INFO:teuthology.orchestra.run.trial043.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40731.log.gz 2026-02-20T11:25:47.715 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.41815.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41815.log.gz 2026-02-20T11:25:47.715 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41319.log 2026-02-20T11:25:47.715 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.44677.log: 50.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.44677.log.gz -5 2026-02-20T11:25:47.715 INFO:teuthology.orchestra.run.trial043.stderr: --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-20T11:25:47.715 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.41319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47975.log 2026-02-20T11:25:47.715 INFO:teuthology.orchestra.run.trial043.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41319.log.gz 2026-02-20T11:25:47.715 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40909.log 2026-02-20T11:25:47.715 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.47975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48029.log 2026-02-20T11:25:47.715 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47975.log.gz 2026-02-20T11:25:47.715 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.40909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41515.log 2026-02-20T11:25:47.715 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40909.log.gz 2026-02-20T11:25:47.715 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.48029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46787.log 2026-02-20T11:25:47.715 INFO:teuthology.orchestra.run.trial043.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48029.log.gz 2026-02-20T11:25:47.715 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.41515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43703.log 2026-02-20T11:25:47.715 INFO:teuthology.orchestra.run.trial043.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41515.log.gz 2026-02-20T11:25:47.715 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.46787.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.46787.log.gz 2026-02-20T11:25:47.715 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48191.log 2026-02-20T11:25:47.715 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.43703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47921.log 2026-02-20T11:25:47.715 INFO:teuthology.orchestra.run.trial043.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43703.log.gz 2026-02-20T11:25:47.715 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.48191.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48191.log.gz 2026-02-20T11:25:47.715 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49989.log 2026-02-20T11:25:47.716 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.47921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47867.log 2026-02-20T11:25:47.716 INFO:teuthology.orchestra.run.trial043.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47921.log.gz 2026-02-20T11:25:47.716 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.49989.log: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.49989.log.gz 2026-02-20T11:25:47.716 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47381.log 2026-02-20T11:25:47.716 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.47867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46625.log 2026-02-20T11:25:47.716 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47867.log.gz 2026-02-20T11:25:47.716 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.47381.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47381.log.gz 2026-02-20T11:25:47.716 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42570.log 2026-02-20T11:25:47.716 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.46625.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46625.log.gz 2026-02-20T11:25:47.716 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49002.log 2026-02-20T11:25:47.716 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.42570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45760.log 2026-02-20T11:25:47.716 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42570.log.gz 2026-02-20T11:25:47.716 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.49002.log: gzip -5 50.4% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.49002.log.gz /var/log/ceph/ceph-client.admin.45869.log 2026-02-20T11:25:47.716 INFO:teuthology.orchestra.run.trial043.stderr: 2026-02-20T11:25:47.716 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.45760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42465.log 2026-02-20T11:25:47.716 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45760.log.gz 2026-02-20T11:25:47.716 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.45869.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45869.log.gz 2026-02-20T11:25:47.716 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43649.log 2026-02-20T11:25:47.716 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.42465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46463.log 2026-02-20T11:25:47.716 INFO:teuthology.orchestra.run.trial043.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42465.log.gz 2026-02-20T11:25:47.716 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.43649.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43649.log.gz 2026-02-20T11:25:47.716 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47813.log 2026-02-20T11:25:47.716 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.46463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48407.log 2026-02-20T11:25:47.717 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46463.log.gz 2026-02-20T11:25:47.717 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.47813.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.47813.log.gz 2026-02-20T11:25:47.717 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43595.log 2026-02-20T11:25:47.717 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.48407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46301.log 2026-02-20T11:25:47.717 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48407.log.gz 2026-02-20T11:25:47.717 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.43595.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43595.log.gz 2026-02-20T11:25:47.717 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45112.log 2026-02-20T11:25:47.717 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.46301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49272.log 2026-02-20T11:25:47.717 INFO:teuthology.orchestra.run.trial043.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.46301.log.gz 2026-02-20T11:25:47.717 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.45112.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45112.log.gz 2026-02-20T11:25:47.717 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41946.log 2026-02-20T11:25:47.717 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.49272.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.49272.log.gz 2026-02-20T11:25:47.717 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44407.log 2026-02-20T11:25:47.717 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.41946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43000.log 2026-02-20T11:25:47.717 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41946.log.gz 2026-02-20T11:25:47.717 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.44407.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44407.log.gz 2026-02-20T11:25:47.717 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41945.log 2026-02-20T11:25:47.717 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.43000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40122.log 2026-02-20T11:25:47.717 INFO:teuthology.orchestra.run.trial043.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43000.log.gz 2026-02-20T11:25:47.717 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.41945.log: -- /var/log/ceph/ceph-client.admin.49920.log 2026-02-20T11:25:47.717 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41945.log.gz 2026-02-20T11:25:47.717 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.40122.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40122.log.gz 2026-02-20T11:25:47.717 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40533.log 2026-02-20T11:25:47.717 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.49920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41564.log 2026-02-20T11:25:47.717 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49920.log.gz 2026-02-20T11:25:47.717 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.40533.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40533.log.gz 2026-02-20T11:25:47.717 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42416.log 2026-02-20T11:25:47.717 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.41564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44353.log 2026-02-20T11:25:47.718 INFO:teuthology.orchestra.run.trial043.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41564.log.gz 2026-02-20T11:25:47.718 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.42416.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42416.log.gz 2026-02-20T11:25:47.718 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48569.log 2026-02-20T11:25:47.718 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.44353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47435.log 2026-02-20T11:25:47.718 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44353.log.gz 2026-02-20T11:25:47.718 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.48569.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48569.log.gz 2026-02-20T11:25:47.718 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49704.log 2026-02-20T11:25:47.718 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.47435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41816.log 2026-02-20T11:25:47.718 INFO:teuthology.orchestra.run.trial043.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.47435.log.gz 2026-02-20T11:25:47.718 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.49704.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49704.log.gz 2026-02-20T11:25:47.718 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42212.log 2026-02-20T11:25:47.718 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.41816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42675.log 2026-02-20T11:25:47.718 INFO:teuthology.orchestra.run.trial043.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41816.log.gz 2026-02-20T11:25:47.718 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.42212.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42212.log.gz 2026-02-20T11:25:47.718 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46031.log 2026-02-20T11:25:47.718 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.42675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40036.log 2026-02-20T11:25:47.718 INFO:teuthology.orchestra.run.trial043.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42675.log.gz 2026-02-20T11:25:47.718 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.46031.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46031.log.gz 2026-02-20T11:25:47.718 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41104.log 2026-02-20T11:25:47.718 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.40036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41103.log 2026-02-20T11:25:47.718 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40036.log.gz 2026-02-20T11:25:47.718 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.41104.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41104.log.gz 2026-02-20T11:25:47.718 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49056.log 2026-02-20T11:25:47.718 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.41103.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41103.log.gz 2026-02-20T11:25:47.718 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40633.log 2026-02-20T11:25:47.718 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.49056.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.49056.log.gz 2026-02-20T11:25:47.718 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47111.log 2026-02-20T11:25:47.718 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.40633.log: gzip -5 --verbose 50.0% -- /var/log/ceph/ceph-client.admin.47759.log -- replaced with /var/log/ceph/ceph-client.admin.40633.log.gz 2026-02-20T11:25:47.718 INFO:teuthology.orchestra.run.trial043.stderr: 2026-02-20T11:25:47.718 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.47111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45382.log 2026-02-20T11:25:47.719 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47111.log.gz 2026-02-20T11:25:47.719 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.47759.log: 49.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.47759.log.gz -5 2026-02-20T11:25:47.719 INFO:teuthology.orchestra.run.trial043.stderr: --verbose -- /var/log/ceph/ceph.log 2026-02-20T11:25:47.719 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.45382.log: gzip -5 48.4% --verbose -- replaced with /var/log/ceph/ceph-client.admin.45382.log.gz -- 2026-02-20T11:25:47.719 INFO:teuthology.orchestra.run.trial043.stderr: /var/log/ceph/ceph-client.admin.42521.log 2026-02-20T11:25:47.719 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45490.log 2026-02-20T11:25:47.719 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.42521.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42521.log.gz 2026-02-20T11:25:47.719 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44623.log 2026-02-20T11:25:47.719 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.45490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47597.log 2026-02-20T11:25:47.719 INFO:teuthology.orchestra.run.trial043.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45490.log.gz 2026-02-20T11:25:47.719 INFO:teuthology.orchestra.run.trial043.stderr: 87.6% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-20T11:25:47.719 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.44623.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44623.log.gz 2026-02-20T11:25:47.719 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43757.log 2026-02-20T11:25:47.719 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.47597.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.47597.log.gz 2026-02-20T11:25:47.719 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42946.log 2026-02-20T11:25:47.719 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.43757.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43757.log.gz 2026-02-20T11:25:47.719 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44732.log 2026-02-20T11:25:47.719 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.42946.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42946.log.gz 2026-02-20T11:25:47.719 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49326.log 2026-02-20T11:25:47.719 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.44732.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44732.log.gz 2026-02-20T11:25:47.719 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45598.log 2026-02-20T11:25:47.719 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.49326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42837.log 2026-02-20T11:25:47.719 INFO:teuthology.orchestra.run.trial043.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.49326.log.gz 2026-02-20T11:25:47.719 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.45598.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45598.log.gz 2026-02-20T11:25:47.719 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41105.log 2026-02-20T11:25:47.719 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.42837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43487.log 2026-02-20T11:25:47.719 INFO:teuthology.orchestra.run.trial043.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42837.log.gz 2026-02-20T11:25:47.719 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.41105.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41105.log.gz 2026-02-20T11:25:47.719 INFO:teuthology.orchestra.run.trial043.stderr:/var/log/ceph/ceph-client.admin.43487.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43487.log.gz 2026-02-20T11:25:47.720 INFO:teuthology.orchestra.run.trial109.stderr:/var/log/ceph/ceph-mgr.x.log: 91.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-20T11:25:47.811 INFO:teuthology.orchestra.run.trial109.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-20T11:25:47.811 INFO:teuthology.orchestra.run.trial043.stderr: 89.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-20T11:25:48.612 INFO:teuthology.orchestra.run.trial043.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-20T11:25:48.822 INFO:teuthology.orchestra.run.trial043.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-20T11:25:48.824 INFO:teuthology.orchestra.run.trial043.stderr: 2026-02-20T11:25:48.824 INFO:teuthology.orchestra.run.trial043.stderr:real 0m1.176s 2026-02-20T11:25:48.824 INFO:teuthology.orchestra.run.trial043.stderr:user 0m2.188s 2026-02-20T11:25:48.824 INFO:teuthology.orchestra.run.trial043.stderr:sys 0m0.186s 2026-02-20T11:25:49.366 INFO:teuthology.orchestra.run.trial109.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2026-02-20T11:25:49.368 INFO:teuthology.orchestra.run.trial109.stderr: 2026-02-20T11:25:49.368 INFO:teuthology.orchestra.run.trial109.stderr:real 0m1.694s 2026-02-20T11:25:49.368 INFO:teuthology.orchestra.run.trial109.stderr:user 0m1.748s 2026-02-20T11:25:49.368 INFO:teuthology.orchestra.run.trial109.stderr:sys 0m0.112s 2026-02-20T11:25:49.369 INFO:tasks.ceph:Archiving logs... 2026-02-20T11:25:49.369 DEBUG:teuthology.misc:Transferring archived files from trial043:/var/log/ceph to /home/teuthworker/mnt/teuthology/jjperez-2026-02-20_10:21:31-crimson-rados-wip-perezjos-crimson-only-19-02-2026-PR64439-distro-debug-trial/60477/remote/trial043/log 2026-02-20T11:25:49.370 DEBUG:teuthology.orchestra.run.trial043:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-20T11:25:49.722 DEBUG:teuthology.misc:Transferring archived files from trial109:/var/log/ceph to /home/teuthworker/mnt/teuthology/jjperez-2026-02-20_10:21:31-crimson-rados-wip-perezjos-crimson-only-19-02-2026-PR64439-distro-debug-trial/60477/remote/trial109/log 2026-02-20T11:25:49.722 DEBUG:teuthology.orchestra.run.trial109:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-20T11:25:49.915 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph.py", line 2000, in task yield File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/thrashosds.py", line 214, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.2 has failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 158, in __exit__ self.gen.throw(value) File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph.py", line 2010, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 2936, in wait_for_clean num_active_clean = self.get_num_active_clean() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 2715, in get_num_active_clean pgs = self.get_pg_stats() ^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 2481, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 1713, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 1704, in run_cluster_cmd return self.controller.run(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/remote.py", line 596, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/run.py", line 461, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial043 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2026-02-20T11:25:49.916 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-20T11:25:49.923 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/contextutil.py", line 34, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/task/install/__init__.py", line 644, in task yield File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph.py", line 2000, in task yield File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/thrashosds.py", line 214, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.2 has failed 2026-02-20T11:25:50.040 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-20T11:25:50.050 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/contextutil.py", line 34, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/task/install/__init__.py", line 644, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.2 has failed 2026-02-20T11:25:50.051 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-20T11:25:50.051 DEBUG:teuthology.orchestra.run.trial043:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-20T11:25:50.053 DEBUG:teuthology.orchestra.run.trial109:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-20T11:25:50.078 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-20T11:25:50.089 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-20T11:25:50.090 DEBUG:teuthology.orchestra.run.trial043:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-20T11:25:50.118 DEBUG:teuthology.orchestra.run.trial109:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-20T11:25:50.131 INFO:teuthology.orchestra.run.trial043.stderr:bash: line 1: ntpq: command not found 2026-02-20T11:25:50.135 INFO:teuthology.orchestra.run.trial109.stderr:bash: line 1: ntpq: command not found 2026-02-20T11:25:50.467 INFO:teuthology.orchestra.run.trial109.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-20T11:25:50.467 INFO:teuthology.orchestra.run.trial109.stdout:=============================================================================== 2026-02-20T11:25:50.467 INFO:teuthology.orchestra.run.trial109.stdout:^- 149.28.200.179.vultruser> 2 6 377 5 -618us[ -591us] +/- 40ms 2026-02-20T11:25:50.467 INFO:teuthology.orchestra.run.trial109.stdout:^+ chi2.us.ntp.li 2 6 377 5 +106us[ +106us] +/- 11ms 2026-02-20T11:25:50.467 INFO:teuthology.orchestra.run.trial109.stdout:^- 155.248.196.28 2 6 377 6 -90us[ -64us] +/- 49ms 2026-02-20T11:25:50.467 INFO:teuthology.orchestra.run.trial109.stdout:^* ntp.netlinkify.com 2 6 377 5 -46us[ -20us] +/- 7934us 2026-02-20T11:25:50.684 INFO:teuthology.orchestra.run.trial043.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-20T11:25:50.684 INFO:teuthology.orchestra.run.trial043.stdout:=============================================================================== 2026-02-20T11:25:50.684 INFO:teuthology.orchestra.run.trial043.stdout:^* ntp2.zeus.frumentum.media 2 6 377 3 -692us[ -693us] +/- 24ms 2026-02-20T11:25:50.685 INFO:teuthology.orchestra.run.trial043.stdout:^- time3.lshiy.com 2 6 377 5 +9589us[+9588us] +/- 142ms 2026-02-20T11:25:50.685 INFO:teuthology.orchestra.run.trial043.stdout:^+ lithium.constant.com 2 6 377 7 +1086us[+1085us] +/- 47ms 2026-02-20T11:25:50.685 INFO:teuthology.orchestra.run.trial043.stdout:^- ip216-250-115-174.pbiaas> 2 6 377 5 +736us[ +735us] +/- 108ms 2026-02-20T11:25:50.686 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-20T11:25:50.702 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-20T11:25:50.703 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-20T11:25:50.719 DEBUG:teuthology.orchestra.run.trial043:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-20T11:25:50.783 DEBUG:teuthology.orchestra.run.trial109:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-20T11:25:50.840 DEBUG:teuthology.orchestra.run.trial043:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-20T11:25:50.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-20T11:25:50.869 DEBUG:teuthology.orchestra.run.trial109:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-20T11:25:50.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-20T11:25:50.897 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-20T11:25:50.914 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-20T11:25:50.930 INFO:teuthology.task.internal:Duration was 1082.753043 seconds 2026-02-20T11:25:50.931 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-20T11:25:50.947 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-20T11:25:50.947 DEBUG:teuthology.orchestra.run.trial043:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-20T11:25:50.950 DEBUG:teuthology.orchestra.run.trial109:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-20T11:25:50.987 INFO:teuthology.orchestra.run.trial043.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-20T11:25:50.987 INFO:teuthology.orchestra.run.trial109.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-20T11:25:51.176 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-20T11:25:51.176 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial043.front.sepia.ceph.com 2026-02-20T11:25:51.176 DEBUG:teuthology.orchestra.run.trial043:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-20T11:25:51.235 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial109.front.sepia.ceph.com 2026-02-20T11:25:51.236 DEBUG:teuthology.orchestra.run.trial109:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-20T11:25:51.255 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-20T11:25:51.255 DEBUG:teuthology.orchestra.run.trial043:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-20T11:25:51.277 DEBUG:teuthology.orchestra.run.trial109:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-20T11:25:51.360 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-20T11:25:51.360 DEBUG:teuthology.orchestra.run.trial043:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-20T11:25:51.364 DEBUG:teuthology.orchestra.run.trial109:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-20T11:25:51.385 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-20T11:25:51.385 INFO:teuthology.orchestra.run.trial043.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-20T11:25:51.385 INFO:teuthology.orchestra.run.trial043.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-20T11:25:51.385 INFO:teuthology.orchestra.run.trial043.stderr: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-20T11:25:51.386 INFO:teuthology.orchestra.run.trial043.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-20T11:25:51.389 INFO:teuthology.orchestra.run.trial109.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-20T11:25:51.390 INFO:teuthology.orchestra.run.trial109.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-20T11:25:51.390 INFO:teuthology.orchestra.run.trial109.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-20T11:25:51.390 INFO:teuthology.orchestra.run.trial109.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-20T11:25:51.390 INFO:teuthology.orchestra.run.trial109.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-20T11:25:51.395 INFO:teuthology.orchestra.run.trial043.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 83.2% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-20T11:25:51.401 INFO:teuthology.orchestra.run.trial109.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 82.6% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-20T11:25:51.403 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-20T11:25:51.420 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-20T11:25:51.420 DEBUG:teuthology.orchestra.run.trial043:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-20T11:25:51.464 DEBUG:teuthology.orchestra.run.trial109:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-20T11:25:51.490 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-20T11:25:51.505 DEBUG:teuthology.orchestra.run.trial043:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-20T11:25:51.507 DEBUG:teuthology.orchestra.run.trial109:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-20T11:25:51.527 INFO:teuthology.orchestra.run.trial043.stdout:kernel.core_pattern = core 2026-02-20T11:25:51.554 INFO:teuthology.orchestra.run.trial109.stdout:kernel.core_pattern = core 2026-02-20T11:25:51.571 DEBUG:teuthology.orchestra.run.trial043:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-20T11:25:51.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-20T11:25:51.596 DEBUG:teuthology.orchestra.run.trial109:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-20T11:25:51.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-20T11:25:51.628 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-20T11:25:51.647 INFO:teuthology.task.internal:Transferring archived files... 2026-02-20T11:25:51.648 DEBUG:teuthology.misc:Transferring archived files from trial043:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/jjperez-2026-02-20_10:21:31-crimson-rados-wip-perezjos-crimson-only-19-02-2026-PR64439-distro-debug-trial/60477/remote/trial043 2026-02-20T11:25:51.648 DEBUG:teuthology.orchestra.run.trial043:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-20T11:25:51.687 DEBUG:teuthology.misc:Transferring archived files from trial109:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/jjperez-2026-02-20_10:21:31-crimson-rados-wip-perezjos-crimson-only-19-02-2026-PR64439-distro-debug-trial/60477/remote/trial109 2026-02-20T11:25:51.687 DEBUG:teuthology.orchestra.run.trial109:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-20T11:25:51.725 INFO:teuthology.task.internal:Removing archive directory... 2026-02-20T11:25:51.725 DEBUG:teuthology.orchestra.run.trial043:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-20T11:25:51.728 DEBUG:teuthology.orchestra.run.trial109:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-20T11:25:51.772 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-20T11:25:51.787 INFO:teuthology.task.internal:Not uploading archives. 2026-02-20T11:25:51.787 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-20T11:25:51.802 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-20T11:25:51.802 DEBUG:teuthology.orchestra.run.trial043:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-20T11:25:51.805 DEBUG:teuthology.orchestra.run.trial109:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-20T11:25:51.820 INFO:teuthology.orchestra.run.trial043.stdout: 917576 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 20 11:25 /home/ubuntu/cephtest 2026-02-20T11:25:51.828 INFO:teuthology.orchestra.run.trial109.stdout: 917576 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 20 11:25 /home/ubuntu/cephtest 2026-02-20T11:25:51.829 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-20T11:25:51.843 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-20T11:25:51.857 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: BarkError: The WatchDog barked due to Daemon ceph.osd.2 has failed 2026-02-20T11:25:51.862 INFO:teuthology.run:Summary data: description: crimson-rados/thrash/{0-crimson_install 0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} clusters/{crimson-fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm} thrashers/{default} thrashosds-health workloads/pool-snaps-few-objects} duration: 1082.753042936325 failure_reason: The WatchDog barked due to Daemon ceph.osd.2 has failed flavor: debug owner: scheduled_jjperez@soko04.front.sepia.ceph.com status: fail success: false 2026-02-20T11:25:51.862 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-20T11:25:52.039 INFO:teuthology.run:FAIL