2026-02-25T08:02:14.877 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-25T08:02:14.896 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-25T08:02:14.972 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69065 branch: wip-logstoremanager description: crimson-rados/thrash/{0-crimson_install 0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} clusters/{crimson-fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm} thrashers/{default} thrashosds-health workloads/snaps-few-objects} email: null first_in_suite: false flavor: debug job_id: '69065' kernel: branch: distro kdb: 1 sha1: distro last_in_suite: false machine_type: trial name: matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-logstoremanager ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc osd_max_pg_log_entries: 2 osd_min_pg_log_entries: 1 osd_pg_log_trim_min: 0 osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 osd pool default crimson: true paxos service trim min: 10 osd: crimson cpu num: 4 crimson osd obc lru size: 10 debug ms: 20 debug osd: 20 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore omap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd max markdown count: 1000 osd objectstore: seastore osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 osd target transaction size: 5 seastore cachepin size pershard: 64M seastore main device type: RANDOM_BLOCK_SSD seastore max concurrent transactions: 8 crimson_compat: true flavor: debug log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - SLOW_OPS - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub - Degraded data redundancy - is down - osds down - pg .*? is .*?degraded.*?, acting - pg .*? is stuck - pg degraded - PG_BACKFILL_FULL - Low space hindering backfill .*? backfill_toofull - OSD_ROOT_DOWN - pgs degraded - pgs undersized - is active.*backfill_toofull.* sha1: 6aeb6975bc9a4790a399c9cc70dba9635d2a7819 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: debug sha1: 6aeb6975bc9a4790a399c9cc70dba9635d2a7819 thrashosds: min_in: 4 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: c32469671c6b0222f786dd36c34eb702f780a11d owner: scheduled_matan@soko04.front.sepia.ceph.com priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - osd.0 - osd.1 - client.0 - node-exporter.a - - mgr.x - osd.2 - osd.3 - client.1 - prometheus.a - node-exporter.b seed: 869 sha1: 6aeb6975bc9a4790a399c9cc70dba9635d2a7819 sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: c32469671c6b0222f786dd36c34eb702f780a11d targets: trial023.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPnROBW7BbzJe4h6dtZU3g748aCEt0GJxM7VF4d+cj+2EPpPYDUSYz9DHlYNwhr/BpHN+2GfrLzLQfif2W2ABr4= trial030.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGREmmmHSQHT+Z/Iwdj68qbUv3TPtliFzc5BI+FC4YfWOexKFBz7/N0pQrz5NA70I/5YxjJjgdGObk99vDbrzlA= tasks: - install: extra_packages: deb: - ceph-osd-crimson - ceph-osd-crimson-dbg rpm: - ceph-osd-crimson - ceph-osd-crimson-debuginfo - pexec: all: - sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson - install: null - ceph: conf: osd: debug monc: 20 - thrashosds: ceph_objectstore_tool: false chance_inject_pause_short: 0 chance_reset_purged_snaps_last: 0 chance_thrash_cluster_full: 0 chance_thrash_pg_upmap: 0 chance_trim_stale_osdmaps: 0 dump_ops_enable: false min_in: 3 noscrub_toggle_delay: 0 reweight_osd: 0 sighup_delay: 0 thrash_primary_affinity: false timeout: 2400 - rados: clients: - client.0 objects: 50 op_weights: copy_from: 0 delete: 50 read: 100 rollback: 0 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-25_07:55:35 tube: trial user: matan verbose: false worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699 2026-02-25T08:02:14.972 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa; will attempt to use it 2026-02-25T08:02:14.973 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks 2026-02-25T08:02:14.973 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-25T08:02:14.974 INFO:teuthology.task.internal:Checking packages... 2026-02-25T08:02:14.974 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'debug' and ceph hash '6aeb6975bc9a4790a399c9cc70dba9635d2a7819' 2026-02-25T08:02:14.974 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-25T08:02:14.974 INFO:teuthology.packaging:ref: None 2026-02-25T08:02:14.974 INFO:teuthology.packaging:tag: None 2026-02-25T08:02:14.974 INFO:teuthology.packaging:branch: wip-logstoremanager 2026-02-25T08:02:14.974 INFO:teuthology.packaging:sha1: 6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:02:14.974 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&ref=wip-logstoremanager 2026-02-25T08:02:15.091 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5338.g6aeb6975 2026-02-25T08:02:15.091 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-25T08:02:15.104 INFO:teuthology.task.internal:no buildpackages task found 2026-02-25T08:02:15.104 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-25T08:02:15.118 INFO:teuthology.task.internal:Saving configuration 2026-02-25T08:02:15.131 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-25T08:02:15.143 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-25T08:02:15.218 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial023.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69065', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-25 07:57:42.796823', 'locked_by': 'scheduled_matan@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPnROBW7BbzJe4h6dtZU3g748aCEt0GJxM7VF4d+cj+2EPpPYDUSYz9DHlYNwhr/BpHN+2GfrLzLQfif2W2ABr4='} 2026-02-25T08:02:15.274 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial030.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69065', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-25 07:57:42.797454', 'locked_by': 'scheduled_matan@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGREmmmHSQHT+Z/Iwdj68qbUv3TPtliFzc5BI+FC4YfWOexKFBz7/N0pQrz5NA70I/5YxjJjgdGObk99vDbrzlA='} 2026-02-25T08:02:15.274 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-25T08:02:15.286 INFO:teuthology.task.internal:roles: ubuntu@trial023.front.sepia.ceph.com - ['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'] 2026-02-25T08:02:15.286 INFO:teuthology.task.internal:roles: ubuntu@trial030.front.sepia.ceph.com - ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b'] 2026-02-25T08:02:15.286 INFO:teuthology.run_tasks:Running task console_log... 2026-02-25T08:02:15.422 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7497e50c65c0>, signals=[15]) 2026-02-25T08:02:15.422 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-25T08:02:15.433 INFO:teuthology.task.internal:Opening connections... 2026-02-25T08:02:15.433 DEBUG:teuthology.task.internal:connecting to ubuntu@trial023.front.sepia.ceph.com 2026-02-25T08:02:15.434 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T08:02:15.494 DEBUG:teuthology.task.internal:connecting to ubuntu@trial030.front.sepia.ceph.com 2026-02-25T08:02:15.494 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial030.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T08:02:15.554 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-25T08:02:15.566 DEBUG:teuthology.orchestra.run.trial023:> uname -m 2026-02-25T08:02:15.581 INFO:teuthology.orchestra.run.trial023.stdout:x86_64 2026-02-25T08:02:15.581 DEBUG:teuthology.orchestra.run.trial023:> cat /etc/os-release 2026-02-25T08:02:15.636 INFO:teuthology.orchestra.run.trial023.stdout:NAME="CentOS Stream" 2026-02-25T08:02:15.636 INFO:teuthology.orchestra.run.trial023.stdout:VERSION="9" 2026-02-25T08:02:15.637 INFO:teuthology.orchestra.run.trial023.stdout:ID="centos" 2026-02-25T08:02:15.637 INFO:teuthology.orchestra.run.trial023.stdout:ID_LIKE="rhel fedora" 2026-02-25T08:02:15.637 INFO:teuthology.orchestra.run.trial023.stdout:VERSION_ID="9" 2026-02-25T08:02:15.637 INFO:teuthology.orchestra.run.trial023.stdout:PLATFORM_ID="platform:el9" 2026-02-25T08:02:15.637 INFO:teuthology.orchestra.run.trial023.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-25T08:02:15.637 INFO:teuthology.orchestra.run.trial023.stdout:ANSI_COLOR="0;31" 2026-02-25T08:02:15.637 INFO:teuthology.orchestra.run.trial023.stdout:LOGO="fedora-logo-icon" 2026-02-25T08:02:15.637 INFO:teuthology.orchestra.run.trial023.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-25T08:02:15.637 INFO:teuthology.orchestra.run.trial023.stdout:HOME_URL="https://centos.org/" 2026-02-25T08:02:15.637 INFO:teuthology.orchestra.run.trial023.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-25T08:02:15.637 INFO:teuthology.orchestra.run.trial023.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-25T08:02:15.637 INFO:teuthology.orchestra.run.trial023.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-25T08:02:15.637 INFO:teuthology.lock.ops:Updating trial023.front.sepia.ceph.com on lock server 2026-02-25T08:02:15.718 DEBUG:teuthology.orchestra.run.trial030:> uname -m 2026-02-25T08:02:15.734 INFO:teuthology.orchestra.run.trial030.stdout:x86_64 2026-02-25T08:02:15.734 DEBUG:teuthology.orchestra.run.trial030:> cat /etc/os-release 2026-02-25T08:02:15.789 INFO:teuthology.orchestra.run.trial030.stdout:NAME="CentOS Stream" 2026-02-25T08:02:15.789 INFO:teuthology.orchestra.run.trial030.stdout:VERSION="9" 2026-02-25T08:02:15.789 INFO:teuthology.orchestra.run.trial030.stdout:ID="centos" 2026-02-25T08:02:15.789 INFO:teuthology.orchestra.run.trial030.stdout:ID_LIKE="rhel fedora" 2026-02-25T08:02:15.790 INFO:teuthology.orchestra.run.trial030.stdout:VERSION_ID="9" 2026-02-25T08:02:15.790 INFO:teuthology.orchestra.run.trial030.stdout:PLATFORM_ID="platform:el9" 2026-02-25T08:02:15.790 INFO:teuthology.orchestra.run.trial030.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-25T08:02:15.790 INFO:teuthology.orchestra.run.trial030.stdout:ANSI_COLOR="0;31" 2026-02-25T08:02:15.790 INFO:teuthology.orchestra.run.trial030.stdout:LOGO="fedora-logo-icon" 2026-02-25T08:02:15.790 INFO:teuthology.orchestra.run.trial030.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-25T08:02:15.790 INFO:teuthology.orchestra.run.trial030.stdout:HOME_URL="https://centos.org/" 2026-02-25T08:02:15.790 INFO:teuthology.orchestra.run.trial030.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-25T08:02:15.790 INFO:teuthology.orchestra.run.trial030.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-25T08:02:15.790 INFO:teuthology.orchestra.run.trial030.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-25T08:02:15.790 INFO:teuthology.lock.ops:Updating trial030.front.sepia.ceph.com on lock server 2026-02-25T08:02:15.868 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-25T08:02:15.881 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-25T08:02:15.894 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-25T08:02:15.895 DEBUG:teuthology.orchestra.run.trial023:> test '!' -e /home/ubuntu/cephtest 2026-02-25T08:02:15.896 DEBUG:teuthology.orchestra.run.trial030:> test '!' -e /home/ubuntu/cephtest 2026-02-25T08:02:15.911 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-25T08:02:15.923 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-25T08:02:15.923 DEBUG:teuthology.orchestra.run.trial023:> test -z $(ls -A /var/lib/ceph) 2026-02-25T08:02:15.952 DEBUG:teuthology.orchestra.run.trial030:> test -z $(ls -A /var/lib/ceph) 2026-02-25T08:02:15.969 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-25T08:02:16.122 INFO:teuthology.run_tasks:Running task kernel... 2026-02-25T08:02:16.141 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'} 2026-02-25T08:02:16.142 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}, 'mgr.x': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, timeout 300 2026-02-25T08:02:16.142 DEBUG:teuthology.orchestra.run.trial023:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-25T08:02:16.142 DEBUG:teuthology.orchestra.run.trial030:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-25T08:02:16.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:02:16.157 DEBUG:teuthology.orchestra.run.trial030:> uname -r 2026-02-25T08:02:16.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:02:16.157 DEBUG:teuthology.orchestra.run.trial023:> uname -r 2026-02-25T08:02:16.212 INFO:teuthology.orchestra.run.trial030.stdout:5.14.0-681.el9.x86_64 2026-02-25T08:02:16.212 INFO:teuthology.task.kernel:Running kernel on trial030: 5.14.0-681.el9.x86_64 2026-02-25T08:02:16.212 DEBUG:teuthology.orchestra.run.trial030:> sudo yum install -y kernel 2026-02-25T08:02:16.213 INFO:teuthology.orchestra.run.trial023.stdout:5.14.0-681.el9.x86_64 2026-02-25T08:02:16.213 INFO:teuthology.task.kernel:Running kernel on trial023: 5.14.0-681.el9.x86_64 2026-02-25T08:02:16.213 DEBUG:teuthology.orchestra.run.trial023:> sudo yum install -y kernel 2026-02-25T08:02:17.543 INFO:teuthology.orchestra.run.trial023.stdout:CentOS Stream 9 - BaseOS 9.4 MB/s | 8.9 MB 00:00 2026-02-25T08:02:17.948 INFO:teuthology.orchestra.run.trial030.stdout:CentOS Stream 9 - BaseOS 6.6 MB/s | 8.9 MB 00:01 2026-02-25T08:02:22.570 INFO:teuthology.orchestra.run.trial023.stdout:CentOS Stream 9 - AppStream 6.0 MB/s | 27 MB 00:04 2026-02-25T08:02:23.309 INFO:teuthology.orchestra.run.trial030.stdout:CentOS Stream 9 - AppStream 5.6 MB/s | 27 MB 00:04 2026-02-25T08:02:25.303 INFO:teuthology.orchestra.run.trial023.stdout:CentOS Stream 9 - CRB 33 MB/s | 8.0 MB 00:00 2026-02-25T08:02:26.251 INFO:teuthology.orchestra.run.trial023.stdout:CentOS Stream 9 - Extras packages 116 kB/s | 20 kB 00:00 2026-02-25T08:02:26.616 INFO:teuthology.orchestra.run.trial023.stdout:Extra Packages for Enterprise Linux 68 MB/s | 20 MB 00:00 2026-02-25T08:02:27.290 INFO:teuthology.orchestra.run.trial030.stdout:CentOS Stream 9 - CRB 5.0 MB/s | 8.0 MB 00:01 2026-02-25T08:02:28.145 INFO:teuthology.orchestra.run.trial030.stdout:CentOS Stream 9 - Extras packages 175 kB/s | 20 kB 00:00 2026-02-25T08:02:28.840 INFO:teuthology.orchestra.run.trial030.stdout:Extra Packages for Enterprise Linux 32 MB/s | 20 MB 00:00 2026-02-25T08:02:29.906 INFO:teuthology.orchestra.run.trial023.stdout:lab-extras 2.7 MB/s | 50 kB 00:00 2026-02-25T08:02:30.910 INFO:teuthology.orchestra.run.trial023.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-25T08:02:30.911 INFO:teuthology.orchestra.run.trial023.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-25T08:02:30.930 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:02:30.932 INFO:teuthology.orchestra.run.trial023.stdout:================================================================================ 2026-02-25T08:02:30.932 INFO:teuthology.orchestra.run.trial023.stdout: Package Architecture Version Repository Size 2026-02-25T08:02:30.932 INFO:teuthology.orchestra.run.trial023.stdout:================================================================================ 2026-02-25T08:02:30.932 INFO:teuthology.orchestra.run.trial023.stdout:Installing: 2026-02-25T08:02:30.932 INFO:teuthology.orchestra.run.trial023.stdout: kernel x86_64 5.14.0-686.el9 baseos 901 k 2026-02-25T08:02:30.932 INFO:teuthology.orchestra.run.trial023.stdout:Installing dependencies: 2026-02-25T08:02:30.932 INFO:teuthology.orchestra.run.trial023.stdout: kernel-core x86_64 5.14.0-686.el9 baseos 17 M 2026-02-25T08:02:30.932 INFO:teuthology.orchestra.run.trial023.stdout: kernel-modules x86_64 5.14.0-686.el9 baseos 40 M 2026-02-25T08:02:30.932 INFO:teuthology.orchestra.run.trial023.stdout: kernel-modules-core x86_64 5.14.0-686.el9 baseos 31 M 2026-02-25T08:02:30.932 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:02:30.933 INFO:teuthology.orchestra.run.trial023.stdout:Transaction Summary 2026-02-25T08:02:30.933 INFO:teuthology.orchestra.run.trial023.stdout:================================================================================ 2026-02-25T08:02:30.933 INFO:teuthology.orchestra.run.trial023.stdout:Install 4 Packages 2026-02-25T08:02:30.933 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:02:30.933 INFO:teuthology.orchestra.run.trial023.stdout:Total download size: 89 M 2026-02-25T08:02:30.933 INFO:teuthology.orchestra.run.trial023.stdout:Installed size: 135 M 2026-02-25T08:02:30.933 INFO:teuthology.orchestra.run.trial023.stdout:Downloading Packages: 2026-02-25T08:02:31.202 INFO:teuthology.orchestra.run.trial023.stdout:(1/4): kernel-5.14.0-686.el9.x86_64.rpm 4.9 MB/s | 901 kB 00:00 2026-02-25T08:02:31.346 INFO:teuthology.orchestra.run.trial023.stdout:(2/4): kernel-core-5.14.0-686.el9.x86_64.rpm 53 MB/s | 17 MB 00:00 2026-02-25T08:02:31.490 INFO:teuthology.orchestra.run.trial023.stdout:(3/4): kernel-modules-core-5.14.0-686.el9.x86_6 107 MB/s | 31 MB 00:00 2026-02-25T08:02:31.565 INFO:teuthology.orchestra.run.trial023.stdout:(4/4): kernel-modules-5.14.0-686.el9.x86_64.rpm 74 MB/s | 40 MB 00:00 2026-02-25T08:02:31.565 INFO:teuthology.orchestra.run.trial023.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:02:31.565 INFO:teuthology.orchestra.run.trial023.stdout:Total 141 MB/s | 89 MB 00:00 2026-02-25T08:02:31.756 INFO:teuthology.orchestra.run.trial023.stdout:Running transaction check 2026-02-25T08:02:31.970 INFO:teuthology.orchestra.run.trial023.stdout:Transaction check succeeded. 2026-02-25T08:02:31.970 INFO:teuthology.orchestra.run.trial023.stdout:Running transaction test 2026-02-25T08:02:32.001 INFO:teuthology.orchestra.run.trial030.stdout:lab-extras 2.7 MB/s | 50 kB 00:00 2026-02-25T08:02:32.353 INFO:teuthology.orchestra.run.trial023.stdout:Transaction test succeeded. 2026-02-25T08:02:32.353 INFO:teuthology.orchestra.run.trial023.stdout:Running transaction 2026-02-25T08:02:33.010 INFO:teuthology.orchestra.run.trial030.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-25T08:02:33.010 INFO:teuthology.orchestra.run.trial030.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-25T08:02:33.030 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:02:33.031 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-25T08:02:33.031 INFO:teuthology.orchestra.run.trial030.stdout: Package Architecture Version Repository Size 2026-02-25T08:02:33.031 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-25T08:02:33.031 INFO:teuthology.orchestra.run.trial030.stdout:Installing: 2026-02-25T08:02:33.032 INFO:teuthology.orchestra.run.trial030.stdout: kernel x86_64 5.14.0-686.el9 baseos 901 k 2026-02-25T08:02:33.032 INFO:teuthology.orchestra.run.trial030.stdout:Installing dependencies: 2026-02-25T08:02:33.032 INFO:teuthology.orchestra.run.trial030.stdout: kernel-core x86_64 5.14.0-686.el9 baseos 17 M 2026-02-25T08:02:33.032 INFO:teuthology.orchestra.run.trial030.stdout: kernel-modules x86_64 5.14.0-686.el9 baseos 40 M 2026-02-25T08:02:33.032 INFO:teuthology.orchestra.run.trial030.stdout: kernel-modules-core x86_64 5.14.0-686.el9 baseos 31 M 2026-02-25T08:02:33.032 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:02:33.032 INFO:teuthology.orchestra.run.trial030.stdout:Transaction Summary 2026-02-25T08:02:33.032 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-25T08:02:33.032 INFO:teuthology.orchestra.run.trial030.stdout:Install 4 Packages 2026-02-25T08:02:33.032 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:02:33.032 INFO:teuthology.orchestra.run.trial030.stdout:Total download size: 89 M 2026-02-25T08:02:33.032 INFO:teuthology.orchestra.run.trial030.stdout:Installed size: 135 M 2026-02-25T08:02:33.033 INFO:teuthology.orchestra.run.trial030.stdout:Downloading Packages: 2026-02-25T08:02:33.121 INFO:teuthology.orchestra.run.trial030.stdout:(1/4): kernel-5.14.0-686.el9.x86_64.rpm 31 MB/s | 901 kB 00:00 2026-02-25T08:02:33.216 INFO:teuthology.orchestra.run.trial030.stdout:(2/4): kernel-core-5.14.0-686.el9.x86_64.rpm 139 MB/s | 17 MB 00:00 2026-02-25T08:02:33.277 INFO:teuthology.orchestra.run.trial030.stdout:(3/4): kernel-modules-core-5.14.0-686.el9.x86_6 199 MB/s | 31 MB 00:00 2026-02-25T08:02:33.337 INFO:teuthology.orchestra.run.trial030.stdout:(4/4): kernel-modules-5.14.0-686.el9.x86_64.rpm 163 MB/s | 40 MB 00:00 2026-02-25T08:02:33.338 INFO:teuthology.orchestra.run.trial030.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:02:33.338 INFO:teuthology.orchestra.run.trial030.stdout:Total 291 MB/s | 89 MB 00:00 2026-02-25T08:02:33.464 INFO:teuthology.orchestra.run.trial023.stdout: Preparing : 1/1 2026-02-25T08:02:33.523 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction check 2026-02-25T08:02:33.729 INFO:teuthology.orchestra.run.trial030.stdout:Transaction check succeeded. 2026-02-25T08:02:33.729 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction test 2026-02-25T08:02:33.757 INFO:teuthology.orchestra.run.trial023.stdout: Installing : kernel-modules-core-5.14.0-686.el9.x86_64 1/4 2026-02-25T08:02:33.779 INFO:teuthology.orchestra.run.trial023.stdout: Installing : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-25T08:02:34.111 INFO:teuthology.orchestra.run.trial030.stdout:Transaction test succeeded. 2026-02-25T08:02:34.112 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction 2026-02-25T08:02:35.224 INFO:teuthology.orchestra.run.trial030.stdout: Preparing : 1/1 2026-02-25T08:02:35.512 INFO:teuthology.orchestra.run.trial030.stdout: Installing : kernel-modules-core-5.14.0-686.el9.x86_64 1/4 2026-02-25T08:02:35.534 INFO:teuthology.orchestra.run.trial030.stdout: Installing : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-25T08:02:36.343 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-25T08:02:36.376 INFO:teuthology.orchestra.run.trial023.stdout: Installing : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-25T08:02:38.109 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-25T08:02:38.141 INFO:teuthology.orchestra.run.trial030.stdout: Installing : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-25T08:02:39.480 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-25T08:02:39.486 INFO:teuthology.orchestra.run.trial023.stdout: Installing : kernel-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:02:41.179 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-25T08:02:41.186 INFO:teuthology.orchestra.run.trial030.stdout: Installing : kernel-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:02:42.718 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:02:44.398 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:03:00.234 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:03:00.256 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:03:00.444 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: kernel-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:03:00.444 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : kernel-5.14.0-686.el9.x86_64 1/4 2026-02-25T08:03:00.444 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-25T08:03:00.444 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-25T08:03:00.501 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:03:00.501 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:03:00.501 INFO:teuthology.orchestra.run.trial023.stdout:Installed: 2026-02-25T08:03:00.502 INFO:teuthology.orchestra.run.trial023.stdout: kernel-5.14.0-686.el9.x86_64 2026-02-25T08:03:00.502 INFO:teuthology.orchestra.run.trial023.stdout: kernel-core-5.14.0-686.el9.x86_64 2026-02-25T08:03:00.502 INFO:teuthology.orchestra.run.trial023.stdout: kernel-modules-5.14.0-686.el9.x86_64 2026-02-25T08:03:00.502 INFO:teuthology.orchestra.run.trial023.stdout: kernel-modules-core-5.14.0-686.el9.x86_64 2026-02-25T08:03:00.502 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:03:00.502 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:03:00.563 DEBUG:teuthology.orchestra.run.trial023:> rpm -q kernel | sort -rV | head -n 1 2026-02-25T08:03:00.633 INFO:teuthology.orchestra.run.trial023.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-25T08:03:00.633 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-25T08:03:00.633 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-681.el9.x86_64 Expected: 5.14.0-686.el9.x86_64 2026-02-25T08:03:00.634 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-25T08:03:00.634 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-25T08:03:00.634 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2026-02-25T08:03:00.634 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial023.front.sepia.ceph.com, path=None, version=distro) 2026-02-25T08:03:00.634 DEBUG:teuthology.orchestra.run.trial023:> rpm -q kernel | sort -rV | head -n 1 2026-02-25T08:03:00.698 INFO:teuthology.orchestra.run.trial023.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-25T08:03:00.699 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-25T08:03:00.699 DEBUG:teuthology.orchestra.run.trial023:> sudo rpm -qi grub2-tools 2026-02-25T08:03:00.768 INFO:teuthology.orchestra.run.trial023.stdout:Name : grub2-tools 2026-02-25T08:03:00.768 INFO:teuthology.orchestra.run.trial023.stdout:Epoch : 1 2026-02-25T08:03:00.768 INFO:teuthology.orchestra.run.trial023.stdout:Version : 2.06 2026-02-25T08:03:00.768 INFO:teuthology.orchestra.run.trial023.stdout:Release : 120.el9 2026-02-25T08:03:00.768 INFO:teuthology.orchestra.run.trial023.stdout:Architecture: x86_64 2026-02-25T08:03:00.769 INFO:teuthology.orchestra.run.trial023.stdout:Install Date: Wed 18 Feb 2026 05:18:42 PM UTC 2026-02-25T08:03:00.769 INFO:teuthology.orchestra.run.trial023.stdout:Group : Unspecified 2026-02-25T08:03:00.769 INFO:teuthology.orchestra.run.trial023.stdout:Size : 9108611 2026-02-25T08:03:00.769 INFO:teuthology.orchestra.run.trial023.stdout:License : GPLv3+ 2026-02-25T08:03:00.769 INFO:teuthology.orchestra.run.trial023.stdout:Signature : RSA/SHA256, Thu 04 Dec 2025 09:42:59 AM UTC, Key ID 05b555b38483c65d 2026-02-25T08:03:00.769 INFO:teuthology.orchestra.run.trial023.stdout:Source RPM : grub2-2.06-120.el9.src.rpm 2026-02-25T08:03:00.769 INFO:teuthology.orchestra.run.trial023.stdout:Build Date : Wed 03 Dec 2025 12:10:38 PM UTC 2026-02-25T08:03:00.769 INFO:teuthology.orchestra.run.trial023.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-25T08:03:00.769 INFO:teuthology.orchestra.run.trial023.stdout:Packager : builder@centos.org 2026-02-25T08:03:00.769 INFO:teuthology.orchestra.run.trial023.stdout:Vendor : CentOS 2026-02-25T08:03:00.769 INFO:teuthology.orchestra.run.trial023.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-25T08:03:00.769 INFO:teuthology.orchestra.run.trial023.stdout:Summary : Support tools for GRUB. 2026-02-25T08:03:00.769 INFO:teuthology.orchestra.run.trial023.stdout:Description : 2026-02-25T08:03:00.770 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:03:00.770 INFO:teuthology.orchestra.run.trial023.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-25T08:03:00.770 INFO:teuthology.orchestra.run.trial023.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-25T08:03:00.770 INFO:teuthology.orchestra.run.trial023.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-25T08:03:00.770 INFO:teuthology.orchestra.run.trial023.stdout:hardware devices. 2026-02-25T08:03:00.770 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:03:00.770 INFO:teuthology.orchestra.run.trial023.stdout:This subpackage provides tools for support of all platforms. 2026-02-25T08:03:00.771 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-25T08:03:00.772 INFO:teuthology.task.kernel:Updating grub on trial023 to boot 5.14.0-686.el9.x86_64 2026-02-25T08:03:00.772 DEBUG:teuthology.orchestra.run.trial023:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-25T08:03:00.834 INFO:teuthology.orchestra.run.trial023.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64.conf 2026-02-25T08:03:00.836 DEBUG:teuthology.orchestra.run.trial023:> sudo command -v grubby && echo yes || echo no 2026-02-25T08:03:00.899 INFO:teuthology.orchestra.run.trial023.stdout:/sbin/grubby 2026-02-25T08:03:00.900 INFO:teuthology.orchestra.run.trial023.stdout:yes 2026-02-25T08:03:00.901 DEBUG:teuthology.orchestra.run.trial023:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-686.el9.x86_64" | head -n 1 2026-02-25T08:03:00.965 INFO:teuthology.orchestra.run.trial023.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64.conf 2026-02-25T08:03:00.966 DEBUG:teuthology.orchestra.run.trial023:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64 2026-02-25T08:03:01.039 DEBUG:teuthology.orchestra.run.trial023:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-25T08:03:01.243 INFO:teuthology.orchestra.run.trial023.stderr:Generating grub configuration file ... 2026-02-25T08:03:01.310 INFO:teuthology.orchestra.run.trial023.stderr:Found linux image: /boot/vmlinuz-5.14.0-686.el9.x86_64 2026-02-25T08:03:01.316 INFO:teuthology.orchestra.run.trial023.stderr:Found initrd image: /boot/initramfs-5.14.0-686.el9.x86_64.img 2026-02-25T08:03:01.569 INFO:teuthology.orchestra.run.trial023.stderr:Found linux image: /boot/vmlinuz-5.14.0-681.el9.x86_64 2026-02-25T08:03:01.575 INFO:teuthology.orchestra.run.trial023.stderr:Found initrd image: /boot/initramfs-5.14.0-681.el9.x86_64.img 2026-02-25T08:03:01.608 INFO:teuthology.orchestra.run.trial023.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-25T08:03:01.615 INFO:teuthology.orchestra.run.trial023.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-25T08:03:01.645 INFO:teuthology.orchestra.run.trial023.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-25T08:03:01.652 INFO:teuthology.orchestra.run.trial023.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-25T08:03:01.674 INFO:teuthology.orchestra.run.trial023.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-25T08:03:01.680 INFO:teuthology.orchestra.run.trial023.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-25T08:03:01.766 INFO:teuthology.orchestra.run.trial023.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-25T08:03:01.774 INFO:teuthology.orchestra.run.trial023.stderr:done 2026-02-25T08:03:01.776 DEBUG:teuthology.orchestra.run.trial023:> test -d /sys/firmware/efi 2026-02-25T08:03:01.791 DEBUG:teuthology.orchestra.run.trial023:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-25T08:03:01.856 INFO:teuthology.orchestra.run.trial023.stdout:/boot/efi/EFI/BOOT 2026-02-25T08:03:01.856 INFO:teuthology.orchestra.run.trial023.stdout:/boot/efi/EFI/centos 2026-02-25T08:03:01.858 DEBUG:teuthology.orchestra.run.trial023:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-25T08:03:01.923 INFO:teuthology.orchestra.run.trial023.stdout:no 2026-02-25T08:03:01.924 DEBUG:teuthology.orchestra.run.trial023:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-25T08:03:01.988 INFO:teuthology.orchestra.run.trial023.stdout:yes 2026-02-25T08:03:01.989 DEBUG:teuthology.orchestra.run.trial023:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-25T08:03:02.019 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:03:02.041 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:03:02.052 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial023 2026-02-25T08:03:02.053 DEBUG:teuthology.orchestra.run.trial023:> sudo bash -c ' 2026-02-25T08:03:02.053 DEBUG:teuthology.orchestra.run.trial023:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-25T08:03:02.053 DEBUG:teuthology.orchestra.run.trial023:> ' 2026-02-25T08:03:02.226 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: kernel-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:03:02.227 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : kernel-5.14.0-686.el9.x86_64 1/4 2026-02-25T08:03:02.227 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-25T08:03:02.227 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-25T08:03:02.283 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:03:02.283 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:03:02.283 INFO:teuthology.orchestra.run.trial030.stdout:Installed: 2026-02-25T08:03:02.283 INFO:teuthology.orchestra.run.trial030.stdout: kernel-5.14.0-686.el9.x86_64 2026-02-25T08:03:02.283 INFO:teuthology.orchestra.run.trial030.stdout: kernel-core-5.14.0-686.el9.x86_64 2026-02-25T08:03:02.283 INFO:teuthology.orchestra.run.trial030.stdout: kernel-modules-5.14.0-686.el9.x86_64 2026-02-25T08:03:02.283 INFO:teuthology.orchestra.run.trial030.stdout: kernel-modules-core-5.14.0-686.el9.x86_64 2026-02-25T08:03:02.283 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:03:02.284 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:03:02.344 DEBUG:teuthology.orchestra.run.trial030:> rpm -q kernel | sort -rV | head -n 1 2026-02-25T08:03:02.371 INFO:teuthology.orchestra.run.trial030.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-25T08:03:02.373 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-25T08:03:02.373 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-681.el9.x86_64 Expected: 5.14.0-686.el9.x86_64 2026-02-25T08:03:02.373 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-25T08:03:02.373 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-25T08:03:02.373 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2026-02-25T08:03:02.373 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial030.front.sepia.ceph.com, path=None, version=distro) 2026-02-25T08:03:02.373 DEBUG:teuthology.orchestra.run.trial030:> rpm -q kernel | sort -rV | head -n 1 2026-02-25T08:03:02.439 INFO:teuthology.orchestra.run.trial030.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-25T08:03:02.440 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-25T08:03:02.440 DEBUG:teuthology.orchestra.run.trial030:> sudo rpm -qi grub2-tools 2026-02-25T08:03:02.512 INFO:teuthology.orchestra.run.trial030.stdout:Name : grub2-tools 2026-02-25T08:03:02.512 INFO:teuthology.orchestra.run.trial030.stdout:Epoch : 1 2026-02-25T08:03:02.512 INFO:teuthology.orchestra.run.trial030.stdout:Version : 2.06 2026-02-25T08:03:02.513 INFO:teuthology.orchestra.run.trial030.stdout:Release : 120.el9 2026-02-25T08:03:02.513 INFO:teuthology.orchestra.run.trial030.stdout:Architecture: x86_64 2026-02-25T08:03:02.513 INFO:teuthology.orchestra.run.trial030.stdout:Install Date: Wed 18 Feb 2026 05:18:42 PM UTC 2026-02-25T08:03:02.513 INFO:teuthology.orchestra.run.trial030.stdout:Group : Unspecified 2026-02-25T08:03:02.513 INFO:teuthology.orchestra.run.trial030.stdout:Size : 9108611 2026-02-25T08:03:02.513 INFO:teuthology.orchestra.run.trial030.stdout:License : GPLv3+ 2026-02-25T08:03:02.513 INFO:teuthology.orchestra.run.trial030.stdout:Signature : RSA/SHA256, Thu 04 Dec 2025 09:42:59 AM UTC, Key ID 05b555b38483c65d 2026-02-25T08:03:02.513 INFO:teuthology.orchestra.run.trial030.stdout:Source RPM : grub2-2.06-120.el9.src.rpm 2026-02-25T08:03:02.513 INFO:teuthology.orchestra.run.trial030.stdout:Build Date : Wed 03 Dec 2025 12:10:38 PM UTC 2026-02-25T08:03:02.513 INFO:teuthology.orchestra.run.trial030.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-25T08:03:02.514 INFO:teuthology.orchestra.run.trial030.stdout:Packager : builder@centos.org 2026-02-25T08:03:02.514 INFO:teuthology.orchestra.run.trial030.stdout:Vendor : CentOS 2026-02-25T08:03:02.514 INFO:teuthology.orchestra.run.trial030.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-25T08:03:02.514 INFO:teuthology.orchestra.run.trial030.stdout:Summary : Support tools for GRUB. 2026-02-25T08:03:02.514 INFO:teuthology.orchestra.run.trial030.stdout:Description : 2026-02-25T08:03:02.514 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:03:02.514 INFO:teuthology.orchestra.run.trial030.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-25T08:03:02.514 INFO:teuthology.orchestra.run.trial030.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-25T08:03:02.514 INFO:teuthology.orchestra.run.trial030.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-25T08:03:02.514 INFO:teuthology.orchestra.run.trial030.stdout:hardware devices. 2026-02-25T08:03:02.515 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:03:02.515 INFO:teuthology.orchestra.run.trial030.stdout:This subpackage provides tools for support of all platforms. 2026-02-25T08:03:02.516 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-25T08:03:02.516 INFO:teuthology.task.kernel:Updating grub on trial030 to boot 5.14.0-686.el9.x86_64 2026-02-25T08:03:02.516 DEBUG:teuthology.orchestra.run.trial030:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-25T08:03:02.580 INFO:teuthology.orchestra.run.trial030.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64.conf 2026-02-25T08:03:02.581 DEBUG:teuthology.orchestra.run.trial030:> sudo command -v grubby && echo yes || echo no 2026-02-25T08:03:02.644 INFO:teuthology.orchestra.run.trial030.stdout:/sbin/grubby 2026-02-25T08:03:02.646 INFO:teuthology.orchestra.run.trial030.stdout:yes 2026-02-25T08:03:02.646 DEBUG:teuthology.orchestra.run.trial030:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-686.el9.x86_64" | head -n 1 2026-02-25T08:03:02.710 INFO:teuthology.orchestra.run.trial030.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64.conf 2026-02-25T08:03:02.710 DEBUG:teuthology.orchestra.run.trial030:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64 2026-02-25T08:03:02.783 DEBUG:teuthology.orchestra.run.trial030:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-25T08:03:02.990 INFO:teuthology.orchestra.run.trial030.stderr:Generating grub configuration file ... 2026-02-25T08:03:03.058 INFO:teuthology.orchestra.run.trial030.stderr:Found linux image: /boot/vmlinuz-5.14.0-686.el9.x86_64 2026-02-25T08:03:03.065 INFO:teuthology.orchestra.run.trial030.stderr:Found initrd image: /boot/initramfs-5.14.0-686.el9.x86_64.img 2026-02-25T08:03:03.317 INFO:teuthology.orchestra.run.trial030.stderr:Found linux image: /boot/vmlinuz-5.14.0-681.el9.x86_64 2026-02-25T08:03:03.324 INFO:teuthology.orchestra.run.trial030.stderr:Found initrd image: /boot/initramfs-5.14.0-681.el9.x86_64.img 2026-02-25T08:03:03.358 INFO:teuthology.orchestra.run.trial030.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-25T08:03:03.364 INFO:teuthology.orchestra.run.trial030.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-25T08:03:03.396 INFO:teuthology.orchestra.run.trial030.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-25T08:03:03.403 INFO:teuthology.orchestra.run.trial030.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-25T08:03:03.425 INFO:teuthology.orchestra.run.trial030.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-25T08:03:03.432 INFO:teuthology.orchestra.run.trial030.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-25T08:03:03.518 INFO:teuthology.orchestra.run.trial030.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-25T08:03:03.527 INFO:teuthology.orchestra.run.trial030.stderr:done 2026-02-25T08:03:03.529 DEBUG:teuthology.orchestra.run.trial030:> test -d /sys/firmware/efi 2026-02-25T08:03:03.545 DEBUG:teuthology.orchestra.run.trial030:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-25T08:03:03.611 INFO:teuthology.orchestra.run.trial030.stdout:/boot/efi/EFI/BOOT 2026-02-25T08:03:03.611 INFO:teuthology.orchestra.run.trial030.stdout:/boot/efi/EFI/centos 2026-02-25T08:03:03.613 DEBUG:teuthology.orchestra.run.trial030:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-25T08:03:03.678 INFO:teuthology.orchestra.run.trial030.stdout:no 2026-02-25T08:03:03.679 DEBUG:teuthology.orchestra.run.trial030:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-25T08:03:03.743 INFO:teuthology.orchestra.run.trial030.stdout:yes 2026-02-25T08:03:03.744 DEBUG:teuthology.orchestra.run.trial030:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-25T08:03:03.809 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial030 2026-02-25T08:03:03.810 DEBUG:teuthology.orchestra.run.trial030:> sudo bash -c ' 2026-02-25T08:03:03.810 DEBUG:teuthology.orchestra.run.trial030:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-25T08:03:03.810 DEBUG:teuthology.orchestra.run.trial030:> ' 2026-02-25T08:03:32.095 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2026-02-25T08:03:32.096 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial023.front.sepia.ceph.com' 2026-02-25T08:03:32.097 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T08:03:33.853 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2026-02-25T08:03:33.854 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial030.front.sepia.ceph.com' 2026-02-25T08:03:33.855 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial030.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T08:03:46.368 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 10.20.193.23 2026-02-25T08:03:48.160 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 10.20.193.30 2026-02-25T08:03:55.375 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial023.front.sepia.ceph.com' 2026-02-25T08:03:55.376 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T08:03:57.161 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial030.front.sepia.ceph.com' 2026-02-25T08:03:57.162 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial030.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T08:04:31.383 DEBUG:teuthology.orchestra.run.trial023:> true 2026-02-25T08:04:31.539 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial023.front.sepia.ceph.com' 2026-02-25T08:04:31.540 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-686.el9.x86_64"... 2026-02-25T08:04:31.540 DEBUG:teuthology.orchestra.run.trial023:> uname -r 2026-02-25T08:04:31.596 INFO:teuthology.orchestra.run.trial023.stdout:5.14.0-686.el9.x86_64 2026-02-25T08:04:31.596 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-686.el9.x86_64 vs 5.14.0-686.el9.x86_64 2026-02-25T08:04:31.596 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-25T08:04:31.596 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-25T08:04:32.597 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-25T08:04:32.598 DEBUG:teuthology.orchestra.run.trial023:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-25T08:04:32.628 INFO:teuthology.orchestra.run.trial023.stdout:ttyS1 2026-02-25T08:04:32.647 DEBUG:teuthology.parallel:result is None 2026-02-25T08:04:33.408 DEBUG:teuthology.orchestra.run.trial030:> true 2026-02-25T08:04:33.558 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial030.front.sepia.ceph.com' 2026-02-25T08:04:33.558 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.14.0-686.el9.x86_64"... 2026-02-25T08:04:33.558 DEBUG:teuthology.orchestra.run.trial030:> uname -r 2026-02-25T08:04:33.614 INFO:teuthology.orchestra.run.trial030.stdout:5.14.0-686.el9.x86_64 2026-02-25T08:04:33.614 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-686.el9.x86_64 vs 5.14.0-686.el9.x86_64 2026-02-25T08:04:33.614 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-25T08:04:33.614 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-25T08:04:34.615 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2026-02-25T08:04:34.615 DEBUG:teuthology.orchestra.run.trial030:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-25T08:04:34.645 INFO:teuthology.orchestra.run.trial030.stdout:ttyS1 2026-02-25T08:04:34.665 DEBUG:teuthology.parallel:result is None 2026-02-25T08:04:34.665 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-25T08:04:34.677 INFO:teuthology.task.internal:Creating test directory... 2026-02-25T08:04:34.678 DEBUG:teuthology.orchestra.run.trial023:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-25T08:04:34.681 DEBUG:teuthology.orchestra.run.trial030:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-25T08:04:34.721 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-25T08:04:34.734 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-25T08:04:34.747 INFO:teuthology.task.internal:Creating archive directory... 2026-02-25T08:04:34.747 DEBUG:teuthology.orchestra.run.trial023:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-25T08:04:34.749 DEBUG:teuthology.orchestra.run.trial030:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-25T08:04:34.784 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-25T08:04:34.796 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-25T08:04:34.796 DEBUG:teuthology.orchestra.run.trial023:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-25T08:04:34.805 DEBUG:teuthology.orchestra.run.trial030:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-25T08:04:34.826 INFO:teuthology.orchestra.run.trial023.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T08:04:34.835 INFO:teuthology.orchestra.run.trial023.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T08:04:34.841 INFO:teuthology.orchestra.run.trial030.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T08:04:34.850 INFO:teuthology.orchestra.run.trial030.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T08:04:34.852 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-25T08:04:34.871 INFO:teuthology.task.internal:Configuring sudo... 2026-02-25T08:04:34.872 DEBUG:teuthology.orchestra.run.trial023:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-25T08:04:34.879 DEBUG:teuthology.orchestra.run.trial030:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-25T08:04:34.916 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-25T08:04:34.934 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-25T08:04:34.934 DEBUG:teuthology.orchestra.run.trial023:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-25T08:04:34.942 DEBUG:teuthology.orchestra.run.trial030:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-25T08:04:34.971 DEBUG:teuthology.orchestra.run.trial023:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T08:04:35.079 DEBUG:teuthology.orchestra.run.trial023:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T08:04:35.104 DEBUG:teuthology.orchestra.run.trial023:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T08:04:35.218 DEBUG:teuthology.orchestra.run.trial023:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T08:04:35.241 DEBUG:teuthology.orchestra.run.trial023:> set -ex 2026-02-25T08:04:35.241 DEBUG:teuthology.orchestra.run.trial023:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-25T08:04:35.305 DEBUG:teuthology.orchestra.run.trial030:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T08:04:35.404 DEBUG:teuthology.orchestra.run.trial030:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T08:04:35.429 DEBUG:teuthology.orchestra.run.trial030:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T08:04:35.561 DEBUG:teuthology.orchestra.run.trial030:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T08:04:35.586 DEBUG:teuthology.orchestra.run.trial030:> set -ex 2026-02-25T08:04:35.586 DEBUG:teuthology.orchestra.run.trial030:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-25T08:04:35.649 DEBUG:teuthology.orchestra.run.trial023:> sudo service rsyslog restart 2026-02-25T08:04:35.651 DEBUG:teuthology.orchestra.run.trial030:> sudo service rsyslog restart 2026-02-25T08:04:35.678 INFO:teuthology.orchestra.run.trial023.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-25T08:04:35.717 INFO:teuthology.orchestra.run.trial030.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-25T08:04:36.241 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-25T08:04:36.253 INFO:teuthology.task.internal:Starting timer... 2026-02-25T08:04:36.253 INFO:teuthology.run_tasks:Running task pcp... 2026-02-25T08:04:36.276 INFO:teuthology.run_tasks:Running task selinux... 2026-02-25T08:04:36.290 DEBUG:teuthology.orchestra.run.trial023:> sudo service auditd rotate 2026-02-25T08:04:36.320 INFO:teuthology.orchestra.run.trial023.stdout:Rotating logs: 2026-02-25T08:04:36.322 DEBUG:teuthology.orchestra.run.trial030:> sudo service auditd rotate 2026-02-25T08:04:36.351 INFO:teuthology.orchestra.run.trial030.stdout:Rotating logs: 2026-02-25T08:04:36.353 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-25T08:04:36.353 DEBUG:teuthology.orchestra.run.trial023:> /usr/sbin/getenforce 2026-02-25T08:04:36.377 INFO:teuthology.orchestra.run.trial023.stdout:Permissive 2026-02-25T08:04:36.377 DEBUG:teuthology.orchestra.run.trial030:> /usr/sbin/getenforce 2026-02-25T08:04:36.409 INFO:teuthology.orchestra.run.trial030.stdout:Permissive 2026-02-25T08:04:36.409 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial023.front.sepia.ceph.com': 'permissive', 'ubuntu@trial030.front.sepia.ceph.com': 'permissive'} 2026-02-25T08:04:36.410 DEBUG:teuthology.orchestra.run.trial023:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-25T08:04:36.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:04:36.442 DEBUG:teuthology.orchestra.run.trial030:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-25T08:04:36.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:04:36.475 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-25T08:04:36.475 DEBUG:teuthology.orchestra.run.trial023:> sudo /usr/sbin/setenforce permissive 2026-02-25T08:04:36.505 DEBUG:teuthology.orchestra.run.trial030:> sudo /usr/sbin/setenforce permissive 2026-02-25T08:04:36.538 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-25T08:04:36.555 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-25T08:04:36.562 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-25T08:04:36.563 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial023.front.sepia.ceph.com,trial030.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-25T08:07:11.088 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial023.front.sepia.ceph.com'), Remote(name='ubuntu@trial030.front.sepia.ceph.com')] 2026-02-25T08:07:11.089 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial023.front.sepia.ceph.com' 2026-02-25T08:07:11.089 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T08:07:11.162 DEBUG:teuthology.orchestra.run.trial023:> true 2026-02-25T08:07:11.234 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial023.front.sepia.ceph.com' 2026-02-25T08:07:11.234 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial030.front.sepia.ceph.com' 2026-02-25T08:07:11.235 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial030.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T08:07:11.304 DEBUG:teuthology.orchestra.run.trial030:> true 2026-02-25T08:07:11.378 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial030.front.sepia.ceph.com' 2026-02-25T08:07:11.379 INFO:teuthology.run_tasks:Running task clock... 2026-02-25T08:07:11.391 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-25T08:07:11.391 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-25T08:07:11.391 DEBUG:teuthology.orchestra.run.trial023:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-25T08:07:11.393 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-25T08:07:11.393 DEBUG:teuthology.orchestra.run.trial030:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-25T08:07:11.419 INFO:teuthology.orchestra.run.trial023.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-25T08:07:11.431 INFO:teuthology.orchestra.run.trial023.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-25T08:07:11.446 INFO:teuthology.orchestra.run.trial030.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-25T08:07:11.452 INFO:teuthology.orchestra.run.trial023.stderr:sudo: ntpd: command not found 2026-02-25T08:07:11.457 INFO:teuthology.orchestra.run.trial030.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-25T08:07:11.462 INFO:teuthology.orchestra.run.trial023.stdout:506 Cannot talk to daemon 2026-02-25T08:07:11.473 INFO:teuthology.orchestra.run.trial023.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-25T08:07:11.477 INFO:teuthology.orchestra.run.trial030.stderr:sudo: ntpd: command not found 2026-02-25T08:07:11.485 INFO:teuthology.orchestra.run.trial023.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-25T08:07:11.486 INFO:teuthology.orchestra.run.trial030.stdout:506 Cannot talk to daemon 2026-02-25T08:07:11.498 INFO:teuthology.orchestra.run.trial030.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-25T08:07:11.510 INFO:teuthology.orchestra.run.trial030.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-25T08:07:11.533 INFO:teuthology.orchestra.run.trial023.stderr:bash: line 1: ntpq: command not found 2026-02-25T08:07:11.536 INFO:teuthology.orchestra.run.trial023.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-25T08:07:11.536 INFO:teuthology.orchestra.run.trial023.stdout:=============================================================================== 2026-02-25T08:07:11.536 INFO:teuthology.orchestra.run.trial023.stdout:^? hou1.us.ntp.li 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-25T08:07:11.536 INFO:teuthology.orchestra.run.trial023.stdout:^? ip216-250-115-174.pbiaas> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-25T08:07:11.536 INFO:teuthology.orchestra.run.trial023.stdout:^? ntp4.glypnod.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-25T08:07:11.536 INFO:teuthology.orchestra.run.trial023.stdout:^? 172-232-15-202.ip.linode> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-25T08:07:11.558 INFO:teuthology.orchestra.run.trial030.stderr:bash: line 1: ntpq: command not found 2026-02-25T08:07:11.560 INFO:teuthology.orchestra.run.trial030.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-25T08:07:11.560 INFO:teuthology.orchestra.run.trial030.stdout:=============================================================================== 2026-02-25T08:07:11.560 INFO:teuthology.orchestra.run.trial030.stdout:^? ip216-250-115-174.pbiaas> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-25T08:07:11.560 INFO:teuthology.orchestra.run.trial030.stdout:^? ntp2.zeus.frumentum.media 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-25T08:07:11.560 INFO:teuthology.orchestra.run.trial030.stdout:^? 172-104-28-175.ip.linode> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-25T08:07:11.561 INFO:teuthology.orchestra.run.trial030.stdout:^? ns2.fortrockdc.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-25T08:07:11.561 INFO:teuthology.run_tasks:Running task install... 2026-02-25T08:07:11.573 DEBUG:teuthology.task.install:project ceph 2026-02-25T08:07:11.573 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}} 2026-02-25T08:07:11.573 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'} 2026-02-25T08:07:11.573 INFO:teuthology.task.install:Using flavor: debug 2026-02-25T08:07:11.576 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-25T08:07:11.576 INFO:teuthology.task.install:extra packages: {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']} 2026-02-25T08:07:11.576 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819', 'tag': None, 'wait_for_package': False} 2026-02-25T08:07:11.576 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:07:11.577 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819', 'tag': None, 'wait_for_package': False} 2026-02-25T08:07:11.577 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:07:11.774 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/ 2026-02-25T08:07:11.774 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5338.g6aeb6975 2026-02-25T08:07:11.775 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/ 2026-02-25T08:07:11.775 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5338.g6aeb6975 2026-02-25T08:07:12.114 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-25T08:07:12.114 DEBUG:teuthology.orchestra.run.trial023:> set -ex 2026-02-25T08:07:12.114 DEBUG:teuthology.orchestra.run.trial023:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-25T08:07:12.140 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo on remote rpm x86_64 2026-02-25T08:07:12.140 DEBUG:teuthology.orchestra.run.trial023:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-25T08:07:12.175 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-25T08:07:12.175 DEBUG:teuthology.orchestra.run.trial030:> set -ex 2026-02-25T08:07:12.175 DEBUG:teuthology.orchestra.run.trial030:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-25T08:07:12.201 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo on remote rpm x86_64 2026-02-25T08:07:12.201 DEBUG:teuthology.orchestra.run.trial030:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-25T08:07:12.205 DEBUG:teuthology.orchestra.run.trial023:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-25T08:07:12.265 DEBUG:teuthology.orchestra.run.trial030:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-25T08:07:12.277 DEBUG:teuthology.orchestra.run.trial023:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-25T08:07:12.338 DEBUG:teuthology.orchestra.run.trial030:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-25T08:07:12.341 INFO:teuthology.orchestra.run.trial023.stdout:check_obsoletes = 1 2026-02-25T08:07:12.343 DEBUG:teuthology.orchestra.run.trial023:> sudo yum clean all 2026-02-25T08:07:12.402 INFO:teuthology.orchestra.run.trial030.stdout:check_obsoletes = 1 2026-02-25T08:07:12.403 DEBUG:teuthology.orchestra.run.trial030:> sudo yum clean all 2026-02-25T08:07:12.490 INFO:teuthology.orchestra.run.trial023.stdout:41 files removed 2026-02-25T08:07:12.505 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install ceph-radosgw 2026-02-25T08:07:12.553 INFO:teuthology.orchestra.run.trial030.stdout:41 files removed 2026-02-25T08:07:12.567 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install ceph-radosgw 2026-02-25T08:07:13.310 INFO:teuthology.orchestra.run.trial023.stdout:ceph packages for x86_64 142 kB/s | 94 kB 00:00 2026-02-25T08:07:13.374 INFO:teuthology.orchestra.run.trial030.stdout:ceph packages for x86_64 142 kB/s | 94 kB 00:00 2026-02-25T08:07:13.988 INFO:teuthology.orchestra.run.trial023.stdout:ceph noarch packages 38 kB/s | 24 kB 00:00 2026-02-25T08:07:14.034 INFO:teuthology.orchestra.run.trial030.stdout:ceph noarch packages 39 kB/s | 24 kB 00:00 2026-02-25T08:07:14.768 INFO:teuthology.orchestra.run.trial030.stdout:ceph source packages 2.9 kB/s | 2.1 kB 00:00 2026-02-25T08:07:14.805 INFO:teuthology.orchestra.run.trial023.stdout:ceph source packages 2.6 kB/s | 2.1 kB 00:00 2026-02-25T08:07:15.581 INFO:teuthology.orchestra.run.trial023.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.9 MB 00:00 2026-02-25T08:07:16.185 INFO:teuthology.orchestra.run.trial030.stdout:CentOS Stream 9 - BaseOS 6.4 MB/s | 8.9 MB 00:01 2026-02-25T08:07:17.029 INFO:teuthology.orchestra.run.trial030.stdout:CentOS Stream 9 - AppStream 84 MB/s | 27 MB 00:00 2026-02-25T08:07:19.386 INFO:teuthology.orchestra.run.trial023.stdout:CentOS Stream 9 - AppStream 8.2 MB/s | 27 MB 00:03 2026-02-25T08:07:20.726 INFO:teuthology.orchestra.run.trial030.stdout:CentOS Stream 9 - CRB 6.2 MB/s | 8.0 MB 00:01 2026-02-25T08:07:21.884 INFO:teuthology.orchestra.run.trial030.stdout:CentOS Stream 9 - Extras packages 48 kB/s | 20 kB 00:00 2026-02-25T08:07:22.815 INFO:teuthology.orchestra.run.trial023.stdout:CentOS Stream 9 - CRB 7.5 MB/s | 8.0 MB 00:01 2026-02-25T08:07:23.490 INFO:teuthology.orchestra.run.trial030.stdout:Extra Packages for Enterprise Linux 13 MB/s | 20 MB 00:01 2026-02-25T08:07:23.716 INFO:teuthology.orchestra.run.trial023.stdout:CentOS Stream 9 - Extras packages 124 kB/s | 20 kB 00:00 2026-02-25T08:07:24.541 INFO:teuthology.orchestra.run.trial023.stdout:Extra Packages for Enterprise Linux 27 MB/s | 20 MB 00:00 2026-02-25T08:07:26.701 INFO:teuthology.orchestra.run.trial030.stdout:lab-extras 2.5 MB/s | 50 kB 00:00 2026-02-25T08:07:27.687 INFO:teuthology.orchestra.run.trial023.stdout:lab-extras 2.5 MB/s | 50 kB 00:00 2026-02-25T08:07:27.909 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:07:27.911 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-25T08:07:27.911 INFO:teuthology.orchestra.run.trial030.stdout: Package Arch Version Repository Size 2026-02-25T08:07:27.911 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-25T08:07:27.911 INFO:teuthology.orchestra.run.trial030.stdout:Installing: 2026-02-25T08:07:27.911 INFO:teuthology.orchestra.run.trial030.stdout: ceph-radosgw x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 11 M 2026-02-25T08:07:27.911 INFO:teuthology.orchestra.run.trial030.stdout:Upgrading: 2026-02-25T08:07:27.911 INFO:teuthology.orchestra.run.trial030.stdout: librados2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 3.3 M 2026-02-25T08:07:27.911 INFO:teuthology.orchestra.run.trial030.stdout: librbd1 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 2.7 M 2026-02-25T08:07:27.911 INFO:teuthology.orchestra.run.trial030.stdout:Installing dependencies: 2026-02-25T08:07:27.911 INFO:teuthology.orchestra.run.trial030.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-25T08:07:27.911 INFO:teuthology.orchestra.run.trial030.stdout: ceph-base x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 5.1 M 2026-02-25T08:07:27.911 INFO:teuthology.orchestra.run.trial030.stdout: ceph-common x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 21 M 2026-02-25T08:07:27.912 INFO:teuthology.orchestra.run.trial030.stdout: ceph-selinux x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 25 k 2026-02-25T08:07:27.912 INFO:teuthology.orchestra.run.trial030.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-25T08:07:27.912 INFO:teuthology.orchestra.run.trial030.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-25T08:07:27.912 INFO:teuthology.orchestra.run.trial030.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-25T08:07:27.912 INFO:teuthology.orchestra.run.trial030.stdout: libcephfs2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 860 k 2026-02-25T08:07:27.912 INFO:teuthology.orchestra.run.trial030.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-25T08:07:27.912 INFO:teuthology.orchestra.run.trial030.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-25T08:07:27.912 INFO:teuthology.orchestra.run.trial030.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-25T08:07:27.912 INFO:teuthology.orchestra.run.trial030.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-25T08:07:27.912 INFO:teuthology.orchestra.run.trial030.stdout: libradosstriper1 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 263 k 2026-02-25T08:07:27.912 INFO:teuthology.orchestra.run.trial030.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-25T08:07:27.912 INFO:teuthology.orchestra.run.trial030.stdout: librgw2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 5.6 M 2026-02-25T08:07:27.912 INFO:teuthology.orchestra.run.trial030.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-25T08:07:27.912 INFO:teuthology.orchestra.run.trial030.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-25T08:07:27.912 INFO:teuthology.orchestra.run.trial030.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-25T08:07:27.912 INFO:teuthology.orchestra.run.trial030.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-25T08:07:27.912 INFO:teuthology.orchestra.run.trial030.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-25T08:07:27.912 INFO:teuthology.orchestra.run.trial030.stdout: python3-ceph-argparse x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 45 k 2026-02-25T08:07:27.912 INFO:teuthology.orchestra.run.trial030.stdout: python3-ceph-common x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 185 k 2026-02-25T08:07:27.912 INFO:teuthology.orchestra.run.trial030.stdout: python3-cephfs x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 204 k 2026-02-25T08:07:27.912 INFO:teuthology.orchestra.run.trial030.stdout: python3-rados x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 324 k 2026-02-25T08:07:27.912 INFO:teuthology.orchestra.run.trial030.stdout: python3-rbd x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 306 k 2026-02-25T08:07:27.912 INFO:teuthology.orchestra.run.trial030.stdout: python3-rgw x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 100 k 2026-02-25T08:07:27.912 INFO:teuthology.orchestra.run.trial030.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-25T08:07:27.912 INFO:teuthology.orchestra.run.trial030.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-25T08:07:27.912 INFO:teuthology.orchestra.run.trial030.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-25T08:07:27.912 INFO:teuthology.orchestra.run.trial030.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-25T08:07:27.912 INFO:teuthology.orchestra.run.trial030.stdout:Installing weak dependencies: 2026-02-25T08:07:27.912 INFO:teuthology.orchestra.run.trial030.stdout: libcephfs-daemon x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 37 k 2026-02-25T08:07:27.913 INFO:teuthology.orchestra.run.trial030.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 26 k 2026-02-25T08:07:27.913 INFO:teuthology.orchestra.run.trial030.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-25T08:07:27.913 INFO:teuthology.orchestra.run.trial030.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-25T08:07:27.913 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:07:27.913 INFO:teuthology.orchestra.run.trial030.stdout:Transaction Summary 2026-02-25T08:07:27.913 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-25T08:07:27.913 INFO:teuthology.orchestra.run.trial030.stdout:Install 35 Packages 2026-02-25T08:07:27.913 INFO:teuthology.orchestra.run.trial030.stdout:Upgrade 2 Packages 2026-02-25T08:07:27.913 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:07:27.913 INFO:teuthology.orchestra.run.trial030.stdout:Total download size: 61 M 2026-02-25T08:07:27.913 INFO:teuthology.orchestra.run.trial030.stdout:Downloading Packages: 2026-02-25T08:07:28.904 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:07:28.905 INFO:teuthology.orchestra.run.trial023.stdout:================================================================================ 2026-02-25T08:07:28.905 INFO:teuthology.orchestra.run.trial023.stdout: Package Arch Version Repository Size 2026-02-25T08:07:28.906 INFO:teuthology.orchestra.run.trial023.stdout:================================================================================ 2026-02-25T08:07:28.906 INFO:teuthology.orchestra.run.trial023.stdout:Installing: 2026-02-25T08:07:28.906 INFO:teuthology.orchestra.run.trial023.stdout: ceph-radosgw x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 11 M 2026-02-25T08:07:28.906 INFO:teuthology.orchestra.run.trial023.stdout:Upgrading: 2026-02-25T08:07:28.906 INFO:teuthology.orchestra.run.trial023.stdout: librados2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 3.3 M 2026-02-25T08:07:28.906 INFO:teuthology.orchestra.run.trial023.stdout: librbd1 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 2.7 M 2026-02-25T08:07:28.906 INFO:teuthology.orchestra.run.trial023.stdout:Installing dependencies: 2026-02-25T08:07:28.906 INFO:teuthology.orchestra.run.trial023.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-25T08:07:28.906 INFO:teuthology.orchestra.run.trial023.stdout: ceph-base x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 5.1 M 2026-02-25T08:07:28.906 INFO:teuthology.orchestra.run.trial023.stdout: ceph-common x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 21 M 2026-02-25T08:07:28.906 INFO:teuthology.orchestra.run.trial023.stdout: ceph-selinux x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 25 k 2026-02-25T08:07:28.906 INFO:teuthology.orchestra.run.trial023.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-25T08:07:28.906 INFO:teuthology.orchestra.run.trial023.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-25T08:07:28.906 INFO:teuthology.orchestra.run.trial023.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-25T08:07:28.906 INFO:teuthology.orchestra.run.trial023.stdout: libcephfs2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 860 k 2026-02-25T08:07:28.906 INFO:teuthology.orchestra.run.trial023.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-25T08:07:28.906 INFO:teuthology.orchestra.run.trial023.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-25T08:07:28.906 INFO:teuthology.orchestra.run.trial023.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-25T08:07:28.906 INFO:teuthology.orchestra.run.trial023.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-25T08:07:28.906 INFO:teuthology.orchestra.run.trial023.stdout: libradosstriper1 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 263 k 2026-02-25T08:07:28.906 INFO:teuthology.orchestra.run.trial023.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-25T08:07:28.906 INFO:teuthology.orchestra.run.trial023.stdout: librgw2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 5.6 M 2026-02-25T08:07:28.906 INFO:teuthology.orchestra.run.trial023.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-25T08:07:28.906 INFO:teuthology.orchestra.run.trial023.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-25T08:07:28.906 INFO:teuthology.orchestra.run.trial023.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-25T08:07:28.906 INFO:teuthology.orchestra.run.trial023.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-25T08:07:28.906 INFO:teuthology.orchestra.run.trial023.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-25T08:07:28.906 INFO:teuthology.orchestra.run.trial023.stdout: python3-ceph-argparse x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 45 k 2026-02-25T08:07:28.906 INFO:teuthology.orchestra.run.trial023.stdout: python3-ceph-common x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 185 k 2026-02-25T08:07:28.906 INFO:teuthology.orchestra.run.trial023.stdout: python3-cephfs x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 204 k 2026-02-25T08:07:28.906 INFO:teuthology.orchestra.run.trial023.stdout: python3-rados x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 324 k 2026-02-25T08:07:28.906 INFO:teuthology.orchestra.run.trial023.stdout: python3-rbd x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 306 k 2026-02-25T08:07:28.906 INFO:teuthology.orchestra.run.trial023.stdout: python3-rgw x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 100 k 2026-02-25T08:07:28.906 INFO:teuthology.orchestra.run.trial023.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-25T08:07:28.906 INFO:teuthology.orchestra.run.trial023.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-25T08:07:28.906 INFO:teuthology.orchestra.run.trial023.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-25T08:07:28.907 INFO:teuthology.orchestra.run.trial023.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-25T08:07:28.907 INFO:teuthology.orchestra.run.trial023.stdout:Installing weak dependencies: 2026-02-25T08:07:28.907 INFO:teuthology.orchestra.run.trial023.stdout: libcephfs-daemon x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 37 k 2026-02-25T08:07:28.907 INFO:teuthology.orchestra.run.trial023.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 26 k 2026-02-25T08:07:28.907 INFO:teuthology.orchestra.run.trial023.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-25T08:07:28.907 INFO:teuthology.orchestra.run.trial023.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-25T08:07:28.907 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:07:28.907 INFO:teuthology.orchestra.run.trial023.stdout:Transaction Summary 2026-02-25T08:07:28.907 INFO:teuthology.orchestra.run.trial023.stdout:================================================================================ 2026-02-25T08:07:28.907 INFO:teuthology.orchestra.run.trial023.stdout:Install 35 Packages 2026-02-25T08:07:28.907 INFO:teuthology.orchestra.run.trial023.stdout:Upgrade 2 Packages 2026-02-25T08:07:28.907 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:07:28.907 INFO:teuthology.orchestra.run.trial023.stdout:Total download size: 61 M 2026-02-25T08:07:28.907 INFO:teuthology.orchestra.run.trial023.stdout:Downloading Packages: 2026-02-25T08:07:31.309 INFO:teuthology.orchestra.run.trial030.stdout:(1/37): ceph-base-20.3.0-5338.g6aeb6975.el9.x86 1.8 MB/s | 5.1 MB 00:02 2026-02-25T08:07:31.374 INFO:teuthology.orchestra.run.trial030.stdout:(2/37): ceph-selinux-20.3.0-5338.g6aeb6975.el9. 386 kB/s | 25 kB 00:00 2026-02-25T08:07:31.439 INFO:teuthology.orchestra.run.trial030.stdout:(3/37): libcephfs-daemon-20.3.0-5338.g6aeb6975. 573 kB/s | 37 kB 00:00 2026-02-25T08:07:31.505 INFO:teuthology.orchestra.run.trial030.stdout:(4/37): libcephfs-proxy2-20.3.0-5338.g6aeb6975. 389 kB/s | 26 kB 00:00 2026-02-25T08:07:32.524 INFO:teuthology.orchestra.run.trial030.stdout:(5/37): libcephfs2-20.3.0-5338.g6aeb6975.el9.x8 844 kB/s | 860 kB 00:01 2026-02-25T08:07:32.936 INFO:teuthology.orchestra.run.trial030.stdout:(6/37): libradosstriper1-20.3.0-5338.g6aeb6975. 639 kB/s | 263 kB 00:00 2026-02-25T08:07:34.485 INFO:teuthology.orchestra.run.trial023.stdout:(1/37): ceph-base-20.3.0-5338.g6aeb6975.el9.x86 1.0 MB/s | 5.1 MB 00:05 2026-02-25T08:07:34.552 INFO:teuthology.orchestra.run.trial023.stdout:(2/37): ceph-selinux-20.3.0-5338.g6aeb6975.el9. 377 kB/s | 25 kB 00:00 2026-02-25T08:07:34.618 INFO:teuthology.orchestra.run.trial023.stdout:(3/37): libcephfs-daemon-20.3.0-5338.g6aeb6975. 561 kB/s | 37 kB 00:00 2026-02-25T08:07:34.685 INFO:teuthology.orchestra.run.trial023.stdout:(4/37): libcephfs-proxy2-20.3.0-5338.g6aeb6975. 381 kB/s | 26 kB 00:00 2026-02-25T08:07:35.502 INFO:teuthology.orchestra.run.trial023.stdout:(5/37): libcephfs2-20.3.0-5338.g6aeb6975.el9.x8 1.0 MB/s | 860 kB 00:00 2026-02-25T08:07:35.776 INFO:teuthology.orchestra.run.trial023.stdout:(6/37): libradosstriper1-20.3.0-5338.g6aeb6975. 962 kB/s | 263 kB 00:00 2026-02-25T08:07:36.111 INFO:teuthology.orchestra.run.trial030.stdout:(7/37): ceph-radosgw-20.3.0-5338.g6aeb6975.el9. 1.5 MB/s | 11 MB 00:07 2026-02-25T08:07:36.174 INFO:teuthology.orchestra.run.trial030.stdout:(8/37): python3-ceph-argparse-20.3.0-5338.g6aeb 709 kB/s | 45 kB 00:00 2026-02-25T08:07:36.366 INFO:teuthology.orchestra.run.trial030.stdout:(9/37): python3-ceph-common-20.3.0-5338.g6aeb69 966 kB/s | 185 kB 00:00 2026-02-25T08:07:36.567 INFO:teuthology.orchestra.run.trial030.stdout:(10/37): python3-cephfs-20.3.0-5338.g6aeb6975.e 1.0 MB/s | 204 kB 00:00 2026-02-25T08:07:37.269 INFO:teuthology.orchestra.run.trial030.stdout:(11/37): python3-rados-20.3.0-5338.g6aeb6975.el 462 kB/s | 324 kB 00:00 2026-02-25T08:07:37.873 INFO:teuthology.orchestra.run.trial030.stdout:(12/37): python3-rbd-20.3.0-5338.g6aeb6975.el9. 506 kB/s | 306 kB 00:00 2026-02-25T08:07:38.081 INFO:teuthology.orchestra.run.trial030.stdout:(13/37): python3-rgw-20.3.0-5338.g6aeb6975.el9. 481 kB/s | 100 kB 00:00 2026-02-25T08:07:38.150 INFO:teuthology.orchestra.run.trial030.stdout:(14/37): mailcap-2.1.49-5.el9.noarch.rpm 485 kB/s | 33 kB 00:00 2026-02-25T08:07:38.270 INFO:teuthology.orchestra.run.trial030.stdout:(15/37): nvme-cli-2.16-1.el9.x86_64.rpm 9.6 MB/s | 1.2 MB 00:00 2026-02-25T08:07:38.290 INFO:teuthology.orchestra.run.trial030.stdout:(16/37): pciutils-3.7.0-7.el9.x86_64.rpm 4.4 MB/s | 93 kB 00:00 2026-02-25T08:07:38.451 INFO:teuthology.orchestra.run.trial030.stdout:(17/37): boost-program-options-1.75.0-13.el9.x8 647 kB/s | 104 kB 00:00 2026-02-25T08:07:38.470 INFO:teuthology.orchestra.run.trial030.stdout:(18/37): libnbd-1.20.3-4.el9.x86_64.rpm 8.8 MB/s | 164 kB 00:00 2026-02-25T08:07:38.481 INFO:teuthology.orchestra.run.trial030.stdout:(19/37): libpmemobj-1.12.1-1.el9.x86_64.rpm 14 MB/s | 160 kB 00:00 2026-02-25T08:07:38.491 INFO:teuthology.orchestra.run.trial030.stdout:(20/37): librabbitmq-0.11.0-7.el9.x86_64.rpm 4.8 MB/s | 45 kB 00:00 2026-02-25T08:07:38.512 INFO:teuthology.orchestra.run.trial030.stdout:(21/37): librdkafka-1.6.1-102.el9.x86_64.rpm 30 MB/s | 662 kB 00:00 2026-02-25T08:07:38.524 INFO:teuthology.orchestra.run.trial030.stdout:(22/37): lttng-ust-2.12.0-6.el9.x86_64.rpm 26 MB/s | 292 kB 00:00 2026-02-25T08:07:38.535 INFO:teuthology.orchestra.run.trial030.stdout:(23/37): qatlib-25.08.0-2.el9.x86_64.rpm 21 MB/s | 240 kB 00:00 2026-02-25T08:07:38.544 INFO:teuthology.orchestra.run.trial030.stdout:(24/37): qatlib-service-25.08.0-2.el9.x86_64.rp 4.1 MB/s | 37 kB 00:00 2026-02-25T08:07:38.553 INFO:teuthology.orchestra.run.trial030.stdout:(25/37): qatzip-libs-1.3.1-1.el9.x86_64.rpm 7.0 MB/s | 66 kB 00:00 2026-02-25T08:07:38.974 INFO:teuthology.orchestra.run.trial030.stdout:(26/37): gperftools-libs-2.9.1-3.el9.x86_64.rpm 731 kB/s | 308 kB 00:00 2026-02-25T08:07:39.651 INFO:teuthology.orchestra.run.trial030.stdout:(27/37): librgw2-20.3.0-5338.g6aeb6975.el9.x86_ 858 kB/s | 5.6 MB 00:06 2026-02-25T08:07:39.685 INFO:teuthology.orchestra.run.trial023.stdout:(7/37): ceph-radosgw-20.3.0-5338.g6aeb6975.el9. 1.1 MB/s | 11 MB 00:10 2026-02-25T08:07:39.754 INFO:teuthology.orchestra.run.trial030.stdout:(28/37): libarrow-9.0.0-15.el9.x86_64.rpm 5.7 MB/s | 4.4 MB 00:00 2026-02-25T08:07:39.789 INFO:teuthology.orchestra.run.trial023.stdout:(8/37): python3-ceph-argparse-20.3.0-5338.g6aeb 437 kB/s | 45 kB 00:00 2026-02-25T08:07:39.859 INFO:teuthology.orchestra.run.trial030.stdout:(29/37): libarrow-doc-9.0.0-15.el9.noarch.rpm 119 kB/s | 25 kB 00:00 2026-02-25T08:07:40.040 INFO:teuthology.orchestra.run.trial030.stdout:(30/37): liboath-2.6.12-1.el9.x86_64.rpm 171 kB/s | 49 kB 00:00 2026-02-25T08:07:40.086 INFO:teuthology.orchestra.run.trial023.stdout:(9/37): python3-ceph-common-20.3.0-5338.g6aeb69 623 kB/s | 185 kB 00:00 2026-02-25T08:07:40.139 INFO:teuthology.orchestra.run.trial030.stdout:(31/37): libunwind-1.6.2-1.el9.x86_64.rpm 240 kB/s | 67 kB 00:00 2026-02-25T08:07:40.275 INFO:teuthology.orchestra.run.trial023.stdout:(10/37): librgw2-20.3.0-5338.g6aeb6975.el9.x86_ 1.3 MB/s | 5.6 MB 00:04 2026-02-25T08:07:40.405 INFO:teuthology.orchestra.run.trial023.stdout:(11/37): python3-cephfs-20.3.0-5338.g6aeb6975.e 637 kB/s | 204 kB 00:00 2026-02-25T08:07:40.498 INFO:teuthology.orchestra.run.trial030.stdout:(32/37): re2-20211101-20.el9.x86_64.rpm 533 kB/s | 191 kB 00:00 2026-02-25T08:07:40.544 INFO:teuthology.orchestra.run.trial023.stdout:(12/37): python3-rados-20.3.0-5338.g6aeb6975.el 1.2 MB/s | 324 kB 00:00 2026-02-25T08:07:40.547 INFO:teuthology.orchestra.run.trial030.stdout:(33/37): parquet-libs-9.0.0-15.el9.x86_64.rpm 1.6 MB/s | 838 kB 00:00 2026-02-25T08:07:40.678 INFO:teuthology.orchestra.run.trial023.stdout:(13/37): python3-rgw-20.3.0-5338.g6aeb6975.el9. 746 kB/s | 100 kB 00:00 2026-02-25T08:07:40.742 INFO:teuthology.orchestra.run.trial023.stdout:(14/37): mailcap-2.1.49-5.el9.noarch.rpm 519 kB/s | 33 kB 00:00 2026-02-25T08:07:40.802 INFO:teuthology.orchestra.run.trial023.stdout:(15/37): python3-rbd-20.3.0-5338.g6aeb6975.el9. 771 kB/s | 306 kB 00:00 2026-02-25T08:07:40.851 INFO:teuthology.orchestra.run.trial023.stdout:(16/37): nvme-cli-2.16-1.el9.x86_64.rpm 11 MB/s | 1.2 MB 00:00 2026-02-25T08:07:40.883 INFO:teuthology.orchestra.run.trial023.stdout:(17/37): pciutils-3.7.0-7.el9.x86_64.rpm 1.1 MB/s | 93 kB 00:00 2026-02-25T08:07:41.054 INFO:teuthology.orchestra.run.trial030.stdout:(34/37): thrift-0.15.0-4.el9.x86_64.rpm 2.8 MB/s | 1.6 MB 00:00 2026-02-25T08:07:41.492 INFO:teuthology.orchestra.run.trial023.stdout:(18/37): boost-program-options-1.75.0-13.el9.x8 162 kB/s | 104 kB 00:00 2026-02-25T08:07:41.581 INFO:teuthology.orchestra.run.trial023.stdout:(19/37): libnbd-1.20.3-4.el9.x86_64.rpm 235 kB/s | 164 kB 00:00 2026-02-25T08:07:42.147 INFO:teuthology.orchestra.run.trial023.stdout:(20/37): librabbitmq-0.11.0-7.el9.x86_64.rpm 80 kB/s | 45 kB 00:00 2026-02-25T08:07:42.191 INFO:teuthology.orchestra.run.trial023.stdout:(21/37): libpmemobj-1.12.1-1.el9.x86_64.rpm 229 kB/s | 160 kB 00:00 2026-02-25T08:07:42.942 INFO:teuthology.orchestra.run.trial023.stdout:(22/37): lttng-ust-2.12.0-6.el9.x86_64.rpm 389 kB/s | 292 kB 00:00 2026-02-25T08:07:42.989 INFO:teuthology.orchestra.run.trial023.stdout:(23/37): librdkafka-1.6.1-102.el9.x86_64.rpm 787 kB/s | 662 kB 00:00 2026-02-25T08:07:43.559 INFO:teuthology.orchestra.run.trial023.stdout:(24/37): qatlib-service-25.08.0-2.el9.x86_64.rp 65 kB/s | 37 kB 00:00 2026-02-25T08:07:43.677 INFO:teuthology.orchestra.run.trial023.stdout:(25/37): qatlib-25.08.0-2.el9.x86_64.rpm 326 kB/s | 240 kB 00:00 2026-02-25T08:07:43.716 INFO:teuthology.orchestra.run.trial023.stdout:(26/37): gperftools-libs-2.9.1-3.el9.x86_64.rpm 7.8 MB/s | 308 kB 00:00 2026-02-25T08:07:43.747 INFO:teuthology.orchestra.run.trial023.stdout:(27/37): libarrow-9.0.0-15.el9.x86_64.rpm 145 MB/s | 4.4 MB 00:00 2026-02-25T08:07:43.753 INFO:teuthology.orchestra.run.trial023.stdout:(28/37): libarrow-doc-9.0.0-15.el9.noarch.rpm 3.7 MB/s | 25 kB 00:00 2026-02-25T08:07:43.759 INFO:teuthology.orchestra.run.trial023.stdout:(29/37): liboath-2.6.12-1.el9.x86_64.rpm 7.8 MB/s | 49 kB 00:00 2026-02-25T08:07:43.765 INFO:teuthology.orchestra.run.trial023.stdout:(30/37): libunwind-1.6.2-1.el9.x86_64.rpm 11 MB/s | 67 kB 00:00 2026-02-25T08:07:43.777 INFO:teuthology.orchestra.run.trial023.stdout:(31/37): parquet-libs-9.0.0-15.el9.x86_64.rpm 68 MB/s | 838 kB 00:00 2026-02-25T08:07:43.785 INFO:teuthology.orchestra.run.trial023.stdout:(32/37): re2-20211101-20.el9.x86_64.rpm 25 MB/s | 191 kB 00:00 2026-02-25T08:07:43.798 INFO:teuthology.orchestra.run.trial023.stdout:(33/37): thrift-0.15.0-4.el9.x86_64.rpm 124 MB/s | 1.6 MB 00:00 2026-02-25T08:07:43.945 INFO:teuthology.orchestra.run.trial030.stdout:(35/37): librados2-20.3.0-5338.g6aeb6975.el9.x8 989 kB/s | 3.3 MB 00:03 2026-02-25T08:07:44.182 INFO:teuthology.orchestra.run.trial023.stdout:(34/37): qatzip-libs-1.3.1-1.el9.x86_64.rpm 106 kB/s | 66 kB 00:00 2026-02-25T08:07:44.278 INFO:teuthology.orchestra.run.trial030.stdout:(36/37): librbd1-20.3.0-5338.g6aeb6975.el9.x86_ 854 kB/s | 2.7 MB 00:03 2026-02-25T08:07:45.624 INFO:teuthology.orchestra.run.trial023.stdout:(35/37): ceph-common-20.3.0-5338.g6aeb6975.el9. 1.3 MB/s | 21 MB 00:16 2026-02-25T08:07:46.426 INFO:teuthology.orchestra.run.trial023.stdout:(36/37): librados2-20.3.0-5338.g6aeb6975.el9.x8 1.2 MB/s | 3.3 MB 00:02 2026-02-25T08:07:47.422 INFO:teuthology.orchestra.run.trial030.stdout:(37/37): ceph-common-20.3.0-5338.g6aeb6975.el9. 1.1 MB/s | 21 MB 00:18 2026-02-25T08:07:47.424 INFO:teuthology.orchestra.run.trial030.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:07:47.424 INFO:teuthology.orchestra.run.trial030.stdout:Total 3.1 MB/s | 61 MB 00:19 2026-02-25T08:07:47.517 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction check 2026-02-25T08:07:47.535 INFO:teuthology.orchestra.run.trial030.stdout:Transaction check succeeded. 2026-02-25T08:07:47.535 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction test 2026-02-25T08:07:47.710 INFO:teuthology.orchestra.run.trial023.stdout:(37/37): librbd1-20.3.0-5338.g6aeb6975.el9.x86_ 781 kB/s | 2.7 MB 00:03 2026-02-25T08:07:47.714 INFO:teuthology.orchestra.run.trial023.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:07:47.714 INFO:teuthology.orchestra.run.trial023.stdout:Total 3.3 MB/s | 61 MB 00:18 2026-02-25T08:07:47.780 INFO:teuthology.orchestra.run.trial030.stdout:Transaction test succeeded. 2026-02-25T08:07:47.781 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction 2026-02-25T08:07:47.809 INFO:teuthology.orchestra.run.trial023.stdout:Running transaction check 2026-02-25T08:07:47.829 INFO:teuthology.orchestra.run.trial023.stdout:Transaction check succeeded. 2026-02-25T08:07:47.829 INFO:teuthology.orchestra.run.trial023.stdout:Running transaction test 2026-02-25T08:07:48.071 INFO:teuthology.orchestra.run.trial030.stdout: Preparing : 1/1 2026-02-25T08:07:48.076 INFO:teuthology.orchestra.run.trial023.stdout:Transaction test succeeded. 2026-02-25T08:07:48.076 INFO:teuthology.orchestra.run.trial023.stdout:Running transaction 2026-02-25T08:07:48.215 INFO:teuthology.orchestra.run.trial030.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/39 2026-02-25T08:07:48.220 INFO:teuthology.orchestra.run.trial030.stdout: Upgrading : librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/39 2026-02-25T08:07:48.267 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/39 2026-02-25T08:07:48.268 INFO:teuthology.orchestra.run.trial030.stdout: Installing : libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/39 2026-02-25T08:07:48.280 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/39 2026-02-25T08:07:48.283 INFO:teuthology.orchestra.run.trial030.stdout: Installing : libcephfs-daemon-2:20.3.0-5338.g6aeb6975.el9.x86_6 4/39 2026-02-25T08:07:48.283 INFO:teuthology.orchestra.run.trial030.stdout: Installing : libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_6 5/39 2026-02-25T08:07:48.309 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_6 5/39 2026-02-25T08:07:48.311 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 6/39 2026-02-25T08:07:48.319 INFO:teuthology.orchestra.run.trial030.stdout: Installing : liboath-2.6.12-1.el9.x86_64 7/39 2026-02-25T08:07:48.321 INFO:teuthology.orchestra.run.trial030.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/39 2026-02-25T08:07:48.325 INFO:teuthology.orchestra.run.trial030.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/39 2026-02-25T08:07:48.329 INFO:teuthology.orchestra.run.trial030.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/39 2026-02-25T08:07:48.341 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-ceph-argparse-2:20.3.0-5338.g6aeb6975.el9. 11/39 2026-02-25T08:07:48.354 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/39 2026-02-25T08:07:48.355 INFO:teuthology.orchestra.run.trial030.stdout: Installing : libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_6 13/39 2026-02-25T08:07:48.373 INFO:teuthology.orchestra.run.trial023.stdout: Preparing : 1/1 2026-02-25T08:07:48.378 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_6 13/39 2026-02-25T08:07:48.382 INFO:teuthology.orchestra.run.trial030.stdout: Installing : thrift-0.15.0-4.el9.x86_64 14/39 2026-02-25T08:07:48.415 INFO:teuthology.orchestra.run.trial030.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/39 2026-02-25T08:07:48.420 INFO:teuthology.orchestra.run.trial030.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/39 2026-02-25T08:07:48.426 INFO:teuthology.orchestra.run.trial030.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/39 2026-02-25T08:07:48.428 INFO:teuthology.orchestra.run.trial030.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/39 2026-02-25T08:07:48.433 INFO:teuthology.orchestra.run.trial030.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 19/39 2026-02-25T08:07:48.514 INFO:teuthology.orchestra.run.trial023.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/39 2026-02-25T08:07:48.518 INFO:teuthology.orchestra.run.trial023.stdout: Upgrading : librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/39 2026-02-25T08:07:48.549 INFO:teuthology.orchestra.run.trial030.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-25T08:07:48.550 INFO:teuthology.orchestra.run.trial030.stdout: Upgrading : librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 21/39 2026-02-25T08:07:48.567 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/39 2026-02-25T08:07:48.568 INFO:teuthology.orchestra.run.trial023.stdout: Installing : libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/39 2026-02-25T08:07:48.578 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 21/39 2026-02-25T08:07:48.581 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/39 2026-02-25T08:07:48.581 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 22/39 2026-02-25T08:07:48.583 INFO:teuthology.orchestra.run.trial023.stdout: Installing : libcephfs-daemon-2:20.3.0-5338.g6aeb6975.el9.x86_6 4/39 2026-02-25T08:07:48.583 INFO:teuthology.orchestra.run.trial023.stdout: Installing : libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_6 5/39 2026-02-25T08:07:48.590 INFO:teuthology.orchestra.run.trial030.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 23/39 2026-02-25T08:07:48.611 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_6 5/39 2026-02-25T08:07:48.614 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 6/39 2026-02-25T08:07:48.623 INFO:teuthology.orchestra.run.trial023.stdout: Installing : liboath-2.6.12-1.el9.x86_64 7/39 2026-02-25T08:07:48.625 INFO:teuthology.orchestra.run.trial023.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/39 2026-02-25T08:07:48.628 INFO:teuthology.orchestra.run.trial023.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/39 2026-02-25T08:07:48.632 INFO:teuthology.orchestra.run.trial023.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/39 2026-02-25T08:07:48.643 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-ceph-argparse-2:20.3.0-5338.g6aeb6975.el9. 11/39 2026-02-25T08:07:48.656 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/39 2026-02-25T08:07:48.656 INFO:teuthology.orchestra.run.trial023.stdout: Installing : libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_6 13/39 2026-02-25T08:07:48.677 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_6 13/39 2026-02-25T08:07:48.681 INFO:teuthology.orchestra.run.trial023.stdout: Installing : thrift-0.15.0-4.el9.x86_64 14/39 2026-02-25T08:07:48.711 INFO:teuthology.orchestra.run.trial023.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/39 2026-02-25T08:07:48.716 INFO:teuthology.orchestra.run.trial023.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/39 2026-02-25T08:07:48.722 INFO:teuthology.orchestra.run.trial023.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/39 2026-02-25T08:07:48.724 INFO:teuthology.orchestra.run.trial023.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/39 2026-02-25T08:07:48.728 INFO:teuthology.orchestra.run.trial023.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 19/39 2026-02-25T08:07:48.841 INFO:teuthology.orchestra.run.trial030.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 24/39 2026-02-25T08:07:48.842 INFO:teuthology.orchestra.run.trial023.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-25T08:07:48.843 INFO:teuthology.orchestra.run.trial030.stdout: Installing : librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 25/39 2026-02-25T08:07:48.843 INFO:teuthology.orchestra.run.trial023.stdout: Upgrading : librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 21/39 2026-02-25T08:07:48.859 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 25/39 2026-02-25T08:07:48.864 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 26/39 2026-02-25T08:07:48.869 INFO:teuthology.orchestra.run.trial030.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 27/39 2026-02-25T08:07:48.871 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 21/39 2026-02-25T08:07:48.871 INFO:teuthology.orchestra.run.trial030.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 28/39 2026-02-25T08:07:48.872 INFO:teuthology.orchestra.run.trial030.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-25T08:07:48.875 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 22/39 2026-02-25T08:07:48.881 INFO:teuthology.orchestra.run.trial023.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 23/39 2026-02-25T08:07:48.890 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-25T08:07:49.004 INFO:teuthology.orchestra.run.trial030.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 30/39 2026-02-25T08:07:49.006 INFO:teuthology.orchestra.run.trial030.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-25T08:07:49.128 INFO:teuthology.orchestra.run.trial023.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 24/39 2026-02-25T08:07:49.129 INFO:teuthology.orchestra.run.trial023.stdout: Installing : librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 25/39 2026-02-25T08:07:49.146 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 25/39 2026-02-25T08:07:49.149 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 26/39 2026-02-25T08:07:49.154 INFO:teuthology.orchestra.run.trial023.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 27/39 2026-02-25T08:07:49.157 INFO:teuthology.orchestra.run.trial023.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 28/39 2026-02-25T08:07:49.158 INFO:teuthology.orchestra.run.trial023.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-25T08:07:49.176 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-25T08:07:49.288 INFO:teuthology.orchestra.run.trial023.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 30/39 2026-02-25T08:07:49.290 INFO:teuthology.orchestra.run.trial023.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-25T08:07:49.297 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-25T08:07:49.297 INFO:teuthology.orchestra.run.trial030.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-25T08:07:49.297 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:07:49.340 INFO:teuthology.orchestra.run.trial030.stdout: Installing : mailcap-2.1.49-5.el9.noarch 32/39 2026-02-25T08:07:49.342 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-ceph-common-2:20.3.0-5338.g6aeb6975.el9.x8 33/39 2026-02-25T08:07:49.572 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-25T08:07:49.573 INFO:teuthology.orchestra.run.trial023.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-25T08:07:49.573 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:07:49.615 INFO:teuthology.orchestra.run.trial023.stdout: Installing : mailcap-2.1.49-5.el9.noarch 32/39 2026-02-25T08:07:49.617 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-ceph-common-2:20.3.0-5338.g6aeb6975.el9.x8 33/39 2026-02-25T08:07:50.297 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 34/39 2026-02-25T08:07:50.299 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 34/39 2026-02-25T08:07:50.564 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 34/39 2026-02-25T08:07:50.564 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 34/39 2026-02-25T08:07:50.566 INFO:teuthology.orchestra.run.trial023.stdout: Installing : ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 34/39 2026-02-25T08:07:50.567 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 35/39 2026-02-25T08:07:50.607 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 35/39 2026-02-25T08:07:50.607 INFO:teuthology.orchestra.run.trial030.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-25T08:07:50.607 INFO:teuthology.orchestra.run.trial030.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-25T08:07:50.607 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:07:50.610 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 36/39 2026-02-25T08:07:50.829 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 34/39 2026-02-25T08:07:50.833 INFO:teuthology.orchestra.run.trial023.stdout: Installing : ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 35/39 2026-02-25T08:07:50.871 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 35/39 2026-02-25T08:07:50.871 INFO:teuthology.orchestra.run.trial023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-25T08:07:50.871 INFO:teuthology.orchestra.run.trial023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-25T08:07:50.871 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:07:50.875 INFO:teuthology.orchestra.run.trial023.stdout: Installing : ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 36/39 2026-02-25T08:07:55.105 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 36/39 2026-02-25T08:07:55.105 INFO:teuthology.orchestra.run.trial030.stdout:skipping the directory /sys 2026-02-25T08:07:55.105 INFO:teuthology.orchestra.run.trial030.stdout:skipping the directory /proc 2026-02-25T08:07:55.105 INFO:teuthology.orchestra.run.trial030.stdout:skipping the directory /mnt 2026-02-25T08:07:55.105 INFO:teuthology.orchestra.run.trial030.stdout:skipping the directory /var/tmp 2026-02-25T08:07:55.105 INFO:teuthology.orchestra.run.trial030.stdout:skipping the directory /home 2026-02-25T08:07:55.105 INFO:teuthology.orchestra.run.trial030.stdout:skipping the directory /root 2026-02-25T08:07:55.106 INFO:teuthology.orchestra.run.trial030.stdout:skipping the directory /tmp 2026-02-25T08:07:55.106 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:07:55.363 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 36/39 2026-02-25T08:07:55.363 INFO:teuthology.orchestra.run.trial023.stdout:skipping the directory /sys 2026-02-25T08:07:55.363 INFO:teuthology.orchestra.run.trial023.stdout:skipping the directory /proc 2026-02-25T08:07:55.363 INFO:teuthology.orchestra.run.trial023.stdout:skipping the directory /mnt 2026-02-25T08:07:55.363 INFO:teuthology.orchestra.run.trial023.stdout:skipping the directory /var/tmp 2026-02-25T08:07:55.363 INFO:teuthology.orchestra.run.trial023.stdout:skipping the directory /home 2026-02-25T08:07:55.363 INFO:teuthology.orchestra.run.trial023.stdout:skipping the directory /root 2026-02-25T08:07:55.363 INFO:teuthology.orchestra.run.trial023.stdout:skipping the directory /tmp 2026-02-25T08:07:55.363 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:07:55.604 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 37/39 2026-02-25T08:07:55.628 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 37/39 2026-02-25T08:07:55.628 INFO:teuthology.orchestra.run.trial030.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:07:55.628 INFO:teuthology.orchestra.run.trial030.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-25T08:07:55.628 INFO:teuthology.orchestra.run.trial030.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-25T08:07:55.628 INFO:teuthology.orchestra.run.trial030.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-25T08:07:55.628 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:07:55.629 INFO:teuthology.orchestra.run.trial030.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-25T08:07:55.641 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-25T08:07:55.641 INFO:teuthology.orchestra.run.trial030.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-25T08:07:55.695 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-25T08:07:55.695 INFO:teuthology.orchestra.run.trial030.stdout:Creating group 'qat' with GID 992. 2026-02-25T08:07:55.695 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:07:55.855 INFO:teuthology.orchestra.run.trial023.stdout: Installing : ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 37/39 2026-02-25T08:07:55.879 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 37/39 2026-02-25T08:07:55.879 INFO:teuthology.orchestra.run.trial023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:07:55.879 INFO:teuthology.orchestra.run.trial023.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-25T08:07:55.879 INFO:teuthology.orchestra.run.trial023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-25T08:07:55.879 INFO:teuthology.orchestra.run.trial023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-25T08:07:55.879 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:07:55.881 INFO:teuthology.orchestra.run.trial023.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-25T08:07:55.893 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-25T08:07:55.893 INFO:teuthology.orchestra.run.trial023.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-25T08:07:55.947 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-25T08:07:55.947 INFO:teuthology.orchestra.run.trial023.stdout:Creating group 'qat' with GID 992. 2026-02-25T08:07:55.947 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:07:55.973 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/39 2026-02-25T08:07:55.973 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/39 2026-02-25T08:07:55.973 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/39 2026-02-25T08:07:55.973 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 4/39 2026-02-25T08:07:55.973 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : libcephfs-daemon-2:20.3.0-5338.g6aeb6975.el9.x86_6 5/39 2026-02-25T08:07:55.973 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_6 6/39 2026-02-25T08:07:55.973 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 7/39 2026-02-25T08:07:55.973 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_6 8/39 2026-02-25T08:07:55.974 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 9/39 2026-02-25T08:07:55.974 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5338.g6aeb6975.el9. 10/39 2026-02-25T08:07:55.974 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-ceph-common-2:20.3.0-5338.g6aeb6975.el9.x8 11/39 2026-02-25T08:07:55.974 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/39 2026-02-25T08:07:55.974 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 13/39 2026-02-25T08:07:55.974 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 14/39 2026-02-25T08:07:55.974 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 15/39 2026-02-25T08:07:55.974 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 16/39 2026-02-25T08:07:55.974 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 17/39 2026-02-25T08:07:55.974 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 18/39 2026-02-25T08:07:55.974 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 19/39 2026-02-25T08:07:55.974 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-25T08:07:55.974 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 21/39 2026-02-25T08:07:55.974 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 22/39 2026-02-25T08:07:55.974 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 23/39 2026-02-25T08:07:55.974 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/39 2026-02-25T08:07:55.975 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 25/39 2026-02-25T08:07:55.975 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 26/39 2026-02-25T08:07:55.975 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 27/39 2026-02-25T08:07:55.975 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 28/39 2026-02-25T08:07:55.975 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 29/39 2026-02-25T08:07:55.975 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 30/39 2026-02-25T08:07:55.975 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 31/39 2026-02-25T08:07:55.975 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 32/39 2026-02-25T08:07:55.975 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 33/39 2026-02-25T08:07:55.975 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : re2-1:20211101-20.el9.x86_64 34/39 2026-02-25T08:07:55.975 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 35/39 2026-02-25T08:07:55.975 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 36/39 2026-02-25T08:07:55.975 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 37/39 2026-02-25T08:07:55.975 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 38/39 2026-02-25T08:07:56.040 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 39/39 2026-02-25T08:07:56.040 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:07:56.040 INFO:teuthology.orchestra.run.trial030.stdout:Upgraded: 2026-02-25T08:07:56.041 INFO:teuthology.orchestra.run.trial030.stdout: librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:07:56.041 INFO:teuthology.orchestra.run.trial030.stdout: librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:07:56.041 INFO:teuthology.orchestra.run.trial030.stdout:Installed: 2026-02-25T08:07:56.041 INFO:teuthology.orchestra.run.trial030.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-25T08:07:56.041 INFO:teuthology.orchestra.run.trial030.stdout: ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:07:56.041 INFO:teuthology.orchestra.run.trial030.stdout: ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:07:56.041 INFO:teuthology.orchestra.run.trial030.stdout: ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:07:56.041 INFO:teuthology.orchestra.run.trial030.stdout: ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:07:56.041 INFO:teuthology.orchestra.run.trial030.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-25T08:07:56.041 INFO:teuthology.orchestra.run.trial030.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-25T08:07:56.041 INFO:teuthology.orchestra.run.trial030.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-25T08:07:56.041 INFO:teuthology.orchestra.run.trial030.stdout: libcephfs-daemon-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:07:56.041 INFO:teuthology.orchestra.run.trial030.stdout: libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:07:56.041 INFO:teuthology.orchestra.run.trial030.stdout: libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:07:56.041 INFO:teuthology.orchestra.run.trial030.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-25T08:07:56.041 INFO:teuthology.orchestra.run.trial030.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-25T08:07:56.041 INFO:teuthology.orchestra.run.trial030.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-25T08:07:56.041 INFO:teuthology.orchestra.run.trial030.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-25T08:07:56.041 INFO:teuthology.orchestra.run.trial030.stdout: libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:07:56.041 INFO:teuthology.orchestra.run.trial030.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-25T08:07:56.041 INFO:teuthology.orchestra.run.trial030.stdout: librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:07:56.041 INFO:teuthology.orchestra.run.trial030.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-25T08:07:56.041 INFO:teuthology.orchestra.run.trial030.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-25T08:07:56.041 INFO:teuthology.orchestra.run.trial030.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-25T08:07:56.041 INFO:teuthology.orchestra.run.trial030.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-25T08:07:56.042 INFO:teuthology.orchestra.run.trial030.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-25T08:07:56.042 INFO:teuthology.orchestra.run.trial030.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-25T08:07:56.042 INFO:teuthology.orchestra.run.trial030.stdout: python3-ceph-argparse-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:07:56.042 INFO:teuthology.orchestra.run.trial030.stdout: python3-ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:07:56.042 INFO:teuthology.orchestra.run.trial030.stdout: python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:07:56.042 INFO:teuthology.orchestra.run.trial030.stdout: python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:07:56.042 INFO:teuthology.orchestra.run.trial030.stdout: python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:07:56.042 INFO:teuthology.orchestra.run.trial030.stdout: python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:07:56.042 INFO:teuthology.orchestra.run.trial030.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-25T08:07:56.042 INFO:teuthology.orchestra.run.trial030.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-25T08:07:56.042 INFO:teuthology.orchestra.run.trial030.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-25T08:07:56.042 INFO:teuthology.orchestra.run.trial030.stdout: re2-1:20211101-20.el9.x86_64 2026-02-25T08:07:56.042 INFO:teuthology.orchestra.run.trial030.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-25T08:07:56.042 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:07:56.042 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:07:56.082 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install ceph-test 2026-02-25T08:07:56.233 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/39 2026-02-25T08:07:56.234 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/39 2026-02-25T08:07:56.234 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/39 2026-02-25T08:07:56.234 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 4/39 2026-02-25T08:07:56.234 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : libcephfs-daemon-2:20.3.0-5338.g6aeb6975.el9.x86_6 5/39 2026-02-25T08:07:56.234 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_6 6/39 2026-02-25T08:07:56.234 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 7/39 2026-02-25T08:07:56.234 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_6 8/39 2026-02-25T08:07:56.234 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 9/39 2026-02-25T08:07:56.234 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5338.g6aeb6975.el9. 10/39 2026-02-25T08:07:56.234 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-ceph-common-2:20.3.0-5338.g6aeb6975.el9.x8 11/39 2026-02-25T08:07:56.234 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/39 2026-02-25T08:07:56.234 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 13/39 2026-02-25T08:07:56.234 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 14/39 2026-02-25T08:07:56.234 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 15/39 2026-02-25T08:07:56.234 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 16/39 2026-02-25T08:07:56.234 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 17/39 2026-02-25T08:07:56.234 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 18/39 2026-02-25T08:07:56.234 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 19/39 2026-02-25T08:07:56.234 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-25T08:07:56.234 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 21/39 2026-02-25T08:07:56.234 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 22/39 2026-02-25T08:07:56.234 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 23/39 2026-02-25T08:07:56.234 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/39 2026-02-25T08:07:56.234 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 25/39 2026-02-25T08:07:56.234 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 26/39 2026-02-25T08:07:56.234 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 27/39 2026-02-25T08:07:56.234 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 28/39 2026-02-25T08:07:56.234 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 29/39 2026-02-25T08:07:56.234 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 30/39 2026-02-25T08:07:56.234 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 31/39 2026-02-25T08:07:56.234 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 32/39 2026-02-25T08:07:56.234 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 33/39 2026-02-25T08:07:56.234 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : re2-1:20211101-20.el9.x86_64 34/39 2026-02-25T08:07:56.234 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 35/39 2026-02-25T08:07:56.234 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 36/39 2026-02-25T08:07:56.234 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 37/39 2026-02-25T08:07:56.234 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 38/39 2026-02-25T08:07:56.301 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 39/39 2026-02-25T08:07:56.302 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:07:56.302 INFO:teuthology.orchestra.run.trial023.stdout:Upgraded: 2026-02-25T08:07:56.302 INFO:teuthology.orchestra.run.trial023.stdout: librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:07:56.302 INFO:teuthology.orchestra.run.trial023.stdout: librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:07:56.302 INFO:teuthology.orchestra.run.trial023.stdout:Installed: 2026-02-25T08:07:56.302 INFO:teuthology.orchestra.run.trial023.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-25T08:07:56.302 INFO:teuthology.orchestra.run.trial023.stdout: ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:07:56.302 INFO:teuthology.orchestra.run.trial023.stdout: ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:07:56.302 INFO:teuthology.orchestra.run.trial023.stdout: ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:07:56.302 INFO:teuthology.orchestra.run.trial023.stdout: ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:07:56.302 INFO:teuthology.orchestra.run.trial023.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-25T08:07:56.302 INFO:teuthology.orchestra.run.trial023.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-25T08:07:56.302 INFO:teuthology.orchestra.run.trial023.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-25T08:07:56.302 INFO:teuthology.orchestra.run.trial023.stdout: libcephfs-daemon-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:07:56.302 INFO:teuthology.orchestra.run.trial023.stdout: libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:07:56.302 INFO:teuthology.orchestra.run.trial023.stdout: libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:07:56.302 INFO:teuthology.orchestra.run.trial023.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-25T08:07:56.302 INFO:teuthology.orchestra.run.trial023.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-25T08:07:56.302 INFO:teuthology.orchestra.run.trial023.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-25T08:07:56.302 INFO:teuthology.orchestra.run.trial023.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-25T08:07:56.302 INFO:teuthology.orchestra.run.trial023.stdout: libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:07:56.303 INFO:teuthology.orchestra.run.trial023.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-25T08:07:56.303 INFO:teuthology.orchestra.run.trial023.stdout: librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:07:56.303 INFO:teuthology.orchestra.run.trial023.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-25T08:07:56.303 INFO:teuthology.orchestra.run.trial023.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-25T08:07:56.303 INFO:teuthology.orchestra.run.trial023.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-25T08:07:56.303 INFO:teuthology.orchestra.run.trial023.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-25T08:07:56.303 INFO:teuthology.orchestra.run.trial023.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-25T08:07:56.303 INFO:teuthology.orchestra.run.trial023.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-25T08:07:56.303 INFO:teuthology.orchestra.run.trial023.stdout: python3-ceph-argparse-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:07:56.303 INFO:teuthology.orchestra.run.trial023.stdout: python3-ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:07:56.303 INFO:teuthology.orchestra.run.trial023.stdout: python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:07:56.303 INFO:teuthology.orchestra.run.trial023.stdout: python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:07:56.303 INFO:teuthology.orchestra.run.trial023.stdout: python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:07:56.303 INFO:teuthology.orchestra.run.trial023.stdout: python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:07:56.303 INFO:teuthology.orchestra.run.trial023.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-25T08:07:56.303 INFO:teuthology.orchestra.run.trial023.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-25T08:07:56.303 INFO:teuthology.orchestra.run.trial023.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-25T08:07:56.303 INFO:teuthology.orchestra.run.trial023.stdout: re2-1:20211101-20.el9.x86_64 2026-02-25T08:07:56.303 INFO:teuthology.orchestra.run.trial023.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-25T08:07:56.303 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:07:56.303 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:07:56.343 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install ceph-test 2026-02-25T08:07:56.394 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:00:30 ago on Wed 25 Feb 2026 08:07:26 AM UTC. 2026-02-25T08:07:56.478 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:07:56.479 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-25T08:07:56.479 INFO:teuthology.orchestra.run.trial030.stdout: Package Arch Version Repository Size 2026-02-25T08:07:56.479 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-25T08:07:56.479 INFO:teuthology.orchestra.run.trial030.stdout:Installing: 2026-02-25T08:07:56.479 INFO:teuthology.orchestra.run.trial030.stdout: ceph-test x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 127 M 2026-02-25T08:07:56.479 INFO:teuthology.orchestra.run.trial030.stdout:Installing dependencies: 2026-02-25T08:07:56.479 INFO:teuthology.orchestra.run.trial030.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-25T08:07:56.479 INFO:teuthology.orchestra.run.trial030.stdout: libasan8 x86_64 15.2.1-7.el9 appstream 525 k 2026-02-25T08:07:56.479 INFO:teuthology.orchestra.run.trial030.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-25T08:07:56.479 INFO:teuthology.orchestra.run.trial030.stdout: libcephsqlite x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 153 k 2026-02-25T08:07:56.479 INFO:teuthology.orchestra.run.trial030.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-25T08:07:56.479 INFO:teuthology.orchestra.run.trial030.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-25T08:07:56.479 INFO:teuthology.orchestra.run.trial030.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-25T08:07:56.479 INFO:teuthology.orchestra.run.trial030.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-25T08:07:56.479 INFO:teuthology.orchestra.run.trial030.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-25T08:07:56.479 INFO:teuthology.orchestra.run.trial030.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-25T08:07:56.479 INFO:teuthology.orchestra.run.trial030.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-25T08:07:56.479 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:07:56.479 INFO:teuthology.orchestra.run.trial030.stdout:Transaction Summary 2026-02-25T08:07:56.479 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-25T08:07:56.479 INFO:teuthology.orchestra.run.trial030.stdout:Install 12 Packages 2026-02-25T08:07:56.479 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:07:56.479 INFO:teuthology.orchestra.run.trial030.stdout:Total download size: 132 M 2026-02-25T08:07:56.479 INFO:teuthology.orchestra.run.trial030.stdout:Installed size: 988 M 2026-02-25T08:07:56.479 INFO:teuthology.orchestra.run.trial030.stdout:Downloading Packages: 2026-02-25T08:07:56.660 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:00:29 ago on Wed 25 Feb 2026 08:07:27 AM UTC. 2026-02-25T08:07:56.744 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:07:56.744 INFO:teuthology.orchestra.run.trial023.stdout:================================================================================ 2026-02-25T08:07:56.745 INFO:teuthology.orchestra.run.trial023.stdout: Package Arch Version Repository Size 2026-02-25T08:07:56.745 INFO:teuthology.orchestra.run.trial023.stdout:================================================================================ 2026-02-25T08:07:56.745 INFO:teuthology.orchestra.run.trial023.stdout:Installing: 2026-02-25T08:07:56.745 INFO:teuthology.orchestra.run.trial023.stdout: ceph-test x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 127 M 2026-02-25T08:07:56.745 INFO:teuthology.orchestra.run.trial023.stdout:Installing dependencies: 2026-02-25T08:07:56.745 INFO:teuthology.orchestra.run.trial023.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-25T08:07:56.745 INFO:teuthology.orchestra.run.trial023.stdout: libasan8 x86_64 15.2.1-7.el9 appstream 525 k 2026-02-25T08:07:56.745 INFO:teuthology.orchestra.run.trial023.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-25T08:07:56.745 INFO:teuthology.orchestra.run.trial023.stdout: libcephsqlite x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 153 k 2026-02-25T08:07:56.745 INFO:teuthology.orchestra.run.trial023.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-25T08:07:56.745 INFO:teuthology.orchestra.run.trial023.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-25T08:07:56.745 INFO:teuthology.orchestra.run.trial023.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-25T08:07:56.745 INFO:teuthology.orchestra.run.trial023.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-25T08:07:56.745 INFO:teuthology.orchestra.run.trial023.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-25T08:07:56.745 INFO:teuthology.orchestra.run.trial023.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-25T08:07:56.745 INFO:teuthology.orchestra.run.trial023.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-25T08:07:56.745 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:07:56.745 INFO:teuthology.orchestra.run.trial023.stdout:Transaction Summary 2026-02-25T08:07:56.745 INFO:teuthology.orchestra.run.trial023.stdout:================================================================================ 2026-02-25T08:07:56.745 INFO:teuthology.orchestra.run.trial023.stdout:Install 12 Packages 2026-02-25T08:07:56.746 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:07:56.746 INFO:teuthology.orchestra.run.trial023.stdout:Total download size: 132 M 2026-02-25T08:07:56.746 INFO:teuthology.orchestra.run.trial023.stdout:Installed size: 988 M 2026-02-25T08:07:56.746 INFO:teuthology.orchestra.run.trial023.stdout:Downloading Packages: 2026-02-25T08:07:56.809 INFO:teuthology.orchestra.run.trial030.stdout:(1/12): hwloc-libs-2.4.1-6.el9.x86_64.rpm 27 MB/s | 2.1 MB 00:00 2026-02-25T08:07:56.818 INFO:teuthology.orchestra.run.trial030.stdout:(2/12): libatomic-11.5.0-14.el9.x86_64.rpm 2.5 MB/s | 23 kB 00:00 2026-02-25T08:07:56.828 INFO:teuthology.orchestra.run.trial030.stdout:(3/12): lksctp-tools-1.0.19-2.el9.x86_64.rpm 9.5 MB/s | 94 kB 00:00 2026-02-25T08:07:57.003 INFO:teuthology.orchestra.run.trial023.stdout:(1/12): hwloc-libs-2.4.1-6.el9.x86_64.rpm 64 MB/s | 2.1 MB 00:00 2026-02-25T08:07:57.007 INFO:teuthology.orchestra.run.trial023.stdout:(2/12): libatomic-11.5.0-14.el9.x86_64.rpm 5.2 MB/s | 23 kB 00:00 2026-02-25T08:07:57.012 INFO:teuthology.orchestra.run.trial023.stdout:(3/12): lksctp-tools-1.0.19-2.el9.x86_64.rpm 18 MB/s | 94 kB 00:00 2026-02-25T08:07:57.271 INFO:teuthology.orchestra.run.trial030.stdout:(4/12): libcephsqlite-20.3.0-5338.g6aeb6975.el9 283 kB/s | 153 kB 00:00 2026-02-25T08:07:57.407 INFO:teuthology.orchestra.run.trial030.stdout:(5/12): libasan8-15.2.1-7.el9.x86_64.rpm 907 kB/s | 525 kB 00:00 2026-02-25T08:07:57.507 INFO:teuthology.orchestra.run.trial030.stdout:(6/12): libxslt-1.1.34-12.el9.x86_64.rpm 2.3 MB/s | 233 kB 00:00 2026-02-25T08:07:57.510 INFO:teuthology.orchestra.run.trial023.stdout:[MIRROR] libasan8-15.2.1-7.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libasan8-15.2.1-7.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-25T08:07:57.691 INFO:teuthology.orchestra.run.trial030.stdout:(7/12): protobuf-3.14.0-17.el9.x86_64.rpm 5.5 MB/s | 1.0 MB 00:00 2026-02-25T08:07:57.728 INFO:teuthology.orchestra.run.trial030.stdout:(8/12): libubsan-11.5.0-14.el9.x86_64.rpm 386 kB/s | 176 kB 00:00 2026-02-25T08:07:57.759 INFO:teuthology.orchestra.run.trial023.stdout:(4/12): libcephsqlite-20.3.0-5338.g6aeb6975.el9 194 kB/s | 153 kB 00:00 2026-02-25T08:07:57.779 INFO:teuthology.orchestra.run.trial030.stdout:(9/12): socat-1.7.4.1-8.el9.x86_64.rpm 3.4 MB/s | 303 kB 00:00 2026-02-25T08:07:57.824 INFO:teuthology.orchestra.run.trial030.stdout:(10/12): xmlstarlet-1.6.1-20.el9.x86_64.rpm 662 kB/s | 64 kB 00:00 2026-02-25T08:07:58.071 INFO:teuthology.orchestra.run.trial030.stdout:(11/12): yaml-cpp-0.6.3-5.el9.x86_64.rpm 416 kB/s | 122 kB 00:00 2026-02-25T08:07:58.176 INFO:teuthology.orchestra.run.trial023.stdout:[MIRROR] libasan8-15.2.1-7.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libasan8-15.2.1-7.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-25T08:07:58.176 INFO:teuthology.orchestra.run.trial023.stdout:[MIRROR] libubsan-11.5.0-14.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libubsan-11.5.0-14.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-25T08:07:58.551 INFO:teuthology.orchestra.run.trial023.stdout:[MIRROR] libasan8-15.2.1-7.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libasan8-15.2.1-7.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-25T08:07:58.588 INFO:teuthology.orchestra.run.trial023.stdout:[MIRROR] libubsan-11.5.0-14.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libubsan-11.5.0-14.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-25T08:07:58.943 INFO:teuthology.orchestra.run.trial023.stdout:[MIRROR] libubsan-11.5.0-14.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libubsan-11.5.0-14.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-25T08:07:58.973 INFO:teuthology.orchestra.run.trial023.stdout:[MIRROR] libasan8-15.2.1-7.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libasan8-15.2.1-7.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-25T08:07:59.158 INFO:teuthology.orchestra.run.trial023.stdout:(5/12): libasan8-15.2.1-7.el9.x86_64.rpm 245 kB/s | 525 kB 00:02 2026-02-25T08:07:59.295 INFO:teuthology.orchestra.run.trial023.stdout:[MIRROR] libubsan-11.5.0-14.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libubsan-11.5.0-14.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-25T08:07:59.325 INFO:teuthology.orchestra.run.trial023.stdout:(6/12): libubsan-11.5.0-14.el9.x86_64.rpm 112 kB/s | 176 kB 00:01 2026-02-25T08:07:59.574 INFO:teuthology.orchestra.run.trial023.stdout:[MIRROR] libxslt-1.1.34-12.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libxslt-1.1.34-12.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-25T08:07:59.748 INFO:teuthology.orchestra.run.trial023.stdout:[MIRROR] protobuf-3.14.0-17.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/protobuf-3.14.0-17.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-25T08:07:59.985 INFO:teuthology.orchestra.run.trial023.stdout:[MIRROR] libxslt-1.1.34-12.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libxslt-1.1.34-12.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-25T08:08:00.164 INFO:teuthology.orchestra.run.trial023.stdout:[MIRROR] protobuf-3.14.0-17.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/protobuf-3.14.0-17.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-25T08:08:00.335 INFO:teuthology.orchestra.run.trial023.stdout:[MIRROR] libxslt-1.1.34-12.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libxslt-1.1.34-12.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-25T08:08:00.522 INFO:teuthology.orchestra.run.trial023.stdout:[MIRROR] protobuf-3.14.0-17.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/protobuf-3.14.0-17.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-25T08:08:00.691 INFO:teuthology.orchestra.run.trial023.stdout:[MIRROR] libxslt-1.1.34-12.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libxslt-1.1.34-12.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-25T08:08:00.723 INFO:teuthology.orchestra.run.trial023.stdout:(7/12): libxslt-1.1.34-12.el9.x86_64.rpm 149 kB/s | 233 kB 00:01 2026-02-25T08:08:00.757 INFO:teuthology.orchestra.run.trial023.stdout:(8/12): socat-1.7.4.1-8.el9.x86_64.rpm 8.7 MB/s | 303 kB 00:00 2026-02-25T08:08:00.783 INFO:teuthology.orchestra.run.trial023.stdout:(9/12): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.4 MB/s | 64 kB 00:00 2026-02-25T08:08:00.877 INFO:teuthology.orchestra.run.trial023.stdout:[MIRROR] protobuf-3.14.0-17.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/protobuf-3.14.0-17.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-25T08:08:00.935 INFO:teuthology.orchestra.run.trial023.stdout:(10/12): protobuf-3.14.0-17.el9.x86_64.rpm 639 kB/s | 1.0 MB 00:01 2026-02-25T08:08:00.936 INFO:teuthology.orchestra.run.trial023.stdout:(11/12): yaml-cpp-0.6.3-5.el9.x86_64.rpm 795 kB/s | 122 kB 00:00 2026-02-25T08:13:28.563 INFO:teuthology.orchestra.run.trial023.stdout:(12/12): ceph-test-20.3.0-5338.g6aeb6975.el9.x8 392 kB/s | 127 MB 05:31 2026-02-25T08:13:28.564 INFO:teuthology.orchestra.run.trial023.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:13:28.565 INFO:teuthology.orchestra.run.trial023.stdout:Total 406 kB/s | 132 MB 05:31 2026-02-25T08:13:28.637 INFO:teuthology.orchestra.run.trial023.stdout:Running transaction check 2026-02-25T08:13:28.648 INFO:teuthology.orchestra.run.trial023.stdout:Transaction check succeeded. 2026-02-25T08:13:28.648 INFO:teuthology.orchestra.run.trial023.stdout:Running transaction test 2026-02-25T08:13:29.079 INFO:teuthology.orchestra.run.trial023.stdout:Transaction test succeeded. 2026-02-25T08:13:29.079 INFO:teuthology.orchestra.run.trial023.stdout:Running transaction 2026-02-25T08:13:29.538 INFO:teuthology.orchestra.run.trial023.stdout: Preparing : 1/1 2026-02-25T08:13:29.547 INFO:teuthology.orchestra.run.trial023.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/12 2026-02-25T08:13:29.556 INFO:teuthology.orchestra.run.trial023.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/12 2026-02-25T08:13:29.561 INFO:teuthology.orchestra.run.trial023.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/12 2026-02-25T08:13:29.566 INFO:teuthology.orchestra.run.trial023.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/12 2026-02-25T08:13:29.568 INFO:teuthology.orchestra.run.trial023.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/12 2026-02-25T08:13:29.573 INFO:teuthology.orchestra.run.trial023.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 6/12 2026-02-25T08:13:29.574 INFO:teuthology.orchestra.run.trial023.stdout: Installing : libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-25T08:13:29.589 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-25T08:13:29.593 INFO:teuthology.orchestra.run.trial023.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 8/12 2026-02-25T08:13:29.605 INFO:teuthology.orchestra.run.trial023.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 9/12 2026-02-25T08:13:29.613 INFO:teuthology.orchestra.run.trial023.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 10/12 2026-02-25T08:13:29.613 INFO:teuthology.orchestra.run.trial023.stdout: Installing : libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 11/12 2026-02-25T08:13:36.668 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 11/12 2026-02-25T08:13:36.687 INFO:teuthology.orchestra.run.trial023.stdout: Installing : ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/12 2026-02-25T08:13:37.499 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/12 2026-02-25T08:13:37.499 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/12 2026-02-25T08:13:37.499 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/12 2026-02-25T08:13:37.499 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/12 2026-02-25T08:13:37.499 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/12 2026-02-25T08:13:37.499 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/12 2026-02-25T08:13:37.499 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : libasan8-15.2.1-7.el9.x86_64 6/12 2026-02-25T08:13:37.499 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 7/12 2026-02-25T08:13:37.499 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 8/12 2026-02-25T08:13:37.499 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 9/12 2026-02-25T08:13:37.499 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 10/12 2026-02-25T08:13:37.499 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 11/12 2026-02-25T08:13:37.558 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 12/12 2026-02-25T08:13:37.558 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:13:37.558 INFO:teuthology.orchestra.run.trial023.stdout:Installed: 2026-02-25T08:13:37.558 INFO:teuthology.orchestra.run.trial023.stdout: ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:13:37.558 INFO:teuthology.orchestra.run.trial023.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-25T08:13:37.558 INFO:teuthology.orchestra.run.trial023.stdout: libasan8-15.2.1-7.el9.x86_64 2026-02-25T08:13:37.558 INFO:teuthology.orchestra.run.trial023.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-25T08:13:37.558 INFO:teuthology.orchestra.run.trial023.stdout: libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:13:37.558 INFO:teuthology.orchestra.run.trial023.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-25T08:13:37.558 INFO:teuthology.orchestra.run.trial023.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-25T08:13:37.558 INFO:teuthology.orchestra.run.trial023.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-25T08:13:37.558 INFO:teuthology.orchestra.run.trial023.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-25T08:13:37.558 INFO:teuthology.orchestra.run.trial023.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-25T08:13:37.558 INFO:teuthology.orchestra.run.trial023.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-25T08:13:37.558 INFO:teuthology.orchestra.run.trial023.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-25T08:13:37.558 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:13:37.558 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:13:37.599 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install ceph 2026-02-25T08:13:37.914 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:06:10 ago on Wed 25 Feb 2026 08:07:27 AM UTC. 2026-02-25T08:13:38.001 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:13:38.003 INFO:teuthology.orchestra.run.trial023.stdout:======================================================================================= 2026-02-25T08:13:38.003 INFO:teuthology.orchestra.run.trial023.stdout: Package Arch Version Repository Size 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout:======================================================================================= 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout:Installing: 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout: ceph x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 6.5 k 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout:Installing dependencies: 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 57 k 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout: ceph-mds x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 2.0 M 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout: ceph-mgr x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 937 k 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 297 k 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout: ceph-mon x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 4.9 M 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout: ceph-osd x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 163 k 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout: ceph-osd-classic x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 15 M 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout: ceph-osd-crimson x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 88 M 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 17 k 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout: cephadm noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 1.0 M 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-25T08:13:38.004 INFO:teuthology.orchestra.run.trial023.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-pyasn1 noarch 0.4.8-7.el9 appstream 157 k 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-pyasn1-modules noarch 0.4.8-7.el9 appstream 277 k 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-25T08:13:38.005 INFO:teuthology.orchestra.run.trial023.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-25T08:13:38.006 INFO:teuthology.orchestra.run.trial023.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-25T08:13:38.006 INFO:teuthology.orchestra.run.trial023.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-25T08:13:38.006 INFO:teuthology.orchestra.run.trial023.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-25T08:13:38.006 INFO:teuthology.orchestra.run.trial023.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-25T08:13:38.006 INFO:teuthology.orchestra.run.trial023.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-25T08:13:38.006 INFO:teuthology.orchestra.run.trial023.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-25T08:13:38.006 INFO:teuthology.orchestra.run.trial023.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-25T08:13:38.006 INFO:teuthology.orchestra.run.trial023.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-25T08:13:38.006 INFO:teuthology.orchestra.run.trial023.stdout:Installing weak dependencies: 2026-02-25T08:13:38.006 INFO:teuthology.orchestra.run.trial023.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 184 k 2026-02-25T08:13:38.006 INFO:teuthology.orchestra.run.trial023.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 51 M 2026-02-25T08:13:38.006 INFO:teuthology.orchestra.run.trial023.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 7.4 M 2026-02-25T08:13:38.006 INFO:teuthology.orchestra.run.trial023.stdout: ceph-mgr-k8sevents noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 22 k 2026-02-25T08:13:38.006 INFO:teuthology.orchestra.run.trial023.stdout: ceph-mgr-rook noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 50 k 2026-02-25T08:13:38.006 INFO:teuthology.orchestra.run.trial023.stdout: ceph-volume noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 301 k 2026-02-25T08:13:38.006 INFO:teuthology.orchestra.run.trial023.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-25T08:13:38.006 INFO:teuthology.orchestra.run.trial023.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-25T08:13:38.006 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:13:38.006 INFO:teuthology.orchestra.run.trial023.stdout:Transaction Summary 2026-02-25T08:13:38.006 INFO:teuthology.orchestra.run.trial023.stdout:======================================================================================= 2026-02-25T08:13:38.006 INFO:teuthology.orchestra.run.trial023.stdout:Install 96 Packages 2026-02-25T08:13:38.006 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:13:38.006 INFO:teuthology.orchestra.run.trial023.stdout:Total download size: 226 M 2026-02-25T08:13:38.006 INFO:teuthology.orchestra.run.trial023.stdout:Installed size: 1.5 G 2026-02-25T08:13:38.006 INFO:teuthology.orchestra.run.trial023.stdout:Downloading Packages: 2026-02-25T08:13:38.641 INFO:teuthology.orchestra.run.trial023.stdout:(1/96): ceph-20.3.0-5338.g6aeb6975.el9.x86_64.r 23 kB/s | 6.5 kB 00:00 2026-02-25T08:13:39.345 INFO:teuthology.orchestra.run.trial023.stdout:(2/96): ceph-mgr-20.3.0-5338.g6aeb6975.el9.x86_ 945 kB/s | 937 kB 00:00 2026-02-25T08:13:39.483 INFO:teuthology.orchestra.run.trial023.stdout:(3/96): ceph-osd-20.3.0-5338.g6aeb6975.el9.x86_ 1.2 MB/s | 163 kB 00:00 2026-02-25T08:13:40.042 INFO:teuthology.orchestra.run.trial023.stdout:(4/96): ceph-mds-20.3.0-5338.g6aeb6975.el9.x86_ 1.2 MB/s | 2.0 MB 00:01 2026-02-25T08:13:49.002 INFO:teuthology.orchestra.run.trial023.stdout:(5/96): ceph-mon-20.3.0-5338.g6aeb6975.el9.x86_ 481 kB/s | 4.9 MB 00:10 2026-02-25T08:13:49.211 INFO:teuthology.orchestra.run.trial023.stdout:(6/96): ceph-grafana-dashboards-20.3.0-5338.g6a 274 kB/s | 57 kB 00:00 2026-02-25T08:13:49.858 INFO:teuthology.orchestra.run.trial023.stdout:(7/96): ceph-mgr-cephadm-20.3.0-5338.g6aeb6975. 284 kB/s | 184 kB 00:00 2026-02-25T08:14:00.419 INFO:teuthology.orchestra.run.trial030.stdout:(12/12): ceph-test-20.3.0-5338.g6aeb6975.el9.x8 357 kB/s | 127 MB 06:03 2026-02-25T08:14:00.420 INFO:teuthology.orchestra.run.trial030.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:14:00.421 INFO:teuthology.orchestra.run.trial030.stdout:Total 370 kB/s | 132 MB 06:03 2026-02-25T08:14:00.491 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction check 2026-02-25T08:14:00.502 INFO:teuthology.orchestra.run.trial030.stdout:Transaction check succeeded. 2026-02-25T08:14:00.502 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction test 2026-02-25T08:14:00.937 INFO:teuthology.orchestra.run.trial030.stdout:Transaction test succeeded. 2026-02-25T08:14:00.938 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction 2026-02-25T08:14:01.396 INFO:teuthology.orchestra.run.trial030.stdout: Preparing : 1/1 2026-02-25T08:14:01.405 INFO:teuthology.orchestra.run.trial030.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/12 2026-02-25T08:14:01.414 INFO:teuthology.orchestra.run.trial030.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/12 2026-02-25T08:14:01.419 INFO:teuthology.orchestra.run.trial030.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/12 2026-02-25T08:14:01.423 INFO:teuthology.orchestra.run.trial030.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/12 2026-02-25T08:14:01.425 INFO:teuthology.orchestra.run.trial030.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/12 2026-02-25T08:14:01.430 INFO:teuthology.orchestra.run.trial030.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 6/12 2026-02-25T08:14:01.431 INFO:teuthology.orchestra.run.trial030.stdout: Installing : libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-25T08:14:01.447 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-25T08:14:01.450 INFO:teuthology.orchestra.run.trial030.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 8/12 2026-02-25T08:14:01.461 INFO:teuthology.orchestra.run.trial030.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 9/12 2026-02-25T08:14:01.469 INFO:teuthology.orchestra.run.trial030.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 10/12 2026-02-25T08:14:01.470 INFO:teuthology.orchestra.run.trial030.stdout: Installing : libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 11/12 2026-02-25T08:14:06.003 INFO:teuthology.orchestra.run.trial023.stdout:(8/96): ceph-osd-classic-20.3.0-5338.g6aeb6975. 570 kB/s | 15 MB 00:26 2026-02-25T08:14:08.416 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 11/12 2026-02-25T08:14:08.435 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/12 2026-02-25T08:14:09.216 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/12 2026-02-25T08:14:09.216 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/12 2026-02-25T08:14:09.217 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/12 2026-02-25T08:14:09.217 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/12 2026-02-25T08:14:09.217 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/12 2026-02-25T08:14:09.217 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/12 2026-02-25T08:14:09.217 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : libasan8-15.2.1-7.el9.x86_64 6/12 2026-02-25T08:14:09.217 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 7/12 2026-02-25T08:14:09.217 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 8/12 2026-02-25T08:14:09.217 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 9/12 2026-02-25T08:14:09.217 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 10/12 2026-02-25T08:14:09.217 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 11/12 2026-02-25T08:14:09.275 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 12/12 2026-02-25T08:14:09.275 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:14:09.275 INFO:teuthology.orchestra.run.trial030.stdout:Installed: 2026-02-25T08:14:09.275 INFO:teuthology.orchestra.run.trial030.stdout: ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:14:09.275 INFO:teuthology.orchestra.run.trial030.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-25T08:14:09.275 INFO:teuthology.orchestra.run.trial030.stdout: libasan8-15.2.1-7.el9.x86_64 2026-02-25T08:14:09.275 INFO:teuthology.orchestra.run.trial030.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-25T08:14:09.275 INFO:teuthology.orchestra.run.trial030.stdout: libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:14:09.275 INFO:teuthology.orchestra.run.trial030.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-25T08:14:09.275 INFO:teuthology.orchestra.run.trial030.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-25T08:14:09.276 INFO:teuthology.orchestra.run.trial030.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-25T08:14:09.276 INFO:teuthology.orchestra.run.trial030.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-25T08:14:09.276 INFO:teuthology.orchestra.run.trial030.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-25T08:14:09.276 INFO:teuthology.orchestra.run.trial030.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-25T08:14:09.276 INFO:teuthology.orchestra.run.trial030.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-25T08:14:09.276 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:14:09.276 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:14:09.317 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install ceph 2026-02-25T08:14:09.643 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:06:43 ago on Wed 25 Feb 2026 08:07:26 AM UTC. 2026-02-25T08:14:09.732 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:14:09.735 INFO:teuthology.orchestra.run.trial030.stdout:======================================================================================= 2026-02-25T08:14:09.735 INFO:teuthology.orchestra.run.trial030.stdout: Package Arch Version Repository Size 2026-02-25T08:14:09.735 INFO:teuthology.orchestra.run.trial030.stdout:======================================================================================= 2026-02-25T08:14:09.735 INFO:teuthology.orchestra.run.trial030.stdout:Installing: 2026-02-25T08:14:09.735 INFO:teuthology.orchestra.run.trial030.stdout: ceph x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 6.5 k 2026-02-25T08:14:09.735 INFO:teuthology.orchestra.run.trial030.stdout:Installing dependencies: 2026-02-25T08:14:09.735 INFO:teuthology.orchestra.run.trial030.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-25T08:14:09.735 INFO:teuthology.orchestra.run.trial030.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 57 k 2026-02-25T08:14:09.736 INFO:teuthology.orchestra.run.trial030.stdout: ceph-mds x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 2.0 M 2026-02-25T08:14:09.736 INFO:teuthology.orchestra.run.trial030.stdout: ceph-mgr x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 937 k 2026-02-25T08:14:09.736 INFO:teuthology.orchestra.run.trial030.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 297 k 2026-02-25T08:14:09.736 INFO:teuthology.orchestra.run.trial030.stdout: ceph-mon x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 4.9 M 2026-02-25T08:14:09.736 INFO:teuthology.orchestra.run.trial030.stdout: ceph-osd x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 163 k 2026-02-25T08:14:09.736 INFO:teuthology.orchestra.run.trial030.stdout: ceph-osd-classic x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 15 M 2026-02-25T08:14:09.736 INFO:teuthology.orchestra.run.trial030.stdout: ceph-osd-crimson x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 88 M 2026-02-25T08:14:09.736 INFO:teuthology.orchestra.run.trial030.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 17 k 2026-02-25T08:14:09.736 INFO:teuthology.orchestra.run.trial030.stdout: cephadm noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 1.0 M 2026-02-25T08:14:09.736 INFO:teuthology.orchestra.run.trial030.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-25T08:14:09.736 INFO:teuthology.orchestra.run.trial030.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-25T08:14:09.736 INFO:teuthology.orchestra.run.trial030.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-25T08:14:09.737 INFO:teuthology.orchestra.run.trial030.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-25T08:14:09.737 INFO:teuthology.orchestra.run.trial030.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-25T08:14:09.737 INFO:teuthology.orchestra.run.trial030.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-25T08:14:09.737 INFO:teuthology.orchestra.run.trial030.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-25T08:14:09.737 INFO:teuthology.orchestra.run.trial030.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-25T08:14:09.737 INFO:teuthology.orchestra.run.trial030.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-25T08:14:09.737 INFO:teuthology.orchestra.run.trial030.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-25T08:14:09.737 INFO:teuthology.orchestra.run.trial030.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-25T08:14:09.737 INFO:teuthology.orchestra.run.trial030.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-25T08:14:09.737 INFO:teuthology.orchestra.run.trial030.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-25T08:14:09.737 INFO:teuthology.orchestra.run.trial030.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-25T08:14:09.737 INFO:teuthology.orchestra.run.trial030.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-25T08:14:09.738 INFO:teuthology.orchestra.run.trial030.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-25T08:14:09.738 INFO:teuthology.orchestra.run.trial030.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-25T08:14:09.738 INFO:teuthology.orchestra.run.trial030.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-25T08:14:09.738 INFO:teuthology.orchestra.run.trial030.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-25T08:14:09.738 INFO:teuthology.orchestra.run.trial030.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-25T08:14:09.738 INFO:teuthology.orchestra.run.trial030.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-25T08:14:09.738 INFO:teuthology.orchestra.run.trial030.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-25T08:14:09.738 INFO:teuthology.orchestra.run.trial030.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-25T08:14:09.738 INFO:teuthology.orchestra.run.trial030.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-25T08:14:09.738 INFO:teuthology.orchestra.run.trial030.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-25T08:14:09.738 INFO:teuthology.orchestra.run.trial030.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-25T08:14:09.738 INFO:teuthology.orchestra.run.trial030.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-25T08:14:09.738 INFO:teuthology.orchestra.run.trial030.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-25T08:14:09.739 INFO:teuthology.orchestra.run.trial030.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-25T08:14:09.739 INFO:teuthology.orchestra.run.trial030.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-25T08:14:09.739 INFO:teuthology.orchestra.run.trial030.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-25T08:14:09.739 INFO:teuthology.orchestra.run.trial030.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-25T08:14:09.739 INFO:teuthology.orchestra.run.trial030.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-25T08:14:09.739 INFO:teuthology.orchestra.run.trial030.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-25T08:14:09.739 INFO:teuthology.orchestra.run.trial030.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-25T08:14:09.739 INFO:teuthology.orchestra.run.trial030.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-25T08:14:09.739 INFO:teuthology.orchestra.run.trial030.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-25T08:14:09.739 INFO:teuthology.orchestra.run.trial030.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-25T08:14:09.739 INFO:teuthology.orchestra.run.trial030.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-25T08:14:09.739 INFO:teuthology.orchestra.run.trial030.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-25T08:14:09.739 INFO:teuthology.orchestra.run.trial030.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-25T08:14:09.740 INFO:teuthology.orchestra.run.trial030.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-25T08:14:09.740 INFO:teuthology.orchestra.run.trial030.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-25T08:14:09.740 INFO:teuthology.orchestra.run.trial030.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-25T08:14:09.740 INFO:teuthology.orchestra.run.trial030.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-25T08:14:09.740 INFO:teuthology.orchestra.run.trial030.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-25T08:14:09.740 INFO:teuthology.orchestra.run.trial030.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-25T08:14:09.740 INFO:teuthology.orchestra.run.trial030.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-25T08:14:09.740 INFO:teuthology.orchestra.run.trial030.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-25T08:14:09.740 INFO:teuthology.orchestra.run.trial030.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-25T08:14:09.740 INFO:teuthology.orchestra.run.trial030.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-25T08:14:09.740 INFO:teuthology.orchestra.run.trial030.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-25T08:14:09.740 INFO:teuthology.orchestra.run.trial030.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-25T08:14:09.740 INFO:teuthology.orchestra.run.trial030.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-25T08:14:09.741 INFO:teuthology.orchestra.run.trial030.stdout: python3-pyasn1 noarch 0.4.8-7.el9 appstream 157 k 2026-02-25T08:14:09.741 INFO:teuthology.orchestra.run.trial030.stdout: python3-pyasn1-modules noarch 0.4.8-7.el9 appstream 277 k 2026-02-25T08:14:09.741 INFO:teuthology.orchestra.run.trial030.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-25T08:14:09.741 INFO:teuthology.orchestra.run.trial030.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-25T08:14:09.741 INFO:teuthology.orchestra.run.trial030.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-25T08:14:09.741 INFO:teuthology.orchestra.run.trial030.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-25T08:14:09.741 INFO:teuthology.orchestra.run.trial030.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-25T08:14:09.741 INFO:teuthology.orchestra.run.trial030.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-25T08:14:09.741 INFO:teuthology.orchestra.run.trial030.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-25T08:14:09.741 INFO:teuthology.orchestra.run.trial030.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-25T08:14:09.741 INFO:teuthology.orchestra.run.trial030.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-25T08:14:09.741 INFO:teuthology.orchestra.run.trial030.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-25T08:14:09.741 INFO:teuthology.orchestra.run.trial030.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-25T08:14:09.742 INFO:teuthology.orchestra.run.trial030.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-25T08:14:09.742 INFO:teuthology.orchestra.run.trial030.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-25T08:14:09.742 INFO:teuthology.orchestra.run.trial030.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-25T08:14:09.742 INFO:teuthology.orchestra.run.trial030.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-25T08:14:09.742 INFO:teuthology.orchestra.run.trial030.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-25T08:14:09.742 INFO:teuthology.orchestra.run.trial030.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-25T08:14:09.742 INFO:teuthology.orchestra.run.trial030.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-25T08:14:09.742 INFO:teuthology.orchestra.run.trial030.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-25T08:14:09.742 INFO:teuthology.orchestra.run.trial030.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-25T08:14:09.742 INFO:teuthology.orchestra.run.trial030.stdout:Installing weak dependencies: 2026-02-25T08:14:09.742 INFO:teuthology.orchestra.run.trial030.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 184 k 2026-02-25T08:14:09.742 INFO:teuthology.orchestra.run.trial030.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 51 M 2026-02-25T08:14:09.743 INFO:teuthology.orchestra.run.trial030.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 7.4 M 2026-02-25T08:14:09.743 INFO:teuthology.orchestra.run.trial030.stdout: ceph-mgr-k8sevents noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 22 k 2026-02-25T08:14:09.743 INFO:teuthology.orchestra.run.trial030.stdout: ceph-mgr-rook noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 50 k 2026-02-25T08:14:09.743 INFO:teuthology.orchestra.run.trial030.stdout: ceph-volume noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 301 k 2026-02-25T08:14:09.743 INFO:teuthology.orchestra.run.trial030.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-25T08:14:09.743 INFO:teuthology.orchestra.run.trial030.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-25T08:14:09.743 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:14:09.743 INFO:teuthology.orchestra.run.trial030.stdout:Transaction Summary 2026-02-25T08:14:09.743 INFO:teuthology.orchestra.run.trial030.stdout:======================================================================================= 2026-02-25T08:14:09.743 INFO:teuthology.orchestra.run.trial030.stdout:Install 96 Packages 2026-02-25T08:14:09.743 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:14:09.744 INFO:teuthology.orchestra.run.trial030.stdout:Total download size: 226 M 2026-02-25T08:14:09.744 INFO:teuthology.orchestra.run.trial030.stdout:Installed size: 1.5 G 2026-02-25T08:14:09.744 INFO:teuthology.orchestra.run.trial030.stdout:Downloading Packages: 2026-02-25T08:14:10.610 INFO:teuthology.orchestra.run.trial030.stdout:(1/96): ceph-20.3.0-5338.g6aeb6975.el9.x86_64.r 22 kB/s | 6.5 kB 00:00 2026-02-25T08:14:13.757 INFO:teuthology.orchestra.run.trial030.stdout:(2/96): ceph-mgr-20.3.0-5338.g6aeb6975.el9.x86_ 272 kB/s | 937 kB 00:03 2026-02-25T08:14:14.386 INFO:teuthology.orchestra.run.trial030.stdout:(3/96): ceph-osd-20.3.0-5338.g6aeb6975.el9.x86_ 259 kB/s | 163 kB 00:00 2026-02-25T08:14:19.742 INFO:teuthology.orchestra.run.trial030.stdout:(4/96): ceph-mds-20.3.0-5338.g6aeb6975.el9.x86_ 220 kB/s | 2.0 MB 00:09 2026-02-25T08:14:19.925 INFO:teuthology.orchestra.run.trial030.stdout:(5/96): ceph-mon-20.3.0-5338.g6aeb6975.el9.x86_ 535 kB/s | 4.9 MB 00:09 2026-02-25T08:14:20.287 INFO:teuthology.orchestra.run.trial030.stdout:(6/96): ceph-grafana-dashboards-20.3.0-5338.g6a 158 kB/s | 57 kB 00:00 2026-02-25T08:14:21.710 INFO:teuthology.orchestra.run.trial030.stdout:(7/96): ceph-mgr-cephadm-20.3.0-5338.g6aeb6975. 129 kB/s | 184 kB 00:01 2026-02-25T08:14:27.720 INFO:teuthology.orchestra.run.trial023.stdout:(9/96): ceph-mgr-diskprediction-local-20.3.0-53 349 kB/s | 7.4 MB 00:21 2026-02-25T08:14:27.858 INFO:teuthology.orchestra.run.trial023.stdout:(10/96): ceph-mgr-k8sevents-20.3.0-5338.g6aeb69 162 kB/s | 22 kB 00:00 2026-02-25T08:14:29.071 INFO:teuthology.orchestra.run.trial023.stdout:(11/96): ceph-mgr-modules-core-20.3.0-5338.g6ae 245 kB/s | 297 kB 00:01 2026-02-25T08:14:29.354 INFO:teuthology.orchestra.run.trial023.stdout:(12/96): ceph-mgr-rook-20.3.0-5338.g6aeb6975.el 178 kB/s | 50 kB 00:00 2026-02-25T08:14:29.588 INFO:teuthology.orchestra.run.trial023.stdout:(13/96): ceph-prometheus-alerts-20.3.0-5338.g6a 74 kB/s | 17 kB 00:00 2026-02-25T08:14:31.096 INFO:teuthology.orchestra.run.trial023.stdout:(14/96): ceph-volume-20.3.0-5338.g6aeb6975.el9. 200 kB/s | 301 kB 00:01 2026-02-25T08:14:34.319 INFO:teuthology.orchestra.run.trial023.stdout:(15/96): cephadm-20.3.0-5338.g6aeb6975.el9.noar 323 kB/s | 1.0 MB 00:03 2026-02-25T08:14:34.374 INFO:teuthology.orchestra.run.trial023.stdout:(16/96): cryptsetup-2.8.1-3.el9.x86_64.rpm 6.2 MB/s | 351 kB 00:00 2026-02-25T08:14:34.400 INFO:teuthology.orchestra.run.trial023.stdout:(17/96): ledmon-libs-1.1.0-3.el9.x86_64.rpm 1.5 MB/s | 40 kB 00:00 2026-02-25T08:14:34.424 INFO:teuthology.orchestra.run.trial023.stdout:(18/96): libconfig-1.7.2-9.el9.x86_64.rpm 3.1 MB/s | 72 kB 00:00 2026-02-25T08:14:34.456 INFO:teuthology.orchestra.run.trial023.stdout:(19/96): libgfortran-11.5.0-14.el9.x86_64.rpm 24 MB/s | 794 kB 00:00 2026-02-25T08:14:34.481 INFO:teuthology.orchestra.run.trial023.stdout:(20/96): libquadmath-11.5.0-14.el9.x86_64.rpm 7.2 MB/s | 184 kB 00:00 2026-02-25T08:14:34.509 INFO:teuthology.orchestra.run.trial023.stdout:(21/96): python3-cffi-1.14.5-5.el9.x86_64.rpm 9.0 MB/s | 253 kB 00:00 2026-02-25T08:14:34.551 INFO:teuthology.orchestra.run.trial023.stdout:(22/96): python3-cryptography-36.0.1-5.el9.x86_ 30 MB/s | 1.2 MB 00:00 2026-02-25T08:14:34.576 INFO:teuthology.orchestra.run.trial023.stdout:(23/96): python3-ply-3.11-14.el9.noarch.rpm 4.2 MB/s | 106 kB 00:00 2026-02-25T08:14:34.601 INFO:teuthology.orchestra.run.trial023.stdout:(24/96): python3-pycparser-2.20-6.el9.noarch.rp 5.3 MB/s | 135 kB 00:00 2026-02-25T08:14:34.625 INFO:teuthology.orchestra.run.trial023.stdout:(25/96): python3-pyparsing-2.4.7-9.el9.noarch.r 6.0 MB/s | 150 kB 00:00 2026-02-25T08:14:34.649 INFO:teuthology.orchestra.run.trial023.stdout:(26/96): python3-requests-2.25.1-10.el9.noarch. 5.3 MB/s | 126 kB 00:00 2026-02-25T08:14:34.676 INFO:teuthology.orchestra.run.trial023.stdout:(27/96): python3-urllib3-1.26.5-7.el9.noarch.rp 7.9 MB/s | 218 kB 00:00 2026-02-25T08:14:34.703 INFO:teuthology.orchestra.run.trial023.stdout:(28/96): unzip-6.0-59.el9.x86_64.rpm 6.7 MB/s | 182 kB 00:00 2026-02-25T08:14:34.730 INFO:teuthology.orchestra.run.trial023.stdout:(29/96): zip-3.0-35.el9.x86_64.rpm 9.5 MB/s | 266 kB 00:00 2026-02-25T08:14:35.190 INFO:teuthology.orchestra.run.trial023.stdout:(30/96): flexiblas-3.0.4-9.el9.x86_64.rpm 64 kB/s | 30 kB 00:00 2026-02-25T08:14:38.838 INFO:teuthology.orchestra.run.trial023.stdout:(31/96): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 839 kB/s | 3.0 MB 00:03 2026-02-25T08:14:38.940 INFO:teuthology.orchestra.run.trial023.stdout:(32/96): flexiblas-openblas-openmp-3.0.4-9.el9. 144 kB/s | 15 kB 00:00 2026-02-25T08:14:39.291 INFO:teuthology.orchestra.run.trial023.stdout:(33/96): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 703 kB/s | 246 kB 00:00 2026-02-25T08:14:39.559 INFO:teuthology.orchestra.run.trial023.stdout:(34/96): lua-5.4.4-4.el9.x86_64.rpm 702 kB/s | 188 kB 00:00 2026-02-25T08:14:39.662 INFO:teuthology.orchestra.run.trial023.stdout:(35/96): openblas-0.3.29-1.el9.x86_64.rpm 409 kB/s | 42 kB 00:00 2026-02-25T08:14:46.029 INFO:teuthology.orchestra.run.trial023.stdout:(36/96): openblas-openmp-0.3.29-1.el9.x86_64.rp 850 kB/s | 5.3 MB 00:06 2026-02-25T08:14:53.242 INFO:teuthology.orchestra.run.trial023.stdout:(37/96): python3-babel-2.9.1-2.el9.noarch.rpm 847 kB/s | 6.0 MB 00:07 2026-02-25T08:14:53.612 INFO:teuthology.orchestra.run.trial023.stdout:(38/96): python3-devel-3.9.25-3.el9.x86_64.rpm 660 kB/s | 244 kB 00:00 2026-02-25T08:14:53.964 INFO:teuthology.orchestra.run.trial023.stdout:(39/96): python3-jinja2-2.11.3-8.el9.noarch.rpm 708 kB/s | 249 kB 00:00 2026-02-25T08:14:54.068 INFO:teuthology.orchestra.run.trial023.stdout:(40/96): python3-jmespath-1.0.1-1.el9.noarch.rp 457 kB/s | 48 kB 00:00 2026-02-25T08:14:54.337 INFO:teuthology.orchestra.run.trial023.stdout:(41/96): python3-libstoragemgmt-1.10.1-1.el9.x8 658 kB/s | 177 kB 00:00 2026-02-25T08:14:55.184 INFO:teuthology.orchestra.run.trial023.stdout:(42/96): python3-lxml-4.6.5-3.el9.x86_64.rpm 1.4 MB/s | 1.2 MB 00:00 2026-02-25T08:14:55.255 INFO:teuthology.orchestra.run.trial030.stdout:(8/96): ceph-osd-classic-20.3.0-5338.g6aeb6975. 370 kB/s | 15 MB 00:40 2026-02-25T08:14:55.288 INFO:teuthology.orchestra.run.trial023.stdout:(43/96): python3-markupsafe-1.1.1-12.el9.x86_64 335 kB/s | 35 kB 00:00 2026-02-25T08:14:55.650 INFO:teuthology.orchestra.run.trial023.stdout:(44/96): python3-numpy-1.23.5-2.el9.x86_64.rpm 17 MB/s | 6.1 MB 00:00 2026-02-25T08:14:55.756 INFO:teuthology.orchestra.run.trial023.stdout:(45/96): python3-numpy-f2py-1.23.5-2.el9.x86_64 4.1 MB/s | 442 kB 00:00 2026-02-25T08:14:55.861 INFO:teuthology.orchestra.run.trial023.stdout:(46/96): python3-packaging-20.9-5.el9.noarch.rp 738 kB/s | 77 kB 00:00 2026-02-25T08:14:55.966 INFO:teuthology.orchestra.run.trial023.stdout:(47/96): python3-protobuf-3.14.0-17.el9.noarch. 2.5 MB/s | 267 kB 00:00 2026-02-25T08:14:56.070 INFO:teuthology.orchestra.run.trial023.stdout:(48/96): python3-pyasn1-0.4.8-7.el9.noarch.rpm 1.5 MB/s | 157 kB 00:00 2026-02-25T08:14:56.176 INFO:teuthology.orchestra.run.trial023.stdout:(49/96): python3-pyasn1-modules-0.4.8-7.el9.noa 2.6 MB/s | 277 kB 00:00 2026-02-25T08:14:56.280 INFO:teuthology.orchestra.run.trial023.stdout:(50/96): python3-requests-oauthlib-1.3.0-12.el9 515 kB/s | 54 kB 00:00 2026-02-25T08:14:56.826 INFO:teuthology.orchestra.run.trial023.stdout:(51/96): python3-scipy-1.9.3-2.el9.x86_64.rpm 35 MB/s | 19 MB 00:00 2026-02-25T08:14:56.930 INFO:teuthology.orchestra.run.trial023.stdout:(52/96): python3-toml-0.10.2-6.el9.noarch.rpm 402 kB/s | 42 kB 00:00 2026-02-25T08:14:57.034 INFO:teuthology.orchestra.run.trial023.stdout:(53/96): xmlsec1-1.2.29-13.el9.x86_64.rpm 1.8 MB/s | 189 kB 00:00 2026-02-25T08:14:57.137 INFO:teuthology.orchestra.run.trial023.stdout:(54/96): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 875 kB/s | 90 kB 00:00 2026-02-25T08:14:57.161 INFO:teuthology.orchestra.run.trial023.stdout:(55/96): lua-devel-5.4.4-4.el9.x86_64.rpm 934 kB/s | 22 kB 00:00 2026-02-25T08:14:57.188 INFO:teuthology.orchestra.run.trial023.stdout:(56/96): protobuf-compiler-3.14.0-17.el9.x86_64 31 MB/s | 862 kB 00:00 2026-02-25T08:14:57.446 INFO:teuthology.orchestra.run.trial023.stdout:(57/96): abseil-cpp-20211102.0-4.el9.x86_64.rpm 2.1 MB/s | 551 kB 00:00 2026-02-25T08:14:57.479 INFO:teuthology.orchestra.run.trial023.stdout:(58/96): grpc-data-1.46.7-10.el9.noarch.rpm 584 kB/s | 19 kB 00:00 2026-02-25T08:14:57.518 INFO:teuthology.orchestra.run.trial023.stdout:(59/96): luarocks-3.9.2-5.el9.noarch.rpm 3.8 MB/s | 151 kB 00:00 2026-02-25T08:14:57.574 INFO:teuthology.orchestra.run.trial023.stdout:(60/96): python3-asyncssh-2.13.2-5.el9.noarch.r 9.6 MB/s | 548 kB 00:00 2026-02-25T08:14:57.609 INFO:teuthology.orchestra.run.trial023.stdout:(61/96): python3-autocommand-2.2.2-8.el9.noarch 862 kB/s | 29 kB 00:00 2026-02-25T08:14:57.644 INFO:teuthology.orchestra.run.trial023.stdout:(62/96): python3-backports-tarfile-1.2.0-1.el9. 1.7 MB/s | 60 kB 00:00 2026-02-25T08:14:57.678 INFO:teuthology.orchestra.run.trial023.stdout:(63/96): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.2 MB/s | 43 kB 00:00 2026-02-25T08:14:57.713 INFO:teuthology.orchestra.run.trial023.stdout:(64/96): python3-cachetools-4.2.4-1.el9.noarch. 943 kB/s | 32 kB 00:00 2026-02-25T08:14:57.747 INFO:teuthology.orchestra.run.trial023.stdout:(65/96): python3-certifi-2023.05.07-4.el9.noarc 417 kB/s | 14 kB 00:00 2026-02-25T08:14:57.785 INFO:teuthology.orchestra.run.trial023.stdout:(66/96): python3-cheroot-10.0.1-4.el9.noarch.rp 4.4 MB/s | 173 kB 00:00 2026-02-25T08:14:57.831 INFO:teuthology.orchestra.run.trial023.stdout:(67/96): python3-cherrypy-18.6.1-2.el9.noarch.r 7.7 MB/s | 358 kB 00:00 2026-02-25T08:14:57.873 INFO:teuthology.orchestra.run.trial023.stdout:(68/96): python3-google-auth-2.45.0-1.el9.noarc 5.9 MB/s | 254 kB 00:00 2026-02-25T08:14:57.974 INFO:teuthology.orchestra.run.trial023.stdout:(69/96): python3-grpcio-1.46.7-10.el9.x86_64.rp 20 MB/s | 2.0 MB 00:00 2026-02-25T08:14:58.010 INFO:teuthology.orchestra.run.trial023.stdout:(70/96): python3-grpcio-tools-1.46.7-10.el9.x86 3.9 MB/s | 144 kB 00:00 2026-02-25T08:14:58.047 INFO:teuthology.orchestra.run.trial023.stdout:(71/96): python3-influxdb-5.3.1-1.el9.noarch.rp 3.7 MB/s | 139 kB 00:00 2026-02-25T08:14:58.082 INFO:teuthology.orchestra.run.trial023.stdout:(72/96): python3-isodate-0.6.1-3.el9.noarch.rpm 1.6 MB/s | 56 kB 00:00 2026-02-25T08:14:58.116 INFO:teuthology.orchestra.run.trial023.stdout:(73/96): python3-jaraco-8.2.1-3.el9.noarch.rpm 316 kB/s | 11 kB 00:00 2026-02-25T08:14:58.150 INFO:teuthology.orchestra.run.trial023.stdout:(74/96): python3-jaraco-classes-3.2.1-5.el9.noa 524 kB/s | 18 kB 00:00 2026-02-25T08:14:58.184 INFO:teuthology.orchestra.run.trial023.stdout:(75/96): python3-jaraco-collections-3.0.0-8.el9 677 kB/s | 23 kB 00:00 2026-02-25T08:14:58.219 INFO:teuthology.orchestra.run.trial023.stdout:(76/96): python3-jaraco-context-6.0.1-3.el9.noa 560 kB/s | 20 kB 00:00 2026-02-25T08:14:58.254 INFO:teuthology.orchestra.run.trial023.stdout:(77/96): python3-jaraco-functools-3.5.0-2.el9.n 566 kB/s | 19 kB 00:00 2026-02-25T08:14:58.288 INFO:teuthology.orchestra.run.trial023.stdout:(78/96): python3-jaraco-text-4.0.0-2.el9.noarch 772 kB/s | 26 kB 00:00 2026-02-25T08:14:58.333 INFO:teuthology.orchestra.run.trial023.stdout:(79/96): python3-kubernetes-26.1.0-3.el9.noarch 23 MB/s | 1.0 MB 00:00 2026-02-25T08:14:58.368 INFO:teuthology.orchestra.run.trial023.stdout:(80/96): python3-more-itertools-8.12.0-2.el9.no 2.2 MB/s | 79 kB 00:00 2026-02-25T08:14:58.403 INFO:teuthology.orchestra.run.trial023.stdout:(81/96): python3-msgpack-1.0.3-2.el9.x86_64.rpm 2.4 MB/s | 86 kB 00:00 2026-02-25T08:14:58.438 INFO:teuthology.orchestra.run.trial023.stdout:(82/96): python3-natsort-7.1.1-5.el9.noarch.rpm 1.6 MB/s | 58 kB 00:00 2026-02-25T08:14:58.473 INFO:teuthology.orchestra.run.trial023.stdout:(83/96): python3-portend-3.1.0-2.el9.noarch.rpm 474 kB/s | 16 kB 00:00 2026-02-25T08:14:58.508 INFO:teuthology.orchestra.run.trial023.stdout:(84/96): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.5 MB/s | 90 kB 00:00 2026-02-25T08:14:58.542 INFO:teuthology.orchestra.run.trial023.stdout:(85/96): python3-repoze-lru-0.7-16.el9.noarch.r 904 kB/s | 31 kB 00:00 2026-02-25T08:14:58.578 INFO:teuthology.orchestra.run.trial023.stdout:(86/96): python3-routes-2.5.1-5.el9.noarch.rpm 5.1 MB/s | 188 kB 00:00 2026-02-25T08:14:58.613 INFO:teuthology.orchestra.run.trial023.stdout:(87/96): python3-rsa-4.9-2.el9.noarch.rpm 1.7 MB/s | 59 kB 00:00 2026-02-25T08:14:58.649 INFO:teuthology.orchestra.run.trial023.stdout:(88/96): python3-saml-1.16.0-1.el9.noarch.rpm 3.4 MB/s | 125 kB 00:00 2026-02-25T08:14:58.683 INFO:teuthology.orchestra.run.trial023.stdout:(89/96): python3-tempora-5.0.0-2.el9.noarch.rpm 1.0 MB/s | 36 kB 00:00 2026-02-25T08:14:58.719 INFO:teuthology.orchestra.run.trial023.stdout:(90/96): python3-typing-extensions-4.15.0-1.el9 2.4 MB/s | 86 kB 00:00 2026-02-25T08:14:58.753 INFO:teuthology.orchestra.run.trial023.stdout:(91/96): python3-websocket-client-1.2.3-2.el9.n 2.5 MB/s | 90 kB 00:00 2026-02-25T08:14:58.788 INFO:teuthology.orchestra.run.trial023.stdout:(92/96): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 1.4 MB/s | 48 kB 00:00 2026-02-25T08:14:58.822 INFO:teuthology.orchestra.run.trial023.stdout:(93/96): python3-xmltodict-0.12.0-15.el9.noarch 652 kB/s | 22 kB 00:00 2026-02-25T08:14:58.856 INFO:teuthology.orchestra.run.trial023.stdout:(94/96): python3-zc-lockfile-2.0-10.el9.noarch. 590 kB/s | 20 kB 00:00 2026-02-25T08:15:14.157 INFO:teuthology.orchestra.run.trial030.stdout:(9/96): ceph-mgr-diskprediction-local-20.3.0-53 401 kB/s | 7.4 MB 00:18 2026-02-25T08:15:14.227 INFO:teuthology.orchestra.run.trial030.stdout:(10/96): ceph-mgr-k8sevents-20.3.0-5338.g6aeb69 313 kB/s | 22 kB 00:00 2026-02-25T08:15:15.096 INFO:teuthology.orchestra.run.trial030.stdout:(11/96): ceph-mgr-modules-core-20.3.0-5338.g6ae 342 kB/s | 297 kB 00:00 2026-02-25T08:15:15.306 INFO:teuthology.orchestra.run.trial030.stdout:(12/96): ceph-mgr-rook-20.3.0-5338.g6aeb6975.el 240 kB/s | 50 kB 00:00 2026-02-25T08:15:15.445 INFO:teuthology.orchestra.run.trial030.stdout:(13/96): ceph-prometheus-alerts-20.3.0-5338.g6a 124 kB/s | 17 kB 00:00 2026-02-25T08:15:17.686 INFO:teuthology.orchestra.run.trial030.stdout:(14/96): ceph-volume-20.3.0-5338.g6aeb6975.el9. 135 kB/s | 301 kB 00:02 2026-02-25T08:15:21.361 INFO:teuthology.orchestra.run.trial030.stdout:(15/96): cephadm-20.3.0-5338.g6aeb6975.el9.noar 283 kB/s | 1.0 MB 00:03 2026-02-25T08:15:21.386 INFO:teuthology.orchestra.run.trial030.stdout:(16/96): cryptsetup-2.8.1-3.el9.x86_64.rpm 14 MB/s | 351 kB 00:00 2026-02-25T08:15:21.390 INFO:teuthology.orchestra.run.trial030.stdout:(17/96): ledmon-libs-1.1.0-3.el9.x86_64.rpm 9.1 MB/s | 40 kB 00:00 2026-02-25T08:15:21.395 INFO:teuthology.orchestra.run.trial030.stdout:(18/96): libconfig-1.7.2-9.el9.x86_64.rpm 15 MB/s | 72 kB 00:00 2026-02-25T08:15:21.404 INFO:teuthology.orchestra.run.trial030.stdout:(19/96): libgfortran-11.5.0-14.el9.x86_64.rpm 86 MB/s | 794 kB 00:00 2026-02-25T08:15:21.411 INFO:teuthology.orchestra.run.trial030.stdout:(20/96): libquadmath-11.5.0-14.el9.x86_64.rpm 25 MB/s | 184 kB 00:00 2026-02-25T08:15:21.418 INFO:teuthology.orchestra.run.trial030.stdout:(21/96): python3-cffi-1.14.5-5.el9.x86_64.rpm 37 MB/s | 253 kB 00:00 2026-02-25T08:15:21.450 INFO:teuthology.orchestra.run.trial030.stdout:(22/96): python3-cryptography-36.0.1-5.el9.x86_ 40 MB/s | 1.2 MB 00:00 2026-02-25T08:15:21.456 INFO:teuthology.orchestra.run.trial030.stdout:(23/96): python3-ply-3.11-14.el9.noarch.rpm 17 MB/s | 106 kB 00:00 2026-02-25T08:15:21.462 INFO:teuthology.orchestra.run.trial030.stdout:(24/96): python3-pycparser-2.20-6.el9.noarch.rp 21 MB/s | 135 kB 00:00 2026-02-25T08:15:21.468 INFO:teuthology.orchestra.run.trial030.stdout:(25/96): python3-pyparsing-2.4.7-9.el9.noarch.r 23 MB/s | 150 kB 00:00 2026-02-25T08:15:21.475 INFO:teuthology.orchestra.run.trial030.stdout:(26/96): python3-requests-2.25.1-10.el9.noarch. 18 MB/s | 126 kB 00:00 2026-02-25T08:15:21.482 INFO:teuthology.orchestra.run.trial030.stdout:(27/96): python3-urllib3-1.26.5-7.el9.noarch.rp 31 MB/s | 218 kB 00:00 2026-02-25T08:15:21.489 INFO:teuthology.orchestra.run.trial030.stdout:(28/96): unzip-6.0-59.el9.x86_64.rpm 25 MB/s | 182 kB 00:00 2026-02-25T08:15:21.496 INFO:teuthology.orchestra.run.trial030.stdout:(29/96): zip-3.0-35.el9.x86_64.rpm 42 MB/s | 266 kB 00:00 2026-02-25T08:15:21.552 INFO:teuthology.orchestra.run.trial030.stdout:(30/96): flexiblas-3.0.4-9.el9.x86_64.rpm 530 kB/s | 30 kB 00:00 2026-02-25T08:15:21.684 INFO:teuthology.orchestra.run.trial030.stdout:(31/96): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 23 MB/s | 3.0 MB 00:00 2026-02-25T08:15:21.704 INFO:teuthology.orchestra.run.trial030.stdout:(32/96): flexiblas-openblas-openmp-3.0.4-9.el9. 756 kB/s | 15 kB 00:00 2026-02-25T08:15:21.725 INFO:teuthology.orchestra.run.trial030.stdout:(33/96): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 11 MB/s | 246 kB 00:00 2026-02-25T08:15:21.747 INFO:teuthology.orchestra.run.trial030.stdout:(34/96): lua-5.4.4-4.el9.x86_64.rpm 8.6 MB/s | 188 kB 00:00 2026-02-25T08:15:21.766 INFO:teuthology.orchestra.run.trial030.stdout:(35/96): openblas-0.3.29-1.el9.x86_64.rpm 2.1 MB/s | 42 kB 00:00 2026-02-25T08:15:21.815 INFO:teuthology.orchestra.run.trial030.stdout:(36/96): openblas-openmp-0.3.29-1.el9.x86_64.rp 109 MB/s | 5.3 MB 00:00 2026-02-25T08:15:21.869 INFO:teuthology.orchestra.run.trial030.stdout:(37/96): python3-babel-2.9.1-2.el9.noarch.rpm 111 MB/s | 6.0 MB 00:00 2026-02-25T08:15:21.891 INFO:teuthology.orchestra.run.trial030.stdout:(38/96): python3-devel-3.9.25-3.el9.x86_64.rpm 11 MB/s | 244 kB 00:00 2026-02-25T08:15:21.913 INFO:teuthology.orchestra.run.trial030.stdout:(39/96): python3-jinja2-2.11.3-8.el9.noarch.rpm 11 MB/s | 249 kB 00:00 2026-02-25T08:15:21.934 INFO:teuthology.orchestra.run.trial030.stdout:(40/96): python3-jmespath-1.0.1-1.el9.noarch.rp 2.3 MB/s | 48 kB 00:00 2026-02-25T08:15:21.955 INFO:teuthology.orchestra.run.trial030.stdout:(41/96): python3-libstoragemgmt-1.10.1-1.el9.x8 8.2 MB/s | 177 kB 00:00 2026-02-25T08:15:21.980 INFO:teuthology.orchestra.run.trial030.stdout:(42/96): python3-lxml-4.6.5-3.el9.x86_64.rpm 49 MB/s | 1.2 MB 00:00 2026-02-25T08:15:22.000 INFO:teuthology.orchestra.run.trial030.stdout:(43/96): python3-markupsafe-1.1.1-12.el9.x86_64 1.7 MB/s | 35 kB 00:00 2026-02-25T08:15:22.166 INFO:teuthology.orchestra.run.trial030.stdout:(44/96): python3-numpy-1.23.5-2.el9.x86_64.rpm 37 MB/s | 6.1 MB 00:00 2026-02-25T08:15:22.190 INFO:teuthology.orchestra.run.trial030.stdout:(45/96): python3-numpy-f2py-1.23.5-2.el9.x86_64 19 MB/s | 442 kB 00:00 2026-02-25T08:15:22.210 INFO:teuthology.orchestra.run.trial030.stdout:(46/96): python3-packaging-20.9-5.el9.noarch.rp 3.7 MB/s | 77 kB 00:00 2026-02-25T08:15:22.232 INFO:teuthology.orchestra.run.trial030.stdout:(47/96): python3-protobuf-3.14.0-17.el9.noarch. 12 MB/s | 267 kB 00:00 2026-02-25T08:15:22.253 INFO:teuthology.orchestra.run.trial030.stdout:(48/96): python3-pyasn1-0.4.8-7.el9.noarch.rpm 7.1 MB/s | 157 kB 00:00 2026-02-25T08:15:22.276 INFO:teuthology.orchestra.run.trial030.stdout:(49/96): python3-pyasn1-modules-0.4.8-7.el9.noa 12 MB/s | 277 kB 00:00 2026-02-25T08:15:22.296 INFO:teuthology.orchestra.run.trial030.stdout:(50/96): python3-requests-oauthlib-1.3.0-12.el9 2.5 MB/s | 54 kB 00:00 2026-02-25T08:15:22.447 INFO:teuthology.orchestra.run.trial030.stdout:(51/96): python3-scipy-1.9.3-2.el9.x86_64.rpm 128 MB/s | 19 MB 00:00 2026-02-25T08:15:22.542 INFO:teuthology.orchestra.run.trial030.stdout:(52/96): python3-toml-0.10.2-6.el9.noarch.rpm 439 kB/s | 42 kB 00:00 2026-02-25T08:15:22.563 INFO:teuthology.orchestra.run.trial030.stdout:(53/96): xmlsec1-1.2.29-13.el9.x86_64.rpm 8.7 MB/s | 189 kB 00:00 2026-02-25T08:15:22.583 INFO:teuthology.orchestra.run.trial030.stdout:(54/96): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 4.4 MB/s | 90 kB 00:00 2026-02-25T08:15:22.626 INFO:teuthology.orchestra.run.trial030.stdout:(55/96): lua-devel-5.4.4-4.el9.x86_64.rpm 523 kB/s | 22 kB 00:00 2026-02-25T08:15:22.712 INFO:teuthology.orchestra.run.trial030.stdout:(56/96): protobuf-compiler-3.14.0-17.el9.x86_64 9.8 MB/s | 862 kB 00:00 2026-02-25T08:15:22.956 INFO:teuthology.orchestra.run.trial030.stdout:(57/96): abseil-cpp-20211102.0-4.el9.x86_64.rpm 2.2 MB/s | 551 kB 00:00 2026-02-25T08:15:22.990 INFO:teuthology.orchestra.run.trial030.stdout:(58/96): grpc-data-1.46.7-10.el9.noarch.rpm 583 kB/s | 19 kB 00:00 2026-02-25T08:15:23.028 INFO:teuthology.orchestra.run.trial030.stdout:(59/96): luarocks-3.9.2-5.el9.noarch.rpm 3.9 MB/s | 151 kB 00:00 2026-02-25T08:15:23.083 INFO:teuthology.orchestra.run.trial030.stdout:(60/96): python3-asyncssh-2.13.2-5.el9.noarch.r 9.7 MB/s | 548 kB 00:00 2026-02-25T08:15:23.117 INFO:teuthology.orchestra.run.trial030.stdout:(61/96): python3-autocommand-2.2.2-8.el9.noarch 874 kB/s | 29 kB 00:00 2026-02-25T08:15:23.152 INFO:teuthology.orchestra.run.trial030.stdout:(62/96): python3-backports-tarfile-1.2.0-1.el9. 1.7 MB/s | 60 kB 00:00 2026-02-25T08:15:23.185 INFO:teuthology.orchestra.run.trial030.stdout:(63/96): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.3 MB/s | 43 kB 00:00 2026-02-25T08:15:23.219 INFO:teuthology.orchestra.run.trial030.stdout:(64/96): python3-cachetools-4.2.4-1.el9.noarch. 959 kB/s | 32 kB 00:00 2026-02-25T08:15:23.252 INFO:teuthology.orchestra.run.trial030.stdout:(65/96): python3-certifi-2023.05.07-4.el9.noarc 423 kB/s | 14 kB 00:00 2026-02-25T08:15:23.291 INFO:teuthology.orchestra.run.trial030.stdout:(66/96): python3-cheroot-10.0.1-4.el9.noarch.rp 4.5 MB/s | 173 kB 00:00 2026-02-25T08:15:23.335 INFO:teuthology.orchestra.run.trial030.stdout:(67/96): python3-cherrypy-18.6.1-2.el9.noarch.r 7.8 MB/s | 358 kB 00:00 2026-02-25T08:15:23.377 INFO:teuthology.orchestra.run.trial030.stdout:(68/96): python3-google-auth-2.45.0-1.el9.noarc 6.0 MB/s | 254 kB 00:00 2026-02-25T08:15:23.477 INFO:teuthology.orchestra.run.trial030.stdout:(69/96): python3-grpcio-1.46.7-10.el9.x86_64.rp 20 MB/s | 2.0 MB 00:00 2026-02-25T08:15:23.513 INFO:teuthology.orchestra.run.trial030.stdout:(70/96): python3-grpcio-tools-1.46.7-10.el9.x86 4.0 MB/s | 144 kB 00:00 2026-02-25T08:15:23.549 INFO:teuthology.orchestra.run.trial030.stdout:(71/96): python3-influxdb-5.3.1-1.el9.noarch.rp 3.8 MB/s | 139 kB 00:00 2026-02-25T08:15:23.583 INFO:teuthology.orchestra.run.trial030.stdout:(72/96): python3-isodate-0.6.1-3.el9.noarch.rpm 1.6 MB/s | 56 kB 00:00 2026-02-25T08:15:23.616 INFO:teuthology.orchestra.run.trial030.stdout:(73/96): python3-jaraco-8.2.1-3.el9.noarch.rpm 321 kB/s | 11 kB 00:00 2026-02-25T08:15:23.650 INFO:teuthology.orchestra.run.trial030.stdout:(74/96): python3-jaraco-classes-3.2.1-5.el9.noa 530 kB/s | 18 kB 00:00 2026-02-25T08:15:23.684 INFO:teuthology.orchestra.run.trial030.stdout:(75/96): python3-jaraco-collections-3.0.0-8.el9 690 kB/s | 23 kB 00:00 2026-02-25T08:15:23.717 INFO:teuthology.orchestra.run.trial030.stdout:(76/96): python3-jaraco-context-6.0.1-3.el9.noa 586 kB/s | 20 kB 00:00 2026-02-25T08:15:23.751 INFO:teuthology.orchestra.run.trial030.stdout:(77/96): python3-jaraco-functools-3.5.0-2.el9.n 582 kB/s | 19 kB 00:00 2026-02-25T08:15:23.785 INFO:teuthology.orchestra.run.trial030.stdout:(78/96): python3-jaraco-text-4.0.0-2.el9.noarch 782 kB/s | 26 kB 00:00 2026-02-25T08:15:23.829 INFO:teuthology.orchestra.run.trial030.stdout:(79/96): python3-kubernetes-26.1.0-3.el9.noarch 23 MB/s | 1.0 MB 00:00 2026-02-25T08:15:23.864 INFO:teuthology.orchestra.run.trial030.stdout:(80/96): python3-more-itertools-8.12.0-2.el9.no 2.2 MB/s | 79 kB 00:00 2026-02-25T08:15:23.898 INFO:teuthology.orchestra.run.trial030.stdout:(81/96): python3-msgpack-1.0.3-2.el9.x86_64.rpm 2.5 MB/s | 86 kB 00:00 2026-02-25T08:15:23.932 INFO:teuthology.orchestra.run.trial030.stdout:(82/96): python3-natsort-7.1.1-5.el9.noarch.rpm 1.7 MB/s | 58 kB 00:00 2026-02-25T08:15:23.965 INFO:teuthology.orchestra.run.trial030.stdout:(83/96): python3-portend-3.1.0-2.el9.noarch.rpm 492 kB/s | 16 kB 00:00 2026-02-25T08:15:24.000 INFO:teuthology.orchestra.run.trial030.stdout:(84/96): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.6 MB/s | 90 kB 00:00 2026-02-25T08:15:24.033 INFO:teuthology.orchestra.run.trial030.stdout:(85/96): python3-repoze-lru-0.7-16.el9.noarch.r 917 kB/s | 31 kB 00:00 2026-02-25T08:15:24.068 INFO:teuthology.orchestra.run.trial030.stdout:(86/96): python3-routes-2.5.1-5.el9.noarch.rpm 5.3 MB/s | 188 kB 00:00 2026-02-25T08:15:24.103 INFO:teuthology.orchestra.run.trial030.stdout:(87/96): python3-rsa-4.9-2.el9.noarch.rpm 1.7 MB/s | 59 kB 00:00 2026-02-25T08:15:24.137 INFO:teuthology.orchestra.run.trial030.stdout:(88/96): python3-saml-1.16.0-1.el9.noarch.rpm 3.5 MB/s | 125 kB 00:00 2026-02-25T08:15:24.171 INFO:teuthology.orchestra.run.trial030.stdout:(89/96): python3-tempora-5.0.0-2.el9.noarch.rpm 1.0 MB/s | 36 kB 00:00 2026-02-25T08:15:24.207 INFO:teuthology.orchestra.run.trial030.stdout:(90/96): python3-typing-extensions-4.15.0-1.el9 2.3 MB/s | 86 kB 00:00 2026-02-25T08:15:24.241 INFO:teuthology.orchestra.run.trial030.stdout:(91/96): python3-websocket-client-1.2.3-2.el9.n 2.6 MB/s | 90 kB 00:00 2026-02-25T08:15:24.275 INFO:teuthology.orchestra.run.trial030.stdout:(92/96): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 1.4 MB/s | 48 kB 00:00 2026-02-25T08:15:24.309 INFO:teuthology.orchestra.run.trial030.stdout:(93/96): python3-xmltodict-0.12.0-15.el9.noarch 663 kB/s | 22 kB 00:00 2026-02-25T08:15:24.342 INFO:teuthology.orchestra.run.trial030.stdout:(94/96): python3-zc-lockfile-2.0-10.el9.noarch. 596 kB/s | 20 kB 00:00 2026-02-25T08:16:06.696 INFO:teuthology.orchestra.run.trial023.stdout:(95/96): ceph-mgr-dashboard-20.3.0-5338.g6aeb69 379 kB/s | 51 MB 02:16 2026-02-25T08:17:39.632 INFO:teuthology.orchestra.run.trial023.stdout:(96/96): ceph-osd-crimson-20.3.0-5338.g6aeb6975 375 kB/s | 88 MB 03:59 2026-02-25T08:17:39.635 INFO:teuthology.orchestra.run.trial023.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:17:39.636 INFO:teuthology.orchestra.run.trial023.stdout:Total 956 kB/s | 226 MB 04:01 2026-02-25T08:17:40.005 INFO:teuthology.orchestra.run.trial023.stdout:Running transaction check 2026-02-25T08:17:40.044 INFO:teuthology.orchestra.run.trial023.stdout:Transaction check succeeded. 2026-02-25T08:17:40.044 INFO:teuthology.orchestra.run.trial023.stdout:Running transaction test 2026-02-25T08:17:40.852 INFO:teuthology.orchestra.run.trial023.stdout:Transaction test succeeded. 2026-02-25T08:17:40.852 INFO:teuthology.orchestra.run.trial023.stdout:Running transaction 2026-02-25T08:17:41.682 INFO:teuthology.orchestra.run.trial023.stdout: Preparing : 1/1 2026-02-25T08:17:41.684 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/96 2026-02-25T08:17:41.688 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/96 2026-02-25T08:17:41.710 INFO:teuthology.orchestra.run.trial023.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 3/96 2026-02-25T08:17:41.743 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-pyasn1-0.4.8-7.el9.noarch 4/96 2026-02-25T08:17:41.749 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 5/96 2026-02-25T08:17:41.751 INFO:teuthology.orchestra.run.trial023.stdout: Installing : lua-5.4.4-4.el9.x86_64 6/96 2026-02-25T08:17:41.756 INFO:teuthology.orchestra.run.trial023.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 7/96 2026-02-25T08:17:41.779 INFO:teuthology.orchestra.run.trial023.stdout: Installing : unzip-6.0-59.el9.x86_64 8/96 2026-02-25T08:17:41.792 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 9/96 2026-02-25T08:17:41.795 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 10/96 2026-02-25T08:17:41.801 INFO:teuthology.orchestra.run.trial023.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 11/96 2026-02-25T08:17:41.803 INFO:teuthology.orchestra.run.trial023.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 12/96 2026-02-25T08:17:41.814 INFO:teuthology.orchestra.run.trial023.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 13/96 2026-02-25T08:17:41.820 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 14/96 2026-02-25T08:17:41.844 INFO:teuthology.orchestra.run.trial023.stdout: Installing : zip-3.0-35.el9.x86_64 15/96 2026-02-25T08:17:41.847 INFO:teuthology.orchestra.run.trial023.stdout: Installing : luarocks-3.9.2-5.el9.noarch 16/96 2026-02-25T08:17:41.897 INFO:teuthology.orchestra.run.trial023.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 17/96 2026-02-25T08:17:41.910 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-pyasn1-modules-0.4.8-7.el9.noarch 18/96 2026-02-25T08:17:41.912 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-rsa-4.9-2.el9.noarch 19/96 2026-02-25T08:17:41.917 INFO:teuthology.orchestra.run.trial023.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 20/96 2026-02-25T08:17:41.922 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 21/96 2026-02-25T08:17:41.927 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 22/96 2026-02-25T08:17:41.931 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 23/96 2026-02-25T08:17:41.945 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 24/96 2026-02-25T08:17:41.949 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 25/96 2026-02-25T08:17:41.954 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 26/96 2026-02-25T08:17:41.965 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 27/96 2026-02-25T08:17:41.974 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 28/96 2026-02-25T08:17:41.981 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 29/96 2026-02-25T08:17:42.000 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 30/96 2026-02-25T08:17:42.011 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 31/96 2026-02-25T08:17:42.027 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 32/96 2026-02-25T08:17:42.031 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 33/96 2026-02-25T08:17:42.039 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 34/96 2026-02-25T08:17:42.079 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 35/96 2026-02-25T08:17:42.404 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 36/96 2026-02-25T08:17:42.413 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 37/96 2026-02-25T08:17:42.417 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 38/96 2026-02-25T08:17:42.423 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 39/96 2026-02-25T08:17:42.425 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 40/96 2026-02-25T08:17:42.452 INFO:teuthology.orchestra.run.trial023.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 41/96 2026-02-25T08:17:42.496 INFO:teuthology.orchestra.run.trial023.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 42/96 2026-02-25T08:17:42.504 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 43/96 2026-02-25T08:17:42.510 INFO:teuthology.orchestra.run.trial023.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 44/96 2026-02-25T08:17:42.514 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 45/96 2026-02-25T08:17:42.519 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 46/96 2026-02-25T08:17:42.523 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 47/96 2026-02-25T08:17:42.530 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 48/96 2026-02-25T08:17:42.533 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 49/96 2026-02-25T08:17:42.561 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 50/96 2026-02-25T08:17:42.571 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 51/96 2026-02-25T08:17:42.577 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 52/96 2026-02-25T08:17:42.583 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 53/96 2026-02-25T08:17:42.618 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 54/96 2026-02-25T08:17:42.842 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 55/96 2026-02-25T08:17:42.865 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 56/96 2026-02-25T08:17:42.869 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 57/96 2026-02-25T08:17:42.928 INFO:teuthology.orchestra.run.trial023.stdout: Installing : openblas-0.3.29-1.el9.x86_64 58/96 2026-02-25T08:17:42.930 INFO:teuthology.orchestra.run.trial023.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 59/96 2026-02-25T08:17:42.951 INFO:teuthology.orchestra.run.trial023.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 60/96 2026-02-25T08:17:43.278 INFO:teuthology.orchestra.run.trial023.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 61/96 2026-02-25T08:17:43.351 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 62/96 2026-02-25T08:17:44.023 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 63/96 2026-02-25T08:17:44.040 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 64/96 2026-02-25T08:17:44.050 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 65/96 2026-02-25T08:17:44.056 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-packaging-20.9-5.el9.noarch 66/96 2026-02-25T08:17:44.070 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-ply-3.11-14.el9.noarch 67/96 2026-02-25T08:17:44.085 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 68/96 2026-02-25T08:17:44.161 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 69/96 2026-02-25T08:17:44.170 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 70/96 2026-02-25T08:17:44.193 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 71/96 2026-02-25T08:17:44.223 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 72/96 2026-02-25T08:17:44.273 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 73/96 2026-02-25T08:17:44.279 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 74/96 2026-02-25T08:17:44.283 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 75/96 2026-02-25T08:17:44.284 INFO:teuthology.orchestra.run.trial023.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 76/96 2026-02-25T08:17:44.303 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-25T08:17:44.303 INFO:teuthology.orchestra.run.trial023.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-25T08:17:44.303 INFO:teuthology.orchestra.run.trial023.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-25T08:17:44.303 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:17:44.311 INFO:teuthology.orchestra.run.trial023.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-25T08:17:44.336 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-25T08:17:44.336 INFO:teuthology.orchestra.run.trial023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-25T08:17:44.336 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:17:44.370 INFO:teuthology.orchestra.run.trial023.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 78/96 2026-02-25T08:17:44.957 INFO:teuthology.orchestra.run.trial023.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 79/96 2026-02-25T08:17:50.378 INFO:teuthology.orchestra.run.trial023.stdout: Installing : ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_6 80/96 2026-02-25T08:17:50.441 INFO:teuthology.orchestra.run.trial023.stdout: Installing : ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_6 81/96 2026-02-25T08:17:50.443 INFO:teuthology.orchestra.run.trial023.stdout: Installing : ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 82/96 2026-02-25T08:17:50.457 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 82/96 2026-02-25T08:17:50.457 INFO:teuthology.orchestra.run.trial023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:17:50.457 INFO:teuthology.orchestra.run.trial023.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-25T08:17:50.457 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:17:50.469 INFO:teuthology.orchestra.run.trial023.stdout: Installing : ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 83/96 2026-02-25T08:17:50.493 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 83/96 2026-02-25T08:17:50.493 INFO:teuthology.orchestra.run.trial023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:17:50.493 INFO:teuthology.orchestra.run.trial023.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-25T08:17:50.493 INFO:teuthology.orchestra.run.trial023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-25T08:17:50.494 INFO:teuthology.orchestra.run.trial023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-25T08:17:50.494 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:17:50.549 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 84/96 2026-02-25T08:17:50.551 INFO:teuthology.orchestra.run.trial023.stdout: Installing : cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 84/96 2026-02-25T08:17:50.560 INFO:teuthology.orchestra.run.trial023.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5338.g6aeb6975.el9 85/96 2026-02-25T08:17:50.584 INFO:teuthology.orchestra.run.trial023.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5338.g6aeb6975.el 86/96 2026-02-25T08:17:50.585 INFO:teuthology.orchestra.run.trial023.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarc 87/96 2026-02-25T08:17:52.484 INFO:teuthology.orchestra.run.trial030.stdout:(95/96): ceph-mgr-dashboard-20.3.0-5338.g6aeb69 246 kB/s | 51 MB 03:30 2026-02-25T08:17:53.481 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarc 87/96 2026-02-25T08:17:53.487 INFO:teuthology.orchestra.run.trial023.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noa 88/96 2026-02-25T08:17:53.945 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noa 88/96 2026-02-25T08:17:53.946 INFO:teuthology.orchestra.run.trial023.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6 89/96 2026-02-25T08:17:53.958 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6 89/96 2026-02-25T08:17:53.958 INFO:teuthology.orchestra.run.trial023.stdout: Installing : ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noa 90/96 2026-02-25T08:17:54.019 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noa 90/96 2026-02-25T08:17:54.059 INFO:teuthology.orchestra.run.trial023.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5338.g6aeb6975.el9. 91/96 2026-02-25T08:17:54.060 INFO:teuthology.orchestra.run.trial023.stdout: Installing : ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 92/96 2026-02-25T08:17:54.080 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 92/96 2026-02-25T08:17:54.081 INFO:teuthology.orchestra.run.trial023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:17:54.081 INFO:teuthology.orchestra.run.trial023.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-25T08:17:54.081 INFO:teuthology.orchestra.run.trial023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-25T08:17:54.081 INFO:teuthology.orchestra.run.trial023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-25T08:17:54.081 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:17:54.093 INFO:teuthology.orchestra.run.trial023.stdout: Installing : ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 93/96 2026-02-25T08:17:54.306 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 93/96 2026-02-25T08:17:54.308 INFO:teuthology.orchestra.run.trial023.stdout: Installing : ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 94/96 2026-02-25T08:17:54.328 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 94/96 2026-02-25T08:17:54.328 INFO:teuthology.orchestra.run.trial023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:17:54.328 INFO:teuthology.orchestra.run.trial023.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-25T08:17:54.329 INFO:teuthology.orchestra.run.trial023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-25T08:17:54.329 INFO:teuthology.orchestra.run.trial023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-25T08:17:54.329 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:17:54.410 INFO:teuthology.orchestra.run.trial023.stdout: Installing : ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 95/96 2026-02-25T08:17:54.430 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 95/96 2026-02-25T08:17:54.430 INFO:teuthology.orchestra.run.trial023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:17:54.430 INFO:teuthology.orchestra.run.trial023.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-25T08:17:54.430 INFO:teuthology.orchestra.run.trial023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-25T08:17:54.430 INFO:teuthology.orchestra.run.trial023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-25T08:17:54.430 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:17:54.431 INFO:teuthology.orchestra.run.trial023.stdout: Installing : ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 96/96 2026-02-25T08:17:54.437 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_6 96/96 2026-02-25T08:17:54.460 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_6 96/96 2026-02-25T08:17:55.823 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 96/96 2026-02-25T08:17:55.824 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/96 2026-02-25T08:17:55.824 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/96 2026-02-25T08:17:55.824 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/96 2026-02-25T08:17:55.824 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 4/96 2026-02-25T08:17:55.824 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 5/96 2026-02-25T08:17:55.824 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_6 6/96 2026-02-25T08:17:55.824 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_6 7/96 2026-02-25T08:17:55.824 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5338.g6aeb6975.el 8/96 2026-02-25T08:17:55.824 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarc 9/96 2026-02-25T08:17:55.824 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noa 10/96 2026-02-25T08:17:55.825 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6 11/96 2026-02-25T08:17:55.825 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noa 12/96 2026-02-25T08:17:55.825 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5338.g6aeb6975.el9. 13/96 2026-02-25T08:17:55.825 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 14/96 2026-02-25T08:17:55.825 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5338.g6aeb6975.el9 15/96 2026-02-25T08:17:55.825 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 16/96 2026-02-25T08:17:55.825 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 17/96 2026-02-25T08:17:55.825 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 18/96 2026-02-25T08:17:55.825 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 19/96 2026-02-25T08:17:55.825 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 20/96 2026-02-25T08:17:55.825 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 21/96 2026-02-25T08:17:55.826 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 22/96 2026-02-25T08:17:55.826 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 23/96 2026-02-25T08:17:55.826 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 24/96 2026-02-25T08:17:55.826 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-ply-3.11-14.el9.noarch 25/96 2026-02-25T08:17:55.826 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 26/96 2026-02-25T08:17:55.826 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 27/96 2026-02-25T08:17:55.826 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 28/96 2026-02-25T08:17:55.826 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 29/96 2026-02-25T08:17:55.826 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : unzip-6.0-59.el9.x86_64 30/96 2026-02-25T08:17:55.826 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : zip-3.0-35.el9.x86_64 31/96 2026-02-25T08:17:55.826 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 32/96 2026-02-25T08:17:55.826 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 33/96 2026-02-25T08:17:55.827 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 34/96 2026-02-25T08:17:55.827 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 35/96 2026-02-25T08:17:55.827 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : lua-5.4.4-4.el9.x86_64 36/96 2026-02-25T08:17:55.827 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 37/96 2026-02-25T08:17:55.827 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 38/96 2026-02-25T08:17:55.827 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 39/96 2026-02-25T08:17:55.827 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 40/96 2026-02-25T08:17:55.827 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 41/96 2026-02-25T08:17:55.827 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 42/96 2026-02-25T08:17:55.827 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 43/96 2026-02-25T08:17:55.827 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 44/96 2026-02-25T08:17:55.827 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 45/96 2026-02-25T08:17:55.827 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 46/96 2026-02-25T08:17:55.828 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 47/96 2026-02-25T08:17:55.828 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 48/96 2026-02-25T08:17:55.828 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 49/96 2026-02-25T08:17:55.828 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-pyasn1-0.4.8-7.el9.noarch 50/96 2026-02-25T08:17:55.828 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-pyasn1-modules-0.4.8-7.el9.noarch 51/96 2026-02-25T08:17:55.828 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 52/96 2026-02-25T08:17:55.828 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 53/96 2026-02-25T08:17:55.828 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 54/96 2026-02-25T08:17:55.828 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 55/96 2026-02-25T08:17:55.828 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 56/96 2026-02-25T08:17:55.828 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 57/96 2026-02-25T08:17:55.828 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 58/96 2026-02-25T08:17:55.828 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 59/96 2026-02-25T08:17:55.829 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 60/96 2026-02-25T08:17:55.829 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 61/96 2026-02-25T08:17:55.829 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 62/96 2026-02-25T08:17:55.829 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 63/96 2026-02-25T08:17:55.829 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 64/96 2026-02-25T08:17:55.829 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 65/96 2026-02-25T08:17:55.829 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 66/96 2026-02-25T08:17:55.829 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 67/96 2026-02-25T08:17:55.829 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 68/96 2026-02-25T08:17:55.829 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 69/96 2026-02-25T08:17:55.829 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 70/96 2026-02-25T08:17:55.829 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 71/96 2026-02-25T08:17:55.830 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 72/96 2026-02-25T08:17:55.830 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 73/96 2026-02-25T08:17:55.830 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 74/96 2026-02-25T08:17:55.830 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 75/96 2026-02-25T08:17:55.830 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 76/96 2026-02-25T08:17:55.830 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 77/96 2026-02-25T08:17:55.830 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 78/96 2026-02-25T08:17:55.830 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 79/96 2026-02-25T08:17:55.830 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 80/96 2026-02-25T08:17:55.830 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 81/96 2026-02-25T08:17:55.830 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/96 2026-02-25T08:17:55.830 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/96 2026-02-25T08:17:55.830 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/96 2026-02-25T08:17:55.831 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 85/96 2026-02-25T08:17:55.831 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/96 2026-02-25T08:17:55.831 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 87/96 2026-02-25T08:17:55.831 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 88/96 2026-02-25T08:17:55.831 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 89/96 2026-02-25T08:17:55.831 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 90/96 2026-02-25T08:17:55.831 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 91/96 2026-02-25T08:17:55.831 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 92/96 2026-02-25T08:17:55.831 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 93/96 2026-02-25T08:17:55.831 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 94/96 2026-02-25T08:17:55.831 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 95/96 2026-02-25T08:17:55.891 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 96/96 2026-02-25T08:17:55.891 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:17:55.891 INFO:teuthology.orchestra.run.trial023.stdout:Installed: 2026-02-25T08:17:55.891 INFO:teuthology.orchestra.run.trial023.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-25T08:17:55.891 INFO:teuthology.orchestra.run.trial023.stdout: ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:17:55.891 INFO:teuthology.orchestra.run.trial023.stdout: ceph-grafana-dashboards-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:17:55.891 INFO:teuthology.orchestra.run.trial023.stdout: ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:17:55.891 INFO:teuthology.orchestra.run.trial023.stdout: ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:17:55.891 INFO:teuthology.orchestra.run.trial023.stdout: ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:17:55.891 INFO:teuthology.orchestra.run.trial023.stdout: ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:17:55.891 INFO:teuthology.orchestra.run.trial023.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:17:55.891 INFO:teuthology.orchestra.run.trial023.stdout: ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:17:55.891 INFO:teuthology.orchestra.run.trial023.stdout: ceph-mgr-modules-core-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:17:55.891 INFO:teuthology.orchestra.run.trial023.stdout: ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:17:55.891 INFO:teuthology.orchestra.run.trial023.stdout: ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:17:55.891 INFO:teuthology.orchestra.run.trial023.stdout: ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:17:55.892 INFO:teuthology.orchestra.run.trial023.stdout: ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:17:55.892 INFO:teuthology.orchestra.run.trial023.stdout: ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:17:55.892 INFO:teuthology.orchestra.run.trial023.stdout: ceph-prometheus-alerts-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:17:55.892 INFO:teuthology.orchestra.run.trial023.stdout: ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:17:55.892 INFO:teuthology.orchestra.run.trial023.stdout: cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:17:55.892 INFO:teuthology.orchestra.run.trial023.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-25T08:17:55.892 INFO:teuthology.orchestra.run.trial023.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-25T08:17:55.892 INFO:teuthology.orchestra.run.trial023.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-25T08:17:55.892 INFO:teuthology.orchestra.run.trial023.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-25T08:17:55.892 INFO:teuthology.orchestra.run.trial023.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-25T08:17:55.892 INFO:teuthology.orchestra.run.trial023.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-25T08:17:55.892 INFO:teuthology.orchestra.run.trial023.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-25T08:17:55.892 INFO:teuthology.orchestra.run.trial023.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-25T08:17:55.892 INFO:teuthology.orchestra.run.trial023.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-25T08:17:55.892 INFO:teuthology.orchestra.run.trial023.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-25T08:17:55.892 INFO:teuthology.orchestra.run.trial023.stdout: lua-5.4.4-4.el9.x86_64 2026-02-25T08:17:55.892 INFO:teuthology.orchestra.run.trial023.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-25T08:17:55.892 INFO:teuthology.orchestra.run.trial023.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-25T08:17:55.892 INFO:teuthology.orchestra.run.trial023.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-25T08:17:55.892 INFO:teuthology.orchestra.run.trial023.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-25T08:17:55.892 INFO:teuthology.orchestra.run.trial023.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-25T08:17:55.892 INFO:teuthology.orchestra.run.trial023.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-25T08:17:55.892 INFO:teuthology.orchestra.run.trial023.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-25T08:17:55.893 INFO:teuthology.orchestra.run.trial023.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-25T08:17:55.893 INFO:teuthology.orchestra.run.trial023.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-25T08:17:55.893 INFO:teuthology.orchestra.run.trial023.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-25T08:17:55.893 INFO:teuthology.orchestra.run.trial023.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-25T08:17:55.893 INFO:teuthology.orchestra.run.trial023.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-25T08:17:55.893 INFO:teuthology.orchestra.run.trial023.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-25T08:17:55.893 INFO:teuthology.orchestra.run.trial023.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-25T08:17:55.893 INFO:teuthology.orchestra.run.trial023.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-25T08:17:55.893 INFO:teuthology.orchestra.run.trial023.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-25T08:17:55.893 INFO:teuthology.orchestra.run.trial023.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-25T08:17:55.893 INFO:teuthology.orchestra.run.trial023.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-25T08:17:55.893 INFO:teuthology.orchestra.run.trial023.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-25T08:17:55.893 INFO:teuthology.orchestra.run.trial023.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-25T08:17:55.893 INFO:teuthology.orchestra.run.trial023.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-25T08:17:55.893 INFO:teuthology.orchestra.run.trial023.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-25T08:17:55.893 INFO:teuthology.orchestra.run.trial023.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-25T08:17:55.893 INFO:teuthology.orchestra.run.trial023.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-25T08:17:55.893 INFO:teuthology.orchestra.run.trial023.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-25T08:17:55.893 INFO:teuthology.orchestra.run.trial023.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-25T08:17:55.893 INFO:teuthology.orchestra.run.trial023.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-25T08:17:55.893 INFO:teuthology.orchestra.run.trial023.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-25T08:17:55.893 INFO:teuthology.orchestra.run.trial023.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-25T08:17:55.893 INFO:teuthology.orchestra.run.trial023.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-25T08:17:55.893 INFO:teuthology.orchestra.run.trial023.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-25T08:17:55.894 INFO:teuthology.orchestra.run.trial023.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-25T08:17:55.894 INFO:teuthology.orchestra.run.trial023.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-25T08:17:55.894 INFO:teuthology.orchestra.run.trial023.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-25T08:17:55.894 INFO:teuthology.orchestra.run.trial023.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-25T08:17:55.894 INFO:teuthology.orchestra.run.trial023.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-25T08:17:55.894 INFO:teuthology.orchestra.run.trial023.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-25T08:17:55.894 INFO:teuthology.orchestra.run.trial023.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-25T08:17:55.894 INFO:teuthology.orchestra.run.trial023.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-25T08:17:55.894 INFO:teuthology.orchestra.run.trial023.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-25T08:17:55.894 INFO:teuthology.orchestra.run.trial023.stdout: python3-ply-3.11-14.el9.noarch 2026-02-25T08:17:55.894 INFO:teuthology.orchestra.run.trial023.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-25T08:17:55.894 INFO:teuthology.orchestra.run.trial023.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-25T08:17:55.894 INFO:teuthology.orchestra.run.trial023.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-25T08:17:55.894 INFO:teuthology.orchestra.run.trial023.stdout: python3-pyasn1-0.4.8-7.el9.noarch 2026-02-25T08:17:55.894 INFO:teuthology.orchestra.run.trial023.stdout: python3-pyasn1-modules-0.4.8-7.el9.noarch 2026-02-25T08:17:55.894 INFO:teuthology.orchestra.run.trial023.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-25T08:17:55.894 INFO:teuthology.orchestra.run.trial023.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-25T08:17:55.894 INFO:teuthology.orchestra.run.trial023.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-25T08:17:55.894 INFO:teuthology.orchestra.run.trial023.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-25T08:17:55.894 INFO:teuthology.orchestra.run.trial023.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-25T08:17:55.894 INFO:teuthology.orchestra.run.trial023.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-25T08:17:55.894 INFO:teuthology.orchestra.run.trial023.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-25T08:17:55.894 INFO:teuthology.orchestra.run.trial023.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-25T08:17:55.894 INFO:teuthology.orchestra.run.trial023.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-25T08:17:55.895 INFO:teuthology.orchestra.run.trial023.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-25T08:17:55.895 INFO:teuthology.orchestra.run.trial023.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-25T08:17:55.895 INFO:teuthology.orchestra.run.trial023.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-25T08:17:55.895 INFO:teuthology.orchestra.run.trial023.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-25T08:17:55.895 INFO:teuthology.orchestra.run.trial023.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-25T08:17:55.895 INFO:teuthology.orchestra.run.trial023.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-25T08:17:55.895 INFO:teuthology.orchestra.run.trial023.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-25T08:17:55.895 INFO:teuthology.orchestra.run.trial023.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-25T08:17:55.895 INFO:teuthology.orchestra.run.trial023.stdout: unzip-6.0-59.el9.x86_64 2026-02-25T08:17:55.895 INFO:teuthology.orchestra.run.trial023.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-25T08:17:55.895 INFO:teuthology.orchestra.run.trial023.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-25T08:17:55.895 INFO:teuthology.orchestra.run.trial023.stdout: zip-3.0-35.el9.x86_64 2026-02-25T08:17:55.895 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:17:55.895 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:17:55.961 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install ceph-base 2026-02-25T08:17:56.286 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:10:29 ago on Wed 25 Feb 2026 08:07:27 AM UTC. 2026-02-25T08:17:56.353 INFO:teuthology.orchestra.run.trial023.stdout:Package ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:17:56.372 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:17:56.372 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:17:56.372 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:17:56.394 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install cephadm 2026-02-25T08:17:56.714 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:10:29 ago on Wed 25 Feb 2026 08:07:27 AM UTC. 2026-02-25T08:17:56.781 INFO:teuthology.orchestra.run.trial023.stdout:Package cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:17:56.800 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:17:56.800 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:17:56.801 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:17:56.822 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install ceph-immutable-object-cache 2026-02-25T08:17:57.140 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:10:30 ago on Wed 25 Feb 2026 08:07:27 AM UTC. 2026-02-25T08:17:57.224 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:17:57.225 INFO:teuthology.orchestra.run.trial023.stdout:================================================================================ 2026-02-25T08:17:57.225 INFO:teuthology.orchestra.run.trial023.stdout: Package Arch Version Repo Size 2026-02-25T08:17:57.225 INFO:teuthology.orchestra.run.trial023.stdout:================================================================================ 2026-02-25T08:17:57.225 INFO:teuthology.orchestra.run.trial023.stdout:Installing: 2026-02-25T08:17:57.225 INFO:teuthology.orchestra.run.trial023.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 149 k 2026-02-25T08:17:57.225 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:17:57.226 INFO:teuthology.orchestra.run.trial023.stdout:Transaction Summary 2026-02-25T08:17:57.226 INFO:teuthology.orchestra.run.trial023.stdout:================================================================================ 2026-02-25T08:17:57.226 INFO:teuthology.orchestra.run.trial023.stdout:Install 1 Package 2026-02-25T08:17:57.226 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:17:57.226 INFO:teuthology.orchestra.run.trial023.stdout:Total download size: 149 k 2026-02-25T08:17:57.226 INFO:teuthology.orchestra.run.trial023.stdout:Installed size: 455 k 2026-02-25T08:17:57.226 INFO:teuthology.orchestra.run.trial023.stdout:Downloading Packages: 2026-02-25T08:17:58.296 INFO:teuthology.orchestra.run.trial023.stdout:ceph-immutable-object-cache-20.3.0-5338.g6aeb69 139 kB/s | 149 kB 00:01 2026-02-25T08:17:58.297 INFO:teuthology.orchestra.run.trial023.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:17:58.297 INFO:teuthology.orchestra.run.trial023.stdout:Total 139 kB/s | 149 kB 00:01 2026-02-25T08:17:58.297 INFO:teuthology.orchestra.run.trial023.stdout:Running transaction check 2026-02-25T08:17:58.301 INFO:teuthology.orchestra.run.trial023.stdout:Transaction check succeeded. 2026-02-25T08:17:58.301 INFO:teuthology.orchestra.run.trial023.stdout:Running transaction test 2026-02-25T08:17:58.333 INFO:teuthology.orchestra.run.trial023.stdout:Transaction test succeeded. 2026-02-25T08:17:58.333 INFO:teuthology.orchestra.run.trial023.stdout:Running transaction 2026-02-25T08:17:58.388 INFO:teuthology.orchestra.run.trial023.stdout: Preparing : 1/1 2026-02-25T08:17:58.390 INFO:teuthology.orchestra.run.trial023.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975. 1/1 2026-02-25T08:17:58.413 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975. 1/1 2026-02-25T08:17:58.413 INFO:teuthology.orchestra.run.trial023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:17:58.413 INFO:teuthology.orchestra.run.trial023.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-25T08:17:58.413 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:17:58.745 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975. 1/1 2026-02-25T08:17:58.745 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:17:58.745 INFO:teuthology.orchestra.run.trial023.stdout:Installed: 2026-02-25T08:17:58.745 INFO:teuthology.orchestra.run.trial023.stdout: ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:17:58.745 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:17:58.745 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:17:58.773 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install ceph-mgr 2026-02-25T08:17:59.095 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:10:32 ago on Wed 25 Feb 2026 08:07:27 AM UTC. 2026-02-25T08:17:59.161 INFO:teuthology.orchestra.run.trial023.stdout:Package ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:17:59.179 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:17:59.180 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:17:59.180 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:17:59.202 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install ceph-mgr-dashboard 2026-02-25T08:17:59.520 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:10:32 ago on Wed 25 Feb 2026 08:07:27 AM UTC. 2026-02-25T08:17:59.585 INFO:teuthology.orchestra.run.trial023.stdout:Package ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:17:59.603 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:17:59.604 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:17:59.604 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:17:59.626 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-25T08:17:59.937 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:10:32 ago on Wed 25 Feb 2026 08:07:27 AM UTC. 2026-02-25T08:18:00.002 INFO:teuthology.orchestra.run.trial023.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:18:00.020 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:18:00.021 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:18:00.021 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:18:00.042 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install ceph-mgr-rook 2026-02-25T08:18:00.361 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:10:33 ago on Wed 25 Feb 2026 08:07:27 AM UTC. 2026-02-25T08:18:00.428 INFO:teuthology.orchestra.run.trial023.stdout:Package ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:18:00.446 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:18:00.447 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:18:00.447 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:18:00.469 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install ceph-mgr-cephadm 2026-02-25T08:18:00.789 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:10:33 ago on Wed 25 Feb 2026 08:07:27 AM UTC. 2026-02-25T08:18:00.855 INFO:teuthology.orchestra.run.trial023.stdout:Package ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:18:00.874 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:18:00.874 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:18:00.875 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:18:00.897 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install ceph-osd 2026-02-25T08:18:01.215 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:10:34 ago on Wed 25 Feb 2026 08:07:27 AM UTC. 2026-02-25T08:18:01.280 INFO:teuthology.orchestra.run.trial023.stdout:Package ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:18:01.299 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:18:01.300 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:18:01.300 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:18:01.321 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install ceph-osd-classic 2026-02-25T08:18:01.633 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:10:34 ago on Wed 25 Feb 2026 08:07:27 AM UTC. 2026-02-25T08:18:01.698 INFO:teuthology.orchestra.run.trial023.stdout:Package ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:18:01.717 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:18:01.717 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:18:01.717 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:18:01.740 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install ceph-fuse 2026-02-25T08:18:02.057 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:10:35 ago on Wed 25 Feb 2026 08:07:27 AM UTC. 2026-02-25T08:18:02.144 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:18:02.144 INFO:teuthology.orchestra.run.trial023.stdout:================================================================================ 2026-02-25T08:18:02.144 INFO:teuthology.orchestra.run.trial023.stdout: Package Arch Version Repo Size 2026-02-25T08:18:02.144 INFO:teuthology.orchestra.run.trial023.stdout:================================================================================ 2026-02-25T08:18:02.145 INFO:teuthology.orchestra.run.trial023.stdout:Installing: 2026-02-25T08:18:02.145 INFO:teuthology.orchestra.run.trial023.stdout: ceph-fuse x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 914 k 2026-02-25T08:18:02.145 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:18:02.145 INFO:teuthology.orchestra.run.trial023.stdout:Transaction Summary 2026-02-25T08:18:02.145 INFO:teuthology.orchestra.run.trial023.stdout:================================================================================ 2026-02-25T08:18:02.145 INFO:teuthology.orchestra.run.trial023.stdout:Install 1 Package 2026-02-25T08:18:02.145 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:18:02.145 INFO:teuthology.orchestra.run.trial023.stdout:Total download size: 914 k 2026-02-25T08:18:02.145 INFO:teuthology.orchestra.run.trial023.stdout:Installed size: 2.9 M 2026-02-25T08:18:02.145 INFO:teuthology.orchestra.run.trial023.stdout:Downloading Packages: 2026-02-25T08:18:04.302 INFO:teuthology.orchestra.run.trial023.stdout:ceph-fuse-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 423 kB/s | 914 kB 00:02 2026-02-25T08:18:04.302 INFO:teuthology.orchestra.run.trial023.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:18:04.302 INFO:teuthology.orchestra.run.trial023.stdout:Total 423 kB/s | 914 kB 00:02 2026-02-25T08:18:04.302 INFO:teuthology.orchestra.run.trial023.stdout:Running transaction check 2026-02-25T08:18:04.307 INFO:teuthology.orchestra.run.trial023.stdout:Transaction check succeeded. 2026-02-25T08:18:04.308 INFO:teuthology.orchestra.run.trial023.stdout:Running transaction test 2026-02-25T08:18:04.341 INFO:teuthology.orchestra.run.trial023.stdout:Transaction test succeeded. 2026-02-25T08:18:04.341 INFO:teuthology.orchestra.run.trial023.stdout:Running transaction 2026-02-25T08:18:04.432 INFO:teuthology.orchestra.run.trial023.stdout: Preparing : 1/1 2026-02-25T08:18:04.450 INFO:teuthology.orchestra.run.trial023.stdout: Installing : ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:18:04.707 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:18:04.769 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:18:04.769 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:18:04.769 INFO:teuthology.orchestra.run.trial023.stdout:Installed: 2026-02-25T08:18:04.769 INFO:teuthology.orchestra.run.trial023.stdout: ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:18:04.769 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:18:04.769 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:18:04.798 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install ceph-volume 2026-02-25T08:18:05.114 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:10:38 ago on Wed 25 Feb 2026 08:07:27 AM UTC. 2026-02-25T08:18:05.180 INFO:teuthology.orchestra.run.trial023.stdout:Package ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:18:05.199 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:18:05.199 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:18:05.199 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:18:05.221 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install librados-devel 2026-02-25T08:18:05.533 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:10:38 ago on Wed 25 Feb 2026 08:07:27 AM UTC. 2026-02-25T08:18:05.619 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:18:05.620 INFO:teuthology.orchestra.run.trial023.stdout:================================================================================ 2026-02-25T08:18:05.620 INFO:teuthology.orchestra.run.trial023.stdout: Package Arch Version Repo Size 2026-02-25T08:18:05.620 INFO:teuthology.orchestra.run.trial023.stdout:================================================================================ 2026-02-25T08:18:05.620 INFO:teuthology.orchestra.run.trial023.stdout:Installing: 2026-02-25T08:18:05.620 INFO:teuthology.orchestra.run.trial023.stdout: librados-devel x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 124 k 2026-02-25T08:18:05.620 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:18:05.620 INFO:teuthology.orchestra.run.trial023.stdout:Transaction Summary 2026-02-25T08:18:05.620 INFO:teuthology.orchestra.run.trial023.stdout:================================================================================ 2026-02-25T08:18:05.620 INFO:teuthology.orchestra.run.trial023.stdout:Install 1 Package 2026-02-25T08:18:05.620 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:18:05.620 INFO:teuthology.orchestra.run.trial023.stdout:Total download size: 124 k 2026-02-25T08:18:05.620 INFO:teuthology.orchestra.run.trial023.stdout:Installed size: 434 k 2026-02-25T08:18:05.620 INFO:teuthology.orchestra.run.trial023.stdout:Downloading Packages: 2026-02-25T08:18:06.334 INFO:teuthology.orchestra.run.trial023.stdout:librados-devel-20.3.0-5338.g6aeb6975.el9.x86_64 173 kB/s | 124 kB 00:00 2026-02-25T08:18:06.334 INFO:teuthology.orchestra.run.trial023.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:18:06.334 INFO:teuthology.orchestra.run.trial023.stdout:Total 173 kB/s | 124 kB 00:00 2026-02-25T08:18:06.335 INFO:teuthology.orchestra.run.trial023.stdout:Running transaction check 2026-02-25T08:18:06.337 INFO:teuthology.orchestra.run.trial023.stdout:Transaction check succeeded. 2026-02-25T08:18:06.337 INFO:teuthology.orchestra.run.trial023.stdout:Running transaction test 2026-02-25T08:18:06.369 INFO:teuthology.orchestra.run.trial023.stdout:Transaction test succeeded. 2026-02-25T08:18:06.369 INFO:teuthology.orchestra.run.trial023.stdout:Running transaction 2026-02-25T08:18:06.423 INFO:teuthology.orchestra.run.trial023.stdout: Preparing : 1/1 2026-02-25T08:18:06.441 INFO:teuthology.orchestra.run.trial023.stdout: Installing : librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:18:06.523 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:18:06.585 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:18:06.586 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:18:06.586 INFO:teuthology.orchestra.run.trial023.stdout:Installed: 2026-02-25T08:18:06.586 INFO:teuthology.orchestra.run.trial023.stdout: librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:18:06.586 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:18:06.586 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:18:06.615 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install libcephfs2 2026-02-25T08:18:06.929 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:10:39 ago on Wed 25 Feb 2026 08:07:27 AM UTC. 2026-02-25T08:18:06.995 INFO:teuthology.orchestra.run.trial023.stdout:Package libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:18:07.013 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:18:07.014 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:18:07.014 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:18:07.037 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install libcephfs-devel 2026-02-25T08:18:07.353 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:10:40 ago on Wed 25 Feb 2026 08:07:27 AM UTC. 2026-02-25T08:18:07.439 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:18:07.439 INFO:teuthology.orchestra.run.trial023.stdout:================================================================================ 2026-02-25T08:18:07.439 INFO:teuthology.orchestra.run.trial023.stdout: Package Arch Version Repo Size 2026-02-25T08:18:07.439 INFO:teuthology.orchestra.run.trial023.stdout:================================================================================ 2026-02-25T08:18:07.439 INFO:teuthology.orchestra.run.trial023.stdout:Installing: 2026-02-25T08:18:07.439 INFO:teuthology.orchestra.run.trial023.stdout: libcephfs-devel x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 37 k 2026-02-25T08:18:07.439 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:18:07.439 INFO:teuthology.orchestra.run.trial023.stdout:Transaction Summary 2026-02-25T08:18:07.439 INFO:teuthology.orchestra.run.trial023.stdout:================================================================================ 2026-02-25T08:18:07.439 INFO:teuthology.orchestra.run.trial023.stdout:Install 1 Package 2026-02-25T08:18:07.439 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:18:07.440 INFO:teuthology.orchestra.run.trial023.stdout:Total download size: 37 k 2026-02-25T08:18:07.440 INFO:teuthology.orchestra.run.trial023.stdout:Installed size: 171 k 2026-02-25T08:18:07.440 INFO:teuthology.orchestra.run.trial023.stdout:Downloading Packages: 2026-02-25T08:18:07.800 INFO:teuthology.orchestra.run.trial023.stdout:libcephfs-devel-20.3.0-5338.g6aeb6975.el9.x86_6 104 kB/s | 37 kB 00:00 2026-02-25T08:18:07.800 INFO:teuthology.orchestra.run.trial023.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:18:07.800 INFO:teuthology.orchestra.run.trial023.stdout:Total 103 kB/s | 37 kB 00:00 2026-02-25T08:18:07.800 INFO:teuthology.orchestra.run.trial023.stdout:Running transaction check 2026-02-25T08:18:07.803 INFO:teuthology.orchestra.run.trial023.stdout:Transaction check succeeded. 2026-02-25T08:18:07.803 INFO:teuthology.orchestra.run.trial023.stdout:Running transaction test 2026-02-25T08:18:07.805 INFO:teuthology.orchestra.run.trial023.stdout:Transaction test succeeded. 2026-02-25T08:18:07.805 INFO:teuthology.orchestra.run.trial023.stdout:Running transaction 2026-02-25T08:18:07.827 INFO:teuthology.orchestra.run.trial023.stdout: Preparing : 1/1 2026-02-25T08:18:07.845 INFO:teuthology.orchestra.run.trial023.stdout: Installing : libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:18:07.909 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:18:07.971 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:18:07.972 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:18:07.972 INFO:teuthology.orchestra.run.trial023.stdout:Installed: 2026-02-25T08:18:07.972 INFO:teuthology.orchestra.run.trial023.stdout: libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:18:07.972 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:18:07.972 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:18:08.000 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install librados2 2026-02-25T08:18:08.318 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:10:41 ago on Wed 25 Feb 2026 08:07:27 AM UTC. 2026-02-25T08:18:08.383 INFO:teuthology.orchestra.run.trial023.stdout:Package librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:18:08.401 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:18:08.401 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:18:08.402 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:18:08.424 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install librbd1 2026-02-25T08:18:08.745 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:10:41 ago on Wed 25 Feb 2026 08:07:27 AM UTC. 2026-02-25T08:18:08.812 INFO:teuthology.orchestra.run.trial023.stdout:Package librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:18:08.830 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:18:08.830 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:18:08.830 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:18:08.851 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install python3-rados 2026-02-25T08:18:09.164 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:10:42 ago on Wed 25 Feb 2026 08:07:27 AM UTC. 2026-02-25T08:18:09.230 INFO:teuthology.orchestra.run.trial023.stdout:Package python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:18:09.247 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:18:09.248 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:18:09.248 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:18:09.269 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install python3-rgw 2026-02-25T08:18:09.583 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:10:42 ago on Wed 25 Feb 2026 08:07:27 AM UTC. 2026-02-25T08:18:09.648 INFO:teuthology.orchestra.run.trial023.stdout:Package python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:18:09.666 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:18:09.666 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:18:09.667 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:18:09.688 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install python3-cephfs 2026-02-25T08:18:10.003 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:10:43 ago on Wed 25 Feb 2026 08:07:27 AM UTC. 2026-02-25T08:18:10.069 INFO:teuthology.orchestra.run.trial023.stdout:Package python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:18:10.086 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:18:10.087 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:18:10.087 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:18:10.109 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install python3-rbd 2026-02-25T08:18:10.424 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:10:43 ago on Wed 25 Feb 2026 08:07:27 AM UTC. 2026-02-25T08:18:10.490 INFO:teuthology.orchestra.run.trial023.stdout:Package python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:18:10.507 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:18:10.508 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:18:10.508 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:18:10.529 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install rbd-fuse 2026-02-25T08:18:10.849 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:10:43 ago on Wed 25 Feb 2026 08:07:27 AM UTC. 2026-02-25T08:18:10.932 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:18:10.932 INFO:teuthology.orchestra.run.trial023.stdout:================================================================================ 2026-02-25T08:18:10.932 INFO:teuthology.orchestra.run.trial023.stdout: Package Architecture Version Repository Size 2026-02-25T08:18:10.932 INFO:teuthology.orchestra.run.trial023.stdout:================================================================================ 2026-02-25T08:18:10.932 INFO:teuthology.orchestra.run.trial023.stdout:Installing: 2026-02-25T08:18:10.932 INFO:teuthology.orchestra.run.trial023.stdout: rbd-fuse x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 86 k 2026-02-25T08:18:10.932 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:18:10.932 INFO:teuthology.orchestra.run.trial023.stdout:Transaction Summary 2026-02-25T08:18:10.933 INFO:teuthology.orchestra.run.trial023.stdout:================================================================================ 2026-02-25T08:18:10.933 INFO:teuthology.orchestra.run.trial023.stdout:Install 1 Package 2026-02-25T08:18:10.933 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:18:10.933 INFO:teuthology.orchestra.run.trial023.stdout:Total download size: 86 k 2026-02-25T08:18:10.933 INFO:teuthology.orchestra.run.trial023.stdout:Installed size: 236 k 2026-02-25T08:18:10.933 INFO:teuthology.orchestra.run.trial023.stdout:Downloading Packages: 2026-02-25T08:18:11.711 INFO:teuthology.orchestra.run.trial023.stdout:rbd-fuse-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 111 kB/s | 86 kB 00:00 2026-02-25T08:18:11.711 INFO:teuthology.orchestra.run.trial023.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:18:11.712 INFO:teuthology.orchestra.run.trial023.stdout:Total 111 kB/s | 86 kB 00:00 2026-02-25T08:18:11.712 INFO:teuthology.orchestra.run.trial023.stdout:Running transaction check 2026-02-25T08:18:11.716 INFO:teuthology.orchestra.run.trial023.stdout:Transaction check succeeded. 2026-02-25T08:18:11.716 INFO:teuthology.orchestra.run.trial023.stdout:Running transaction test 2026-02-25T08:18:11.747 INFO:teuthology.orchestra.run.trial023.stdout:Transaction test succeeded. 2026-02-25T08:18:11.747 INFO:teuthology.orchestra.run.trial023.stdout:Running transaction 2026-02-25T08:18:11.799 INFO:teuthology.orchestra.run.trial023.stdout: Preparing : 1/1 2026-02-25T08:18:11.817 INFO:teuthology.orchestra.run.trial023.stdout: Installing : rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:18:11.904 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:18:11.968 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:18:11.968 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:18:11.968 INFO:teuthology.orchestra.run.trial023.stdout:Installed: 2026-02-25T08:18:11.968 INFO:teuthology.orchestra.run.trial023.stdout: rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:18:11.968 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:18:11.968 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:18:11.997 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install rbd-mirror 2026-02-25T08:18:12.312 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:10:45 ago on Wed 25 Feb 2026 08:07:27 AM UTC. 2026-02-25T08:18:12.395 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:18:12.395 INFO:teuthology.orchestra.run.trial023.stdout:================================================================================ 2026-02-25T08:18:12.395 INFO:teuthology.orchestra.run.trial023.stdout: Package Arch Version Repo Size 2026-02-25T08:18:12.396 INFO:teuthology.orchestra.run.trial023.stdout:================================================================================ 2026-02-25T08:18:12.396 INFO:teuthology.orchestra.run.trial023.stdout:Installing: 2026-02-25T08:18:12.396 INFO:teuthology.orchestra.run.trial023.stdout: rbd-mirror x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 2.6 M 2026-02-25T08:18:12.396 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:18:12.396 INFO:teuthology.orchestra.run.trial023.stdout:Transaction Summary 2026-02-25T08:18:12.396 INFO:teuthology.orchestra.run.trial023.stdout:================================================================================ 2026-02-25T08:18:12.396 INFO:teuthology.orchestra.run.trial023.stdout:Install 1 Package 2026-02-25T08:18:12.396 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:18:12.396 INFO:teuthology.orchestra.run.trial023.stdout:Total download size: 2.6 M 2026-02-25T08:18:12.396 INFO:teuthology.orchestra.run.trial023.stdout:Installed size: 11 M 2026-02-25T08:18:12.396 INFO:teuthology.orchestra.run.trial023.stdout:Downloading Packages: 2026-02-25T08:18:20.716 INFO:teuthology.orchestra.run.trial023.stdout:rbd-mirror-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 319 kB/s | 2.6 MB 00:08 2026-02-25T08:18:20.717 INFO:teuthology.orchestra.run.trial023.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:18:20.718 INFO:teuthology.orchestra.run.trial023.stdout:Total 319 kB/s | 2.6 MB 00:08 2026-02-25T08:18:20.718 INFO:teuthology.orchestra.run.trial023.stdout:Running transaction check 2026-02-25T08:18:20.723 INFO:teuthology.orchestra.run.trial023.stdout:Transaction check succeeded. 2026-02-25T08:18:20.723 INFO:teuthology.orchestra.run.trial023.stdout:Running transaction test 2026-02-25T08:18:20.761 INFO:teuthology.orchestra.run.trial023.stdout:Transaction test succeeded. 2026-02-25T08:18:20.762 INFO:teuthology.orchestra.run.trial023.stdout:Running transaction 2026-02-25T08:18:20.927 INFO:teuthology.orchestra.run.trial023.stdout: Preparing : 1/1 2026-02-25T08:18:20.929 INFO:teuthology.orchestra.run.trial023.stdout: Installing : rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:18:20.952 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:18:20.952 INFO:teuthology.orchestra.run.trial023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:18:20.952 INFO:teuthology.orchestra.run.trial023.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-25T08:18:20.952 INFO:teuthology.orchestra.run.trial023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-25T08:18:20.953 INFO:teuthology.orchestra.run.trial023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-25T08:18:20.953 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:18:21.256 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:18:21.256 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:18:21.257 INFO:teuthology.orchestra.run.trial023.stdout:Installed: 2026-02-25T08:18:21.257 INFO:teuthology.orchestra.run.trial023.stdout: rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:18:21.257 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:18:21.257 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:18:21.285 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install rbd-nbd 2026-02-25T08:18:21.603 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:10:54 ago on Wed 25 Feb 2026 08:07:27 AM UTC. 2026-02-25T08:18:21.687 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:18:21.688 INFO:teuthology.orchestra.run.trial023.stdout:================================================================================ 2026-02-25T08:18:21.688 INFO:teuthology.orchestra.run.trial023.stdout: Package Architecture Version Repository Size 2026-02-25T08:18:21.688 INFO:teuthology.orchestra.run.trial023.stdout:================================================================================ 2026-02-25T08:18:21.688 INFO:teuthology.orchestra.run.trial023.stdout:Installing: 2026-02-25T08:18:21.688 INFO:teuthology.orchestra.run.trial023.stdout: rbd-nbd x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 166 k 2026-02-25T08:18:21.688 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:18:21.688 INFO:teuthology.orchestra.run.trial023.stdout:Transaction Summary 2026-02-25T08:18:21.688 INFO:teuthology.orchestra.run.trial023.stdout:================================================================================ 2026-02-25T08:18:21.689 INFO:teuthology.orchestra.run.trial023.stdout:Install 1 Package 2026-02-25T08:18:21.689 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:18:21.689 INFO:teuthology.orchestra.run.trial023.stdout:Total download size: 166 k 2026-02-25T08:18:21.689 INFO:teuthology.orchestra.run.trial023.stdout:Installed size: 496 k 2026-02-25T08:18:21.689 INFO:teuthology.orchestra.run.trial023.stdout:Downloading Packages: 2026-02-25T08:18:22.327 INFO:teuthology.orchestra.run.trial023.stdout:rbd-nbd-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 260 kB/s | 166 kB 00:00 2026-02-25T08:18:22.328 INFO:teuthology.orchestra.run.trial023.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:18:22.328 INFO:teuthology.orchestra.run.trial023.stdout:Total 260 kB/s | 166 kB 00:00 2026-02-25T08:18:22.328 INFO:teuthology.orchestra.run.trial023.stdout:Running transaction check 2026-02-25T08:18:22.332 INFO:teuthology.orchestra.run.trial023.stdout:Transaction check succeeded. 2026-02-25T08:18:22.333 INFO:teuthology.orchestra.run.trial023.stdout:Running transaction test 2026-02-25T08:18:22.364 INFO:teuthology.orchestra.run.trial023.stdout:Transaction test succeeded. 2026-02-25T08:18:22.364 INFO:teuthology.orchestra.run.trial023.stdout:Running transaction 2026-02-25T08:18:22.420 INFO:teuthology.orchestra.run.trial023.stdout: Preparing : 1/1 2026-02-25T08:18:22.437 INFO:teuthology.orchestra.run.trial023.stdout: Installing : rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:18:22.526 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:18:22.589 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:18:22.589 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:18:22.589 INFO:teuthology.orchestra.run.trial023.stdout:Installed: 2026-02-25T08:18:22.589 INFO:teuthology.orchestra.run.trial023.stdout: rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:18:22.589 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:18:22.589 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:18:22.617 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install ceph-osd-crimson 2026-02-25T08:18:22.933 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:10:55 ago on Wed 25 Feb 2026 08:07:27 AM UTC. 2026-02-25T08:18:22.998 INFO:teuthology.orchestra.run.trial023.stdout:Package ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:18:23.016 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:18:23.016 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:18:23.016 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:18:23.038 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install ceph-osd-crimson-debuginfo 2026-02-25T08:18:23.349 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:10:56 ago on Wed 25 Feb 2026 08:07:27 AM UTC. 2026-02-25T08:18:23.433 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:18:23.434 INFO:teuthology.orchestra.run.trial023.stdout:================================================================================ 2026-02-25T08:18:23.434 INFO:teuthology.orchestra.run.trial023.stdout: Package Arch Version Repo Size 2026-02-25T08:18:23.435 INFO:teuthology.orchestra.run.trial023.stdout:================================================================================ 2026-02-25T08:18:23.435 INFO:teuthology.orchestra.run.trial023.stdout:Installing: 2026-02-25T08:18:23.435 INFO:teuthology.orchestra.run.trial023.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 433 M 2026-02-25T08:18:23.435 INFO:teuthology.orchestra.run.trial023.stdout:Installing weak dependencies: 2026-02-25T08:18:23.435 INFO:teuthology.orchestra.run.trial023.stdout: ceph-debugsource x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 12 M 2026-02-25T08:18:23.435 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:18:23.435 INFO:teuthology.orchestra.run.trial023.stdout:Transaction Summary 2026-02-25T08:18:23.435 INFO:teuthology.orchestra.run.trial023.stdout:================================================================================ 2026-02-25T08:18:23.436 INFO:teuthology.orchestra.run.trial023.stdout:Install 2 Packages 2026-02-25T08:18:23.436 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:18:23.436 INFO:teuthology.orchestra.run.trial023.stdout:Total download size: 445 M 2026-02-25T08:18:23.436 INFO:teuthology.orchestra.run.trial023.stdout:Installed size: 2.7 G 2026-02-25T08:18:23.436 INFO:teuthology.orchestra.run.trial023.stdout:Downloading Packages: 2026-02-25T08:19:03.061 INFO:teuthology.orchestra.run.trial023.stdout:(1/2): ceph-debugsource-20.3.0-5338.g6aeb6975.e 311 kB/s | 12 MB 00:39 2026-02-25T08:21:12.460 INFO:teuthology.orchestra.run.trial030.stdout:(96/96): ceph-osd-crimson-20.3.0-5338.g6aeb6975 218 kB/s | 88 MB 06:52 2026-02-25T08:21:12.462 INFO:teuthology.orchestra.run.trial030.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:21:12.462 INFO:teuthology.orchestra.run.trial030.stdout:Total 547 kB/s | 226 MB 07:02 2026-02-25T08:21:12.829 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction check 2026-02-25T08:21:12.866 INFO:teuthology.orchestra.run.trial030.stdout:Transaction check succeeded. 2026-02-25T08:21:12.866 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction test 2026-02-25T08:21:13.669 INFO:teuthology.orchestra.run.trial030.stdout:Transaction test succeeded. 2026-02-25T08:21:13.669 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction 2026-02-25T08:21:14.505 INFO:teuthology.orchestra.run.trial030.stdout: Preparing : 1/1 2026-02-25T08:21:14.507 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/96 2026-02-25T08:21:14.511 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/96 2026-02-25T08:21:14.532 INFO:teuthology.orchestra.run.trial030.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 3/96 2026-02-25T08:21:14.565 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-pyasn1-0.4.8-7.el9.noarch 4/96 2026-02-25T08:21:14.571 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 5/96 2026-02-25T08:21:14.573 INFO:teuthology.orchestra.run.trial030.stdout: Installing : lua-5.4.4-4.el9.x86_64 6/96 2026-02-25T08:21:14.578 INFO:teuthology.orchestra.run.trial030.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 7/96 2026-02-25T08:21:14.601 INFO:teuthology.orchestra.run.trial030.stdout: Installing : unzip-6.0-59.el9.x86_64 8/96 2026-02-25T08:21:14.614 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 9/96 2026-02-25T08:21:14.617 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 10/96 2026-02-25T08:21:14.623 INFO:teuthology.orchestra.run.trial030.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 11/96 2026-02-25T08:21:14.625 INFO:teuthology.orchestra.run.trial030.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 12/96 2026-02-25T08:21:14.636 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 13/96 2026-02-25T08:21:14.642 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 14/96 2026-02-25T08:21:14.666 INFO:teuthology.orchestra.run.trial030.stdout: Installing : zip-3.0-35.el9.x86_64 15/96 2026-02-25T08:21:14.669 INFO:teuthology.orchestra.run.trial030.stdout: Installing : luarocks-3.9.2-5.el9.noarch 16/96 2026-02-25T08:21:14.719 INFO:teuthology.orchestra.run.trial030.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 17/96 2026-02-25T08:21:14.731 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-pyasn1-modules-0.4.8-7.el9.noarch 18/96 2026-02-25T08:21:14.734 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-rsa-4.9-2.el9.noarch 19/96 2026-02-25T08:21:14.738 INFO:teuthology.orchestra.run.trial030.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 20/96 2026-02-25T08:21:14.744 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 21/96 2026-02-25T08:21:14.748 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 22/96 2026-02-25T08:21:14.751 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 23/96 2026-02-25T08:21:14.766 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 24/96 2026-02-25T08:21:14.770 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 25/96 2026-02-25T08:21:14.775 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 26/96 2026-02-25T08:21:14.786 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 27/96 2026-02-25T08:21:14.795 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 28/96 2026-02-25T08:21:14.802 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 29/96 2026-02-25T08:21:14.821 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 30/96 2026-02-25T08:21:14.832 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 31/96 2026-02-25T08:21:14.848 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 32/96 2026-02-25T08:21:14.852 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 33/96 2026-02-25T08:21:14.860 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 34/96 2026-02-25T08:21:14.900 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 35/96 2026-02-25T08:21:15.222 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 36/96 2026-02-25T08:21:15.232 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 37/96 2026-02-25T08:21:15.236 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 38/96 2026-02-25T08:21:15.241 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 39/96 2026-02-25T08:21:15.243 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 40/96 2026-02-25T08:21:15.270 INFO:teuthology.orchestra.run.trial030.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 41/96 2026-02-25T08:21:15.314 INFO:teuthology.orchestra.run.trial030.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 42/96 2026-02-25T08:21:15.323 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 43/96 2026-02-25T08:21:15.328 INFO:teuthology.orchestra.run.trial030.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 44/96 2026-02-25T08:21:15.332 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 45/96 2026-02-25T08:21:15.338 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 46/96 2026-02-25T08:21:15.341 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 47/96 2026-02-25T08:21:15.348 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 48/96 2026-02-25T08:21:15.351 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 49/96 2026-02-25T08:21:15.379 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 50/96 2026-02-25T08:21:15.390 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 51/96 2026-02-25T08:21:15.396 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 52/96 2026-02-25T08:21:15.403 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 53/96 2026-02-25T08:21:15.437 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 54/96 2026-02-25T08:21:15.659 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 55/96 2026-02-25T08:21:15.683 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 56/96 2026-02-25T08:21:15.687 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 57/96 2026-02-25T08:21:15.745 INFO:teuthology.orchestra.run.trial030.stdout: Installing : openblas-0.3.29-1.el9.x86_64 58/96 2026-02-25T08:21:15.747 INFO:teuthology.orchestra.run.trial030.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 59/96 2026-02-25T08:21:15.769 INFO:teuthology.orchestra.run.trial030.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 60/96 2026-02-25T08:21:16.093 INFO:teuthology.orchestra.run.trial030.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 61/96 2026-02-25T08:21:16.166 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 62/96 2026-02-25T08:21:16.834 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 63/96 2026-02-25T08:21:16.851 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 64/96 2026-02-25T08:21:16.861 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 65/96 2026-02-25T08:21:16.868 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-packaging-20.9-5.el9.noarch 66/96 2026-02-25T08:21:16.881 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-ply-3.11-14.el9.noarch 67/96 2026-02-25T08:21:16.897 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 68/96 2026-02-25T08:21:16.972 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 69/96 2026-02-25T08:21:16.981 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 70/96 2026-02-25T08:21:17.004 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 71/96 2026-02-25T08:21:17.033 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 72/96 2026-02-25T08:21:17.086 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 73/96 2026-02-25T08:21:17.091 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 74/96 2026-02-25T08:21:17.095 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 75/96 2026-02-25T08:21:17.096 INFO:teuthology.orchestra.run.trial030.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 76/96 2026-02-25T08:21:17.114 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-25T08:21:17.115 INFO:teuthology.orchestra.run.trial030.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-25T08:21:17.115 INFO:teuthology.orchestra.run.trial030.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-25T08:21:17.115 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:17.122 INFO:teuthology.orchestra.run.trial030.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-25T08:21:17.147 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-25T08:21:17.147 INFO:teuthology.orchestra.run.trial030.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-25T08:21:17.148 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:17.181 INFO:teuthology.orchestra.run.trial030.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 78/96 2026-02-25T08:21:17.780 INFO:teuthology.orchestra.run.trial030.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 79/96 2026-02-25T08:21:23.260 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_6 80/96 2026-02-25T08:21:23.324 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_6 81/96 2026-02-25T08:21:23.326 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 82/96 2026-02-25T08:21:23.340 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 82/96 2026-02-25T08:21:23.340 INFO:teuthology.orchestra.run.trial030.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:21:23.340 INFO:teuthology.orchestra.run.trial030.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-25T08:21:23.340 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:23.351 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 83/96 2026-02-25T08:21:23.375 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 83/96 2026-02-25T08:21:23.375 INFO:teuthology.orchestra.run.trial030.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:21:23.375 INFO:teuthology.orchestra.run.trial030.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-25T08:21:23.375 INFO:teuthology.orchestra.run.trial030.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-25T08:21:23.375 INFO:teuthology.orchestra.run.trial030.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-25T08:21:23.375 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:23.433 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 84/96 2026-02-25T08:21:23.435 INFO:teuthology.orchestra.run.trial030.stdout: Installing : cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 84/96 2026-02-25T08:21:23.444 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5338.g6aeb6975.el9 85/96 2026-02-25T08:21:23.468 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5338.g6aeb6975.el 86/96 2026-02-25T08:21:23.470 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarc 87/96 2026-02-25T08:21:26.403 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarc 87/96 2026-02-25T08:21:26.410 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noa 88/96 2026-02-25T08:21:26.874 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noa 88/96 2026-02-25T08:21:26.875 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6 89/96 2026-02-25T08:21:26.886 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6 89/96 2026-02-25T08:21:26.887 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noa 90/96 2026-02-25T08:21:26.948 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noa 90/96 2026-02-25T08:21:26.989 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5338.g6aeb6975.el9. 91/96 2026-02-25T08:21:26.990 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 92/96 2026-02-25T08:21:27.010 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 92/96 2026-02-25T08:21:27.010 INFO:teuthology.orchestra.run.trial030.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:21:27.010 INFO:teuthology.orchestra.run.trial030.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-25T08:21:27.010 INFO:teuthology.orchestra.run.trial030.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-25T08:21:27.010 INFO:teuthology.orchestra.run.trial030.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-25T08:21:27.011 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:27.023 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 93/96 2026-02-25T08:21:27.241 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 93/96 2026-02-25T08:21:27.242 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 94/96 2026-02-25T08:21:27.262 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 94/96 2026-02-25T08:21:27.263 INFO:teuthology.orchestra.run.trial030.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:21:27.263 INFO:teuthology.orchestra.run.trial030.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-25T08:21:27.263 INFO:teuthology.orchestra.run.trial030.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-25T08:21:27.263 INFO:teuthology.orchestra.run.trial030.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-25T08:21:27.263 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:27.346 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 95/96 2026-02-25T08:21:27.366 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 95/96 2026-02-25T08:21:27.366 INFO:teuthology.orchestra.run.trial030.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:21:27.366 INFO:teuthology.orchestra.run.trial030.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-25T08:21:27.367 INFO:teuthology.orchestra.run.trial030.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-25T08:21:27.367 INFO:teuthology.orchestra.run.trial030.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-25T08:21:27.367 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:27.367 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 96/96 2026-02-25T08:21:27.374 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_6 96/96 2026-02-25T08:21:27.398 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_6 96/96 2026-02-25T08:21:28.768 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 96/96 2026-02-25T08:21:28.768 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/96 2026-02-25T08:21:28.768 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/96 2026-02-25T08:21:28.768 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/96 2026-02-25T08:21:28.768 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 4/96 2026-02-25T08:21:28.768 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 5/96 2026-02-25T08:21:28.768 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_6 6/96 2026-02-25T08:21:28.769 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_6 7/96 2026-02-25T08:21:28.769 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5338.g6aeb6975.el 8/96 2026-02-25T08:21:28.769 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarc 9/96 2026-02-25T08:21:28.769 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noa 10/96 2026-02-25T08:21:28.769 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6 11/96 2026-02-25T08:21:28.769 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noa 12/96 2026-02-25T08:21:28.769 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5338.g6aeb6975.el9. 13/96 2026-02-25T08:21:28.769 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 14/96 2026-02-25T08:21:28.769 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5338.g6aeb6975.el9 15/96 2026-02-25T08:21:28.769 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 16/96 2026-02-25T08:21:28.769 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 17/96 2026-02-25T08:21:28.769 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 18/96 2026-02-25T08:21:28.769 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 19/96 2026-02-25T08:21:28.770 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 20/96 2026-02-25T08:21:28.770 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 21/96 2026-02-25T08:21:28.770 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 22/96 2026-02-25T08:21:28.770 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 23/96 2026-02-25T08:21:28.770 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 24/96 2026-02-25T08:21:28.770 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-ply-3.11-14.el9.noarch 25/96 2026-02-25T08:21:28.770 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 26/96 2026-02-25T08:21:28.770 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 27/96 2026-02-25T08:21:28.770 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 28/96 2026-02-25T08:21:28.770 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 29/96 2026-02-25T08:21:28.770 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : unzip-6.0-59.el9.x86_64 30/96 2026-02-25T08:21:28.770 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : zip-3.0-35.el9.x86_64 31/96 2026-02-25T08:21:28.771 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 32/96 2026-02-25T08:21:28.771 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 33/96 2026-02-25T08:21:28.771 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 34/96 2026-02-25T08:21:28.771 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 35/96 2026-02-25T08:21:28.771 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : lua-5.4.4-4.el9.x86_64 36/96 2026-02-25T08:21:28.771 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 37/96 2026-02-25T08:21:28.771 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 38/96 2026-02-25T08:21:28.771 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 39/96 2026-02-25T08:21:28.771 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 40/96 2026-02-25T08:21:28.771 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 41/96 2026-02-25T08:21:28.771 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 42/96 2026-02-25T08:21:28.771 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 43/96 2026-02-25T08:21:28.771 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 44/96 2026-02-25T08:21:28.772 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 45/96 2026-02-25T08:21:28.772 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 46/96 2026-02-25T08:21:28.772 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 47/96 2026-02-25T08:21:28.772 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 48/96 2026-02-25T08:21:28.772 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 49/96 2026-02-25T08:21:28.772 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-pyasn1-0.4.8-7.el9.noarch 50/96 2026-02-25T08:21:28.772 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-pyasn1-modules-0.4.8-7.el9.noarch 51/96 2026-02-25T08:21:28.772 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 52/96 2026-02-25T08:21:28.772 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 53/96 2026-02-25T08:21:28.772 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 54/96 2026-02-25T08:21:28.772 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 55/96 2026-02-25T08:21:28.772 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 56/96 2026-02-25T08:21:28.773 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 57/96 2026-02-25T08:21:28.773 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 58/96 2026-02-25T08:21:28.773 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 59/96 2026-02-25T08:21:28.773 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 60/96 2026-02-25T08:21:28.773 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 61/96 2026-02-25T08:21:28.773 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 62/96 2026-02-25T08:21:28.773 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 63/96 2026-02-25T08:21:28.773 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 64/96 2026-02-25T08:21:28.773 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 65/96 2026-02-25T08:21:28.773 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 66/96 2026-02-25T08:21:28.773 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 67/96 2026-02-25T08:21:28.773 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 68/96 2026-02-25T08:21:28.773 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 69/96 2026-02-25T08:21:28.774 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 70/96 2026-02-25T08:21:28.774 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 71/96 2026-02-25T08:21:28.774 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 72/96 2026-02-25T08:21:28.774 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 73/96 2026-02-25T08:21:28.774 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 74/96 2026-02-25T08:21:28.774 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 75/96 2026-02-25T08:21:28.774 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 76/96 2026-02-25T08:21:28.774 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 77/96 2026-02-25T08:21:28.774 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 78/96 2026-02-25T08:21:28.774 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 79/96 2026-02-25T08:21:28.774 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 80/96 2026-02-25T08:21:28.774 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 81/96 2026-02-25T08:21:28.774 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/96 2026-02-25T08:21:28.774 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/96 2026-02-25T08:21:28.775 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/96 2026-02-25T08:21:28.775 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 85/96 2026-02-25T08:21:28.775 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/96 2026-02-25T08:21:28.775 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 87/96 2026-02-25T08:21:28.775 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 88/96 2026-02-25T08:21:28.775 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 89/96 2026-02-25T08:21:28.775 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 90/96 2026-02-25T08:21:28.775 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 91/96 2026-02-25T08:21:28.775 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 92/96 2026-02-25T08:21:28.775 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 93/96 2026-02-25T08:21:28.775 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 94/96 2026-02-25T08:21:28.775 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 95/96 2026-02-25T08:21:28.838 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 96/96 2026-02-25T08:21:28.838 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:28.838 INFO:teuthology.orchestra.run.trial030.stdout:Installed: 2026-02-25T08:21:28.838 INFO:teuthology.orchestra.run.trial030.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-25T08:21:28.838 INFO:teuthology.orchestra.run.trial030.stdout: ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:21:28.838 INFO:teuthology.orchestra.run.trial030.stdout: ceph-grafana-dashboards-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:21:28.839 INFO:teuthology.orchestra.run.trial030.stdout: ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:21:28.839 INFO:teuthology.orchestra.run.trial030.stdout: ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:21:28.839 INFO:teuthology.orchestra.run.trial030.stdout: ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:21:28.839 INFO:teuthology.orchestra.run.trial030.stdout: ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:21:28.839 INFO:teuthology.orchestra.run.trial030.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:21:28.839 INFO:teuthology.orchestra.run.trial030.stdout: ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:21:28.839 INFO:teuthology.orchestra.run.trial030.stdout: ceph-mgr-modules-core-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:21:28.839 INFO:teuthology.orchestra.run.trial030.stdout: ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:21:28.839 INFO:teuthology.orchestra.run.trial030.stdout: ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:21:28.839 INFO:teuthology.orchestra.run.trial030.stdout: ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:21:28.839 INFO:teuthology.orchestra.run.trial030.stdout: ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:21:28.839 INFO:teuthology.orchestra.run.trial030.stdout: ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:21:28.840 INFO:teuthology.orchestra.run.trial030.stdout: ceph-prometheus-alerts-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:21:28.840 INFO:teuthology.orchestra.run.trial030.stdout: ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:21:28.840 INFO:teuthology.orchestra.run.trial030.stdout: cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:21:28.840 INFO:teuthology.orchestra.run.trial030.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-25T08:21:28.840 INFO:teuthology.orchestra.run.trial030.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-25T08:21:28.840 INFO:teuthology.orchestra.run.trial030.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-25T08:21:28.840 INFO:teuthology.orchestra.run.trial030.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-25T08:21:28.840 INFO:teuthology.orchestra.run.trial030.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-25T08:21:28.840 INFO:teuthology.orchestra.run.trial030.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-25T08:21:28.840 INFO:teuthology.orchestra.run.trial030.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-25T08:21:28.840 INFO:teuthology.orchestra.run.trial030.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-25T08:21:28.840 INFO:teuthology.orchestra.run.trial030.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-25T08:21:28.841 INFO:teuthology.orchestra.run.trial030.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-25T08:21:28.841 INFO:teuthology.orchestra.run.trial030.stdout: lua-5.4.4-4.el9.x86_64 2026-02-25T08:21:28.841 INFO:teuthology.orchestra.run.trial030.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-25T08:21:28.841 INFO:teuthology.orchestra.run.trial030.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-25T08:21:28.841 INFO:teuthology.orchestra.run.trial030.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-25T08:21:28.841 INFO:teuthology.orchestra.run.trial030.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-25T08:21:28.841 INFO:teuthology.orchestra.run.trial030.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-25T08:21:28.841 INFO:teuthology.orchestra.run.trial030.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-25T08:21:28.841 INFO:teuthology.orchestra.run.trial030.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-25T08:21:28.841 INFO:teuthology.orchestra.run.trial030.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-25T08:21:28.841 INFO:teuthology.orchestra.run.trial030.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-25T08:21:28.841 INFO:teuthology.orchestra.run.trial030.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-25T08:21:28.842 INFO:teuthology.orchestra.run.trial030.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-25T08:21:28.842 INFO:teuthology.orchestra.run.trial030.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-25T08:21:28.842 INFO:teuthology.orchestra.run.trial030.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-25T08:21:28.842 INFO:teuthology.orchestra.run.trial030.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-25T08:21:28.842 INFO:teuthology.orchestra.run.trial030.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-25T08:21:28.842 INFO:teuthology.orchestra.run.trial030.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-25T08:21:28.842 INFO:teuthology.orchestra.run.trial030.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-25T08:21:28.842 INFO:teuthology.orchestra.run.trial030.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-25T08:21:28.842 INFO:teuthology.orchestra.run.trial030.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-25T08:21:28.842 INFO:teuthology.orchestra.run.trial030.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-25T08:21:28.842 INFO:teuthology.orchestra.run.trial030.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-25T08:21:28.842 INFO:teuthology.orchestra.run.trial030.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-25T08:21:28.842 INFO:teuthology.orchestra.run.trial030.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-25T08:21:28.843 INFO:teuthology.orchestra.run.trial030.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-25T08:21:28.843 INFO:teuthology.orchestra.run.trial030.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-25T08:21:28.843 INFO:teuthology.orchestra.run.trial030.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-25T08:21:28.843 INFO:teuthology.orchestra.run.trial030.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-25T08:21:28.843 INFO:teuthology.orchestra.run.trial030.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-25T08:21:28.843 INFO:teuthology.orchestra.run.trial030.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-25T08:21:28.843 INFO:teuthology.orchestra.run.trial030.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-25T08:21:28.843 INFO:teuthology.orchestra.run.trial030.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-25T08:21:28.843 INFO:teuthology.orchestra.run.trial030.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-25T08:21:28.843 INFO:teuthology.orchestra.run.trial030.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-25T08:21:28.843 INFO:teuthology.orchestra.run.trial030.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-25T08:21:28.843 INFO:teuthology.orchestra.run.trial030.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-25T08:21:28.843 INFO:teuthology.orchestra.run.trial030.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-25T08:21:28.844 INFO:teuthology.orchestra.run.trial030.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-25T08:21:28.844 INFO:teuthology.orchestra.run.trial030.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-25T08:21:28.844 INFO:teuthology.orchestra.run.trial030.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-25T08:21:28.844 INFO:teuthology.orchestra.run.trial030.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-25T08:21:28.844 INFO:teuthology.orchestra.run.trial030.stdout: python3-ply-3.11-14.el9.noarch 2026-02-25T08:21:28.844 INFO:teuthology.orchestra.run.trial030.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-25T08:21:28.844 INFO:teuthology.orchestra.run.trial030.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-25T08:21:28.844 INFO:teuthology.orchestra.run.trial030.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-25T08:21:28.844 INFO:teuthology.orchestra.run.trial030.stdout: python3-pyasn1-0.4.8-7.el9.noarch 2026-02-25T08:21:28.844 INFO:teuthology.orchestra.run.trial030.stdout: python3-pyasn1-modules-0.4.8-7.el9.noarch 2026-02-25T08:21:28.844 INFO:teuthology.orchestra.run.trial030.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-25T08:21:28.844 INFO:teuthology.orchestra.run.trial030.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-25T08:21:28.845 INFO:teuthology.orchestra.run.trial030.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-25T08:21:28.845 INFO:teuthology.orchestra.run.trial030.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-25T08:21:28.845 INFO:teuthology.orchestra.run.trial030.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-25T08:21:28.845 INFO:teuthology.orchestra.run.trial030.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-25T08:21:28.845 INFO:teuthology.orchestra.run.trial030.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-25T08:21:28.845 INFO:teuthology.orchestra.run.trial030.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-25T08:21:28.845 INFO:teuthology.orchestra.run.trial030.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-25T08:21:28.845 INFO:teuthology.orchestra.run.trial030.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-25T08:21:28.845 INFO:teuthology.orchestra.run.trial030.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-25T08:21:28.845 INFO:teuthology.orchestra.run.trial030.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-25T08:21:28.845 INFO:teuthology.orchestra.run.trial030.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-25T08:21:28.845 INFO:teuthology.orchestra.run.trial030.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-25T08:21:28.845 INFO:teuthology.orchestra.run.trial030.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-25T08:21:28.846 INFO:teuthology.orchestra.run.trial030.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-25T08:21:28.846 INFO:teuthology.orchestra.run.trial030.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-25T08:21:28.846 INFO:teuthology.orchestra.run.trial030.stdout: unzip-6.0-59.el9.x86_64 2026-02-25T08:21:28.846 INFO:teuthology.orchestra.run.trial030.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-25T08:21:28.846 INFO:teuthology.orchestra.run.trial030.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-25T08:21:28.846 INFO:teuthology.orchestra.run.trial030.stdout: zip-3.0-35.el9.x86_64 2026-02-25T08:21:28.846 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:28.846 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:21:28.903 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install ceph-base 2026-02-25T08:21:29.222 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:14:03 ago on Wed 25 Feb 2026 08:07:26 AM UTC. 2026-02-25T08:21:29.287 INFO:teuthology.orchestra.run.trial030.stdout:Package ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:21:29.306 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:21:29.306 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:21:29.307 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:21:29.328 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install cephadm 2026-02-25T08:21:29.649 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:14:03 ago on Wed 25 Feb 2026 08:07:26 AM UTC. 2026-02-25T08:21:29.715 INFO:teuthology.orchestra.run.trial030.stdout:Package cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:21:29.733 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:21:29.734 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:21:29.734 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:21:29.755 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install ceph-immutable-object-cache 2026-02-25T08:21:30.069 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:14:04 ago on Wed 25 Feb 2026 08:07:26 AM UTC. 2026-02-25T08:21:30.153 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:21:30.154 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-25T08:21:30.154 INFO:teuthology.orchestra.run.trial030.stdout: Package Arch Version Repo Size 2026-02-25T08:21:30.154 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-25T08:21:30.154 INFO:teuthology.orchestra.run.trial030.stdout:Installing: 2026-02-25T08:21:30.154 INFO:teuthology.orchestra.run.trial030.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 149 k 2026-02-25T08:21:30.154 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:30.154 INFO:teuthology.orchestra.run.trial030.stdout:Transaction Summary 2026-02-25T08:21:30.154 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-25T08:21:30.155 INFO:teuthology.orchestra.run.trial030.stdout:Install 1 Package 2026-02-25T08:21:30.155 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:30.155 INFO:teuthology.orchestra.run.trial030.stdout:Total download size: 149 k 2026-02-25T08:21:30.155 INFO:teuthology.orchestra.run.trial030.stdout:Installed size: 455 k 2026-02-25T08:21:30.155 INFO:teuthology.orchestra.run.trial030.stdout:Downloading Packages: 2026-02-25T08:21:31.028 INFO:teuthology.orchestra.run.trial030.stdout:ceph-immutable-object-cache-20.3.0-5338.g6aeb69 171 kB/s | 149 kB 00:00 2026-02-25T08:21:31.029 INFO:teuthology.orchestra.run.trial030.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:21:31.029 INFO:teuthology.orchestra.run.trial030.stdout:Total 171 kB/s | 149 kB 00:00 2026-02-25T08:21:31.029 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction check 2026-02-25T08:21:31.034 INFO:teuthology.orchestra.run.trial030.stdout:Transaction check succeeded. 2026-02-25T08:21:31.034 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction test 2026-02-25T08:21:31.065 INFO:teuthology.orchestra.run.trial030.stdout:Transaction test succeeded. 2026-02-25T08:21:31.065 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction 2026-02-25T08:21:31.121 INFO:teuthology.orchestra.run.trial030.stdout: Preparing : 1/1 2026-02-25T08:21:31.123 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975. 1/1 2026-02-25T08:21:31.144 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975. 1/1 2026-02-25T08:21:31.145 INFO:teuthology.orchestra.run.trial030.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:21:31.145 INFO:teuthology.orchestra.run.trial030.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-25T08:21:31.145 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:31.474 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975. 1/1 2026-02-25T08:21:31.474 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:31.474 INFO:teuthology.orchestra.run.trial030.stdout:Installed: 2026-02-25T08:21:31.474 INFO:teuthology.orchestra.run.trial030.stdout: ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:21:31.474 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:31.474 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:21:31.501 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install ceph-mgr 2026-02-25T08:21:31.819 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:14:05 ago on Wed 25 Feb 2026 08:07:26 AM UTC. 2026-02-25T08:21:31.884 INFO:teuthology.orchestra.run.trial030.stdout:Package ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:21:31.903 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:21:31.903 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:21:31.904 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:21:31.924 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install ceph-mgr-dashboard 2026-02-25T08:21:32.237 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:14:06 ago on Wed 25 Feb 2026 08:07:26 AM UTC. 2026-02-25T08:21:32.302 INFO:teuthology.orchestra.run.trial030.stdout:Package ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:21:32.321 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:21:32.322 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:21:32.322 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:21:32.344 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-25T08:21:32.659 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:14:06 ago on Wed 25 Feb 2026 08:07:26 AM UTC. 2026-02-25T08:21:32.724 INFO:teuthology.orchestra.run.trial030.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:21:32.742 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:21:32.743 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:21:32.743 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:21:32.764 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install ceph-mgr-rook 2026-02-25T08:21:33.078 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:14:07 ago on Wed 25 Feb 2026 08:07:26 AM UTC. 2026-02-25T08:21:33.143 INFO:teuthology.orchestra.run.trial030.stdout:Package ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:21:33.162 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:21:33.163 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:21:33.163 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:21:33.185 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install ceph-mgr-cephadm 2026-02-25T08:21:33.502 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:14:07 ago on Wed 25 Feb 2026 08:07:26 AM UTC. 2026-02-25T08:21:33.567 INFO:teuthology.orchestra.run.trial030.stdout:Package ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:21:33.586 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:21:33.586 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:21:33.586 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:21:33.607 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install ceph-osd 2026-02-25T08:21:33.921 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:14:07 ago on Wed 25 Feb 2026 08:07:26 AM UTC. 2026-02-25T08:21:33.986 INFO:teuthology.orchestra.run.trial030.stdout:Package ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:21:34.005 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:21:34.005 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:21:34.006 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:21:34.026 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install ceph-osd-classic 2026-02-25T08:21:34.337 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:14:08 ago on Wed 25 Feb 2026 08:07:26 AM UTC. 2026-02-25T08:21:34.402 INFO:teuthology.orchestra.run.trial030.stdout:Package ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:21:34.421 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:21:34.421 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:21:34.422 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:21:34.443 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install ceph-fuse 2026-02-25T08:21:34.758 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:14:08 ago on Wed 25 Feb 2026 08:07:26 AM UTC. 2026-02-25T08:21:34.842 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:21:34.843 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-25T08:21:34.843 INFO:teuthology.orchestra.run.trial030.stdout: Package Arch Version Repo Size 2026-02-25T08:21:34.843 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-25T08:21:34.843 INFO:teuthology.orchestra.run.trial030.stdout:Installing: 2026-02-25T08:21:34.844 INFO:teuthology.orchestra.run.trial030.stdout: ceph-fuse x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 914 k 2026-02-25T08:21:34.844 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:34.844 INFO:teuthology.orchestra.run.trial030.stdout:Transaction Summary 2026-02-25T08:21:34.844 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-25T08:21:34.844 INFO:teuthology.orchestra.run.trial030.stdout:Install 1 Package 2026-02-25T08:21:34.844 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:34.844 INFO:teuthology.orchestra.run.trial030.stdout:Total download size: 914 k 2026-02-25T08:21:34.844 INFO:teuthology.orchestra.run.trial030.stdout:Installed size: 2.9 M 2026-02-25T08:21:34.844 INFO:teuthology.orchestra.run.trial030.stdout:Downloading Packages: 2026-02-25T08:21:38.647 INFO:teuthology.orchestra.run.trial030.stdout:ceph-fuse-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 240 kB/s | 914 kB 00:03 2026-02-25T08:21:38.648 INFO:teuthology.orchestra.run.trial030.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:21:38.648 INFO:teuthology.orchestra.run.trial030.stdout:Total 240 kB/s | 914 kB 00:03 2026-02-25T08:21:38.648 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction check 2026-02-25T08:21:38.654 INFO:teuthology.orchestra.run.trial030.stdout:Transaction check succeeded. 2026-02-25T08:21:38.654 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction test 2026-02-25T08:21:38.687 INFO:teuthology.orchestra.run.trial030.stdout:Transaction test succeeded. 2026-02-25T08:21:38.687 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction 2026-02-25T08:21:38.780 INFO:teuthology.orchestra.run.trial030.stdout: Preparing : 1/1 2026-02-25T08:21:38.797 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:21:39.049 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:21:39.111 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:21:39.111 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:39.111 INFO:teuthology.orchestra.run.trial030.stdout:Installed: 2026-02-25T08:21:39.111 INFO:teuthology.orchestra.run.trial030.stdout: ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:21:39.111 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:39.111 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:21:39.138 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install ceph-volume 2026-02-25T08:21:39.459 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:14:13 ago on Wed 25 Feb 2026 08:07:26 AM UTC. 2026-02-25T08:21:39.525 INFO:teuthology.orchestra.run.trial030.stdout:Package ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:21:39.544 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:21:39.545 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:21:39.545 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:21:39.566 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install librados-devel 2026-02-25T08:21:39.879 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:14:13 ago on Wed 25 Feb 2026 08:07:26 AM UTC. 2026-02-25T08:21:39.963 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:21:39.964 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-25T08:21:39.964 INFO:teuthology.orchestra.run.trial030.stdout: Package Arch Version Repo Size 2026-02-25T08:21:39.964 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-25T08:21:39.965 INFO:teuthology.orchestra.run.trial030.stdout:Installing: 2026-02-25T08:21:39.965 INFO:teuthology.orchestra.run.trial030.stdout: librados-devel x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 124 k 2026-02-25T08:21:39.965 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:39.965 INFO:teuthology.orchestra.run.trial030.stdout:Transaction Summary 2026-02-25T08:21:39.965 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-25T08:21:39.965 INFO:teuthology.orchestra.run.trial030.stdout:Install 1 Package 2026-02-25T08:21:39.965 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:39.965 INFO:teuthology.orchestra.run.trial030.stdout:Total download size: 124 k 2026-02-25T08:21:39.965 INFO:teuthology.orchestra.run.trial030.stdout:Installed size: 434 k 2026-02-25T08:21:39.965 INFO:teuthology.orchestra.run.trial030.stdout:Downloading Packages: 2026-02-25T08:21:40.534 INFO:teuthology.orchestra.run.trial030.stdout:librados-devel-20.3.0-5338.g6aeb6975.el9.x86_64 217 kB/s | 124 kB 00:00 2026-02-25T08:21:40.535 INFO:teuthology.orchestra.run.trial030.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:21:40.535 INFO:teuthology.orchestra.run.trial030.stdout:Total 216 kB/s | 124 kB 00:00 2026-02-25T08:21:40.536 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction check 2026-02-25T08:21:40.538 INFO:teuthology.orchestra.run.trial030.stdout:Transaction check succeeded. 2026-02-25T08:21:40.538 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction test 2026-02-25T08:21:40.570 INFO:teuthology.orchestra.run.trial030.stdout:Transaction test succeeded. 2026-02-25T08:21:40.570 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction 2026-02-25T08:21:40.625 INFO:teuthology.orchestra.run.trial030.stdout: Preparing : 1/1 2026-02-25T08:21:40.642 INFO:teuthology.orchestra.run.trial030.stdout: Installing : librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:21:40.729 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:21:40.794 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:21:40.794 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:40.795 INFO:teuthology.orchestra.run.trial030.stdout:Installed: 2026-02-25T08:21:40.795 INFO:teuthology.orchestra.run.trial030.stdout: librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:21:40.795 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:40.795 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:21:40.823 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install libcephfs2 2026-02-25T08:21:41.139 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:14:15 ago on Wed 25 Feb 2026 08:07:26 AM UTC. 2026-02-25T08:21:41.204 INFO:teuthology.orchestra.run.trial030.stdout:Package libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:21:41.223 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:21:41.223 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:21:41.224 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:21:41.245 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install libcephfs-devel 2026-02-25T08:21:41.561 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:14:15 ago on Wed 25 Feb 2026 08:07:26 AM UTC. 2026-02-25T08:21:41.645 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:21:41.646 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-25T08:21:41.646 INFO:teuthology.orchestra.run.trial030.stdout: Package Arch Version Repo Size 2026-02-25T08:21:41.646 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-25T08:21:41.647 INFO:teuthology.orchestra.run.trial030.stdout:Installing: 2026-02-25T08:21:41.647 INFO:teuthology.orchestra.run.trial030.stdout: libcephfs-devel x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 37 k 2026-02-25T08:21:41.647 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:41.647 INFO:teuthology.orchestra.run.trial030.stdout:Transaction Summary 2026-02-25T08:21:41.647 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-25T08:21:41.647 INFO:teuthology.orchestra.run.trial030.stdout:Install 1 Package 2026-02-25T08:21:41.647 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:41.647 INFO:teuthology.orchestra.run.trial030.stdout:Total download size: 37 k 2026-02-25T08:21:41.647 INFO:teuthology.orchestra.run.trial030.stdout:Installed size: 171 k 2026-02-25T08:21:41.647 INFO:teuthology.orchestra.run.trial030.stdout:Downloading Packages: 2026-02-25T08:21:42.167 INFO:teuthology.orchestra.run.trial030.stdout:libcephfs-devel-20.3.0-5338.g6aeb6975.el9.x86_6 72 kB/s | 37 kB 00:00 2026-02-25T08:21:42.168 INFO:teuthology.orchestra.run.trial030.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:21:42.168 INFO:teuthology.orchestra.run.trial030.stdout:Total 71 kB/s | 37 kB 00:00 2026-02-25T08:21:42.168 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction check 2026-02-25T08:21:42.171 INFO:teuthology.orchestra.run.trial030.stdout:Transaction check succeeded. 2026-02-25T08:21:42.171 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction test 2026-02-25T08:21:42.173 INFO:teuthology.orchestra.run.trial030.stdout:Transaction test succeeded. 2026-02-25T08:21:42.173 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction 2026-02-25T08:21:42.196 INFO:teuthology.orchestra.run.trial030.stdout: Preparing : 1/1 2026-02-25T08:21:42.212 INFO:teuthology.orchestra.run.trial030.stdout: Installing : libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:21:42.274 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:21:42.337 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:21:42.338 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:42.338 INFO:teuthology.orchestra.run.trial030.stdout:Installed: 2026-02-25T08:21:42.338 INFO:teuthology.orchestra.run.trial030.stdout: libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:21:42.338 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:42.338 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:21:42.367 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install librados2 2026-02-25T08:21:42.686 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:14:16 ago on Wed 25 Feb 2026 08:07:26 AM UTC. 2026-02-25T08:21:42.751 INFO:teuthology.orchestra.run.trial030.stdout:Package librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:21:42.768 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:21:42.769 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:21:42.769 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:21:42.791 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install librbd1 2026-02-25T08:21:43.115 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:14:17 ago on Wed 25 Feb 2026 08:07:26 AM UTC. 2026-02-25T08:21:43.180 INFO:teuthology.orchestra.run.trial030.stdout:Package librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:21:43.197 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:21:43.198 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:21:43.198 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:21:43.221 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install python3-rados 2026-02-25T08:21:43.535 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:14:17 ago on Wed 25 Feb 2026 08:07:26 AM UTC. 2026-02-25T08:21:43.600 INFO:teuthology.orchestra.run.trial030.stdout:Package python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:21:43.618 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:21:43.619 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:21:43.619 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:21:43.640 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install python3-rgw 2026-02-25T08:21:43.959 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:14:17 ago on Wed 25 Feb 2026 08:07:26 AM UTC. 2026-02-25T08:21:44.024 INFO:teuthology.orchestra.run.trial030.stdout:Package python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:21:44.041 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:21:44.042 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:21:44.042 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:21:44.064 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install python3-cephfs 2026-02-25T08:21:44.384 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:14:18 ago on Wed 25 Feb 2026 08:07:26 AM UTC. 2026-02-25T08:21:44.450 INFO:teuthology.orchestra.run.trial030.stdout:Package python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:21:44.467 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:21:44.468 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:21:44.468 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:21:44.490 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install python3-rbd 2026-02-25T08:21:44.804 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:14:18 ago on Wed 25 Feb 2026 08:07:26 AM UTC. 2026-02-25T08:21:44.870 INFO:teuthology.orchestra.run.trial030.stdout:Package python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:21:44.888 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:21:44.888 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:21:44.888 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:21:44.910 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install rbd-fuse 2026-02-25T08:21:45.227 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:14:19 ago on Wed 25 Feb 2026 08:07:26 AM UTC. 2026-02-25T08:21:45.310 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:21:45.311 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-25T08:21:45.311 INFO:teuthology.orchestra.run.trial030.stdout: Package Architecture Version Repository Size 2026-02-25T08:21:45.312 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-25T08:21:45.312 INFO:teuthology.orchestra.run.trial030.stdout:Installing: 2026-02-25T08:21:45.312 INFO:teuthology.orchestra.run.trial030.stdout: rbd-fuse x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 86 k 2026-02-25T08:21:45.312 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:45.312 INFO:teuthology.orchestra.run.trial030.stdout:Transaction Summary 2026-02-25T08:21:45.312 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-25T08:21:45.312 INFO:teuthology.orchestra.run.trial030.stdout:Install 1 Package 2026-02-25T08:21:45.312 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:45.312 INFO:teuthology.orchestra.run.trial030.stdout:Total download size: 86 k 2026-02-25T08:21:45.312 INFO:teuthology.orchestra.run.trial030.stdout:Installed size: 236 k 2026-02-25T08:21:45.313 INFO:teuthology.orchestra.run.trial030.stdout:Downloading Packages: 2026-02-25T08:21:45.944 INFO:teuthology.orchestra.run.trial030.stdout:rbd-fuse-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 136 kB/s | 86 kB 00:00 2026-02-25T08:21:45.944 INFO:teuthology.orchestra.run.trial030.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:21:45.944 INFO:teuthology.orchestra.run.trial030.stdout:Total 136 kB/s | 86 kB 00:00 2026-02-25T08:21:45.944 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction check 2026-02-25T08:21:45.948 INFO:teuthology.orchestra.run.trial030.stdout:Transaction check succeeded. 2026-02-25T08:21:45.948 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction test 2026-02-25T08:21:45.979 INFO:teuthology.orchestra.run.trial030.stdout:Transaction test succeeded. 2026-02-25T08:21:45.979 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction 2026-02-25T08:21:46.031 INFO:teuthology.orchestra.run.trial030.stdout: Preparing : 1/1 2026-02-25T08:21:46.048 INFO:teuthology.orchestra.run.trial030.stdout: Installing : rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:21:46.135 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:21:46.197 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:21:46.198 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:46.198 INFO:teuthology.orchestra.run.trial030.stdout:Installed: 2026-02-25T08:21:46.198 INFO:teuthology.orchestra.run.trial030.stdout: rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:21:46.198 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:46.198 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:21:46.227 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install rbd-mirror 2026-02-25T08:21:46.536 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:14:20 ago on Wed 25 Feb 2026 08:07:26 AM UTC. 2026-02-25T08:21:46.620 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:21:46.620 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-25T08:21:46.620 INFO:teuthology.orchestra.run.trial030.stdout: Package Arch Version Repo Size 2026-02-25T08:21:46.620 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-25T08:21:46.620 INFO:teuthology.orchestra.run.trial030.stdout:Installing: 2026-02-25T08:21:46.620 INFO:teuthology.orchestra.run.trial030.stdout: rbd-mirror x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 2.6 M 2026-02-25T08:21:46.621 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:46.621 INFO:teuthology.orchestra.run.trial030.stdout:Transaction Summary 2026-02-25T08:21:46.621 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-25T08:21:46.621 INFO:teuthology.orchestra.run.trial030.stdout:Install 1 Package 2026-02-25T08:21:46.621 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:46.621 INFO:teuthology.orchestra.run.trial030.stdout:Total download size: 2.6 M 2026-02-25T08:21:46.621 INFO:teuthology.orchestra.run.trial030.stdout:Installed size: 11 M 2026-02-25T08:21:46.621 INFO:teuthology.orchestra.run.trial030.stdout:Downloading Packages: 2026-02-25T08:21:55.312 INFO:teuthology.orchestra.run.trial030.stdout:rbd-mirror-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 305 kB/s | 2.6 MB 00:08 2026-02-25T08:21:55.313 INFO:teuthology.orchestra.run.trial030.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:21:55.313 INFO:teuthology.orchestra.run.trial030.stdout:Total 305 kB/s | 2.6 MB 00:08 2026-02-25T08:21:55.313 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction check 2026-02-25T08:21:55.319 INFO:teuthology.orchestra.run.trial030.stdout:Transaction check succeeded. 2026-02-25T08:21:55.319 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction test 2026-02-25T08:21:55.357 INFO:teuthology.orchestra.run.trial030.stdout:Transaction test succeeded. 2026-02-25T08:21:55.357 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction 2026-02-25T08:21:55.521 INFO:teuthology.orchestra.run.trial030.stdout: Preparing : 1/1 2026-02-25T08:21:55.523 INFO:teuthology.orchestra.run.trial030.stdout: Installing : rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:21:55.545 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:21:55.545 INFO:teuthology.orchestra.run.trial030.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:21:55.545 INFO:teuthology.orchestra.run.trial030.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-25T08:21:55.545 INFO:teuthology.orchestra.run.trial030.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-25T08:21:55.546 INFO:teuthology.orchestra.run.trial030.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-25T08:21:55.546 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:55.882 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:21:55.883 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:55.883 INFO:teuthology.orchestra.run.trial030.stdout:Installed: 2026-02-25T08:21:55.883 INFO:teuthology.orchestra.run.trial030.stdout: rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:21:55.883 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:55.883 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:21:55.910 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install rbd-nbd 2026-02-25T08:21:56.236 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:14:30 ago on Wed 25 Feb 2026 08:07:26 AM UTC. 2026-02-25T08:21:56.320 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:21:56.321 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-25T08:21:56.321 INFO:teuthology.orchestra.run.trial030.stdout: Package Architecture Version Repository Size 2026-02-25T08:21:56.321 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-25T08:21:56.321 INFO:teuthology.orchestra.run.trial030.stdout:Installing: 2026-02-25T08:21:56.321 INFO:teuthology.orchestra.run.trial030.stdout: rbd-nbd x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 166 k 2026-02-25T08:21:56.321 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:56.321 INFO:teuthology.orchestra.run.trial030.stdout:Transaction Summary 2026-02-25T08:21:56.321 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-25T08:21:56.321 INFO:teuthology.orchestra.run.trial030.stdout:Install 1 Package 2026-02-25T08:21:56.321 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:56.322 INFO:teuthology.orchestra.run.trial030.stdout:Total download size: 166 k 2026-02-25T08:21:56.322 INFO:teuthology.orchestra.run.trial030.stdout:Installed size: 496 k 2026-02-25T08:21:56.322 INFO:teuthology.orchestra.run.trial030.stdout:Downloading Packages: 2026-02-25T08:21:57.121 INFO:teuthology.orchestra.run.trial030.stdout:rbd-nbd-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 208 kB/s | 166 kB 00:00 2026-02-25T08:21:57.121 INFO:teuthology.orchestra.run.trial030.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:21:57.121 INFO:teuthology.orchestra.run.trial030.stdout:Total 208 kB/s | 166 kB 00:00 2026-02-25T08:21:57.121 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction check 2026-02-25T08:21:57.126 INFO:teuthology.orchestra.run.trial030.stdout:Transaction check succeeded. 2026-02-25T08:21:57.126 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction test 2026-02-25T08:21:57.158 INFO:teuthology.orchestra.run.trial030.stdout:Transaction test succeeded. 2026-02-25T08:21:57.158 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction 2026-02-25T08:21:57.214 INFO:teuthology.orchestra.run.trial030.stdout: Preparing : 1/1 2026-02-25T08:21:57.231 INFO:teuthology.orchestra.run.trial030.stdout: Installing : rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:21:57.317 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:21:57.383 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:21:57.383 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:57.383 INFO:teuthology.orchestra.run.trial030.stdout:Installed: 2026-02-25T08:21:57.383 INFO:teuthology.orchestra.run.trial030.stdout: rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:21:57.383 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:57.383 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:21:57.412 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install ceph-osd-crimson 2026-02-25T08:21:57.729 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:14:31 ago on Wed 25 Feb 2026 08:07:26 AM UTC. 2026-02-25T08:21:57.794 INFO:teuthology.orchestra.run.trial030.stdout:Package ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:21:57.812 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:21:57.813 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:21:57.813 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:21:57.834 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install ceph-osd-crimson-debuginfo 2026-02-25T08:21:58.154 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:14:32 ago on Wed 25 Feb 2026 08:07:26 AM UTC. 2026-02-25T08:21:58.237 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:21:58.238 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-25T08:21:58.238 INFO:teuthology.orchestra.run.trial030.stdout: Package Arch Version Repo Size 2026-02-25T08:21:58.239 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-25T08:21:58.239 INFO:teuthology.orchestra.run.trial030.stdout:Installing: 2026-02-25T08:21:58.239 INFO:teuthology.orchestra.run.trial030.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 433 M 2026-02-25T08:21:58.239 INFO:teuthology.orchestra.run.trial030.stdout:Installing weak dependencies: 2026-02-25T08:21:58.239 INFO:teuthology.orchestra.run.trial030.stdout: ceph-debugsource x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 12 M 2026-02-25T08:21:58.239 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:58.239 INFO:teuthology.orchestra.run.trial030.stdout:Transaction Summary 2026-02-25T08:21:58.239 INFO:teuthology.orchestra.run.trial030.stdout:================================================================================ 2026-02-25T08:21:58.239 INFO:teuthology.orchestra.run.trial030.stdout:Install 2 Packages 2026-02-25T08:21:58.239 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:21:58.240 INFO:teuthology.orchestra.run.trial030.stdout:Total download size: 445 M 2026-02-25T08:21:58.240 INFO:teuthology.orchestra.run.trial030.stdout:Installed size: 2.7 G 2026-02-25T08:21:58.240 INFO:teuthology.orchestra.run.trial030.stdout:Downloading Packages: 2026-02-25T08:22:24.462 INFO:teuthology.orchestra.run.trial030.stdout:(1/2): ceph-debugsource-20.3.0-5338.g6aeb6975.e 470 kB/s | 12 MB 00:26 2026-02-25T08:34:33.520 INFO:teuthology.orchestra.run.trial023.stdout:(2/2): ceph-osd-crimson-debuginfo-20.3.0-5338.g 457 kB/s | 433 MB 16:10 2026-02-25T08:34:33.521 INFO:teuthology.orchestra.run.trial023.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:34:33.521 INFO:teuthology.orchestra.run.trial023.stdout:Total 469 kB/s | 445 MB 16:10 2026-02-25T08:34:33.522 INFO:teuthology.orchestra.run.trial023.stdout:Running transaction check 2026-02-25T08:34:33.527 INFO:teuthology.orchestra.run.trial023.stdout:Transaction check succeeded. 2026-02-25T08:34:33.527 INFO:teuthology.orchestra.run.trial023.stdout:Running transaction test 2026-02-25T08:34:34.898 INFO:teuthology.orchestra.run.trial023.stdout:Transaction test succeeded. 2026-02-25T08:34:34.899 INFO:teuthology.orchestra.run.trial023.stdout:Running transaction 2026-02-25T08:34:38.483 INFO:teuthology.orchestra.run.trial023.stdout: Preparing : 1/1 2026-02-25T08:35:01.353 INFO:teuthology.orchestra.run.trial023.stdout: Installing : ceph-debugsource-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/2 2026-02-25T08:35:01.371 INFO:teuthology.orchestra.run.trial023.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.e 2/2 2026-02-25T08:35:02.197 INFO:teuthology.orchestra.run.trial023.stdout: Running scriptlet: ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.e 2/2 2026-02-25T08:35:02.197 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : ceph-debugsource-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/2 2026-02-25T08:35:02.259 INFO:teuthology.orchestra.run.trial023.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.e 2/2 2026-02-25T08:35:02.260 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:35:02.260 INFO:teuthology.orchestra.run.trial023.stdout:Installed: 2026-02-25T08:35:02.260 INFO:teuthology.orchestra.run.trial023.stdout: ceph-debugsource-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:35:02.260 INFO:teuthology.orchestra.run.trial023.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:35:02.260 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:35:02.260 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:35:02.336 DEBUG:teuthology.parallel:result is None 2026-02-25T08:39:35.517 INFO:teuthology.orchestra.run.trial030.stdout:(2/2): ceph-osd-crimson-debuginfo-20.3.0-5338.g 419 kB/s | 433 MB 17:37 2026-02-25T08:39:35.518 INFO:teuthology.orchestra.run.trial030.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:39:35.518 INFO:teuthology.orchestra.run.trial030.stdout:Total 431 kB/s | 445 MB 17:37 2026-02-25T08:39:35.519 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction check 2026-02-25T08:39:35.524 INFO:teuthology.orchestra.run.trial030.stdout:Transaction check succeeded. 2026-02-25T08:39:35.524 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction test 2026-02-25T08:39:36.905 INFO:teuthology.orchestra.run.trial030.stdout:Transaction test succeeded. 2026-02-25T08:39:36.905 INFO:teuthology.orchestra.run.trial030.stdout:Running transaction 2026-02-25T08:39:40.483 INFO:teuthology.orchestra.run.trial030.stdout: Preparing : 1/1 2026-02-25T08:40:03.113 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-debugsource-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/2 2026-02-25T08:40:03.131 INFO:teuthology.orchestra.run.trial030.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.e 2/2 2026-02-25T08:40:03.392 INFO:teuthology.orchestra.run.trial030.stdout: Running scriptlet: ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.e 2/2 2026-02-25T08:40:03.392 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-debugsource-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/2 2026-02-25T08:40:03.456 INFO:teuthology.orchestra.run.trial030.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.e 2/2 2026-02-25T08:40:03.456 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:40:03.456 INFO:teuthology.orchestra.run.trial030.stdout:Installed: 2026-02-25T08:40:03.456 INFO:teuthology.orchestra.run.trial030.stdout: ceph-debugsource-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:40:03.456 INFO:teuthology.orchestra.run.trial030.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:40:03.456 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:40:03.456 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:40:03.536 DEBUG:teuthology.parallel:result is None 2026-02-25T08:40:03.537 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:40:03.700 DEBUG:teuthology.orchestra.run.trial023:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-25T08:40:03.733 INFO:teuthology.orchestra.run.trial023.stdout:20.3.0-5338.g6aeb6975.el9 2026-02-25T08:40:03.733 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5338.g6aeb6975.el9 2026-02-25T08:40:03.733 INFO:teuthology.task.install:The correct ceph version 20.3.0-5338.g6aeb6975 is installed. 2026-02-25T08:40:03.736 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:40:03.899 DEBUG:teuthology.orchestra.run.trial030:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-25T08:40:03.921 INFO:teuthology.orchestra.run.trial030.stdout:20.3.0-5338.g6aeb6975.el9 2026-02-25T08:40:03.921 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5338.g6aeb6975.el9 2026-02-25T08:40:03.921 INFO:teuthology.task.install:The correct ceph version 20.3.0-5338.g6aeb6975 is installed. 2026-02-25T08:40:03.924 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-25T08:40:03.924 DEBUG:teuthology.orchestra.run.trial023:> set -ex 2026-02-25T08:40:03.924 DEBUG:teuthology.orchestra.run.trial023:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-25T08:40:03.951 DEBUG:teuthology.orchestra.run.trial030:> set -ex 2026-02-25T08:40:03.952 DEBUG:teuthology.orchestra.run.trial030:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-25T08:40:03.988 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-25T08:40:03.988 DEBUG:teuthology.orchestra.run.trial023:> set -ex 2026-02-25T08:40:03.988 DEBUG:teuthology.orchestra.run.trial023:> sudo dd of=/usr/bin/daemon-helper 2026-02-25T08:40:04.015 DEBUG:teuthology.orchestra.run.trial023:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-25T08:40:04.079 DEBUG:teuthology.orchestra.run.trial030:> set -ex 2026-02-25T08:40:04.079 DEBUG:teuthology.orchestra.run.trial030:> sudo dd of=/usr/bin/daemon-helper 2026-02-25T08:40:04.106 DEBUG:teuthology.orchestra.run.trial030:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-25T08:40:04.171 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-25T08:40:04.172 DEBUG:teuthology.orchestra.run.trial023:> set -ex 2026-02-25T08:40:04.172 DEBUG:teuthology.orchestra.run.trial023:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-25T08:40:04.198 DEBUG:teuthology.orchestra.run.trial023:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-25T08:40:04.262 DEBUG:teuthology.orchestra.run.trial030:> set -ex 2026-02-25T08:40:04.262 DEBUG:teuthology.orchestra.run.trial030:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-25T08:40:04.287 DEBUG:teuthology.orchestra.run.trial030:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-25T08:40:04.352 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-25T08:40:04.352 DEBUG:teuthology.orchestra.run.trial023:> set -ex 2026-02-25T08:40:04.352 DEBUG:teuthology.orchestra.run.trial023:> sudo dd of=/usr/bin/stdin-killer 2026-02-25T08:40:04.377 DEBUG:teuthology.orchestra.run.trial023:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-25T08:40:04.441 DEBUG:teuthology.orchestra.run.trial030:> set -ex 2026-02-25T08:40:04.441 DEBUG:teuthology.orchestra.run.trial030:> sudo dd of=/usr/bin/stdin-killer 2026-02-25T08:40:04.467 DEBUG:teuthology.orchestra.run.trial030:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-25T08:40:04.530 INFO:teuthology.run_tasks:Running task pexec... 2026-02-25T08:40:04.543 INFO:teuthology.task.pexec:Executing custom commands... 2026-02-25T08:40:04.543 DEBUG:teuthology.orchestra.run.trial023:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-25T08:40:04.543 DEBUG:teuthology.orchestra.run.trial030:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-25T08:40:04.545 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial023.front.sepia.ceph.com 2026-02-25T08:40:04.545 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-25T08:40:04.572 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial030.front.sepia.ceph.com 2026-02-25T08:40:04.572 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-25T08:40:04.595 DEBUG:teuthology.parallel:result is None 2026-02-25T08:40:04.622 DEBUG:teuthology.parallel:result is None 2026-02-25T08:40:04.622 INFO:teuthology.run_tasks:Running task install... 2026-02-25T08:40:04.634 DEBUG:teuthology.task.install:project ceph 2026-02-25T08:40:04.634 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}} 2026-02-25T08:40:04.634 DEBUG:teuthology.task.install:config {'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'} 2026-02-25T08:40:04.634 INFO:teuthology.task.install:Using flavor: debug 2026-02-25T08:40:04.637 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-25T08:40:04.637 INFO:teuthology.task.install:extra packages: [] 2026-02-25T08:40:04.637 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819', 'tag': None, 'wait_for_package': False} 2026-02-25T08:40:04.637 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:40:04.638 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819', 'tag': None, 'wait_for_package': False} 2026-02-25T08:40:04.638 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:40:04.795 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/ 2026-02-25T08:40:04.795 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5338.g6aeb6975 2026-02-25T08:40:04.806 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/ 2026-02-25T08:40:04.806 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5338.g6aeb6975 2026-02-25T08:40:05.076 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-25T08:40:05.077 DEBUG:teuthology.orchestra.run.trial023:> set -ex 2026-02-25T08:40:05.077 DEBUG:teuthology.orchestra.run.trial023:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-25T08:40:05.102 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-25T08:40:05.102 DEBUG:teuthology.orchestra.run.trial023:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-25T08:40:05.149 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-25T08:40:05.149 DEBUG:teuthology.orchestra.run.trial030:> set -ex 2026-02-25T08:40:05.149 DEBUG:teuthology.orchestra.run.trial030:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-25T08:40:05.166 DEBUG:teuthology.orchestra.run.trial023:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-25T08:40:05.174 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-25T08:40:05.174 DEBUG:teuthology.orchestra.run.trial030:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-25T08:40:05.230 DEBUG:teuthology.orchestra.run.trial023:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-25T08:40:05.239 DEBUG:teuthology.orchestra.run.trial030:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-25T08:40:05.286 INFO:teuthology.orchestra.run.trial023.stdout:check_obsoletes = 1 2026-02-25T08:40:05.295 DEBUG:teuthology.orchestra.run.trial023:> sudo yum clean all 2026-02-25T08:40:05.302 DEBUG:teuthology.orchestra.run.trial030:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-25T08:40:05.358 INFO:teuthology.orchestra.run.trial030.stdout:check_obsoletes = 1 2026-02-25T08:40:05.368 DEBUG:teuthology.orchestra.run.trial030:> sudo yum clean all 2026-02-25T08:40:05.455 INFO:teuthology.orchestra.run.trial023.stdout:56 files removed 2026-02-25T08:40:05.470 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install ceph-radosgw 2026-02-25T08:40:05.528 INFO:teuthology.orchestra.run.trial030.stdout:56 files removed 2026-02-25T08:40:05.544 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install ceph-radosgw 2026-02-25T08:40:06.211 INFO:teuthology.orchestra.run.trial023.stdout:ceph packages for x86_64 161 kB/s | 94 kB 00:00 2026-02-25T08:40:06.276 INFO:teuthology.orchestra.run.trial030.stdout:ceph packages for x86_64 163 kB/s | 94 kB 00:00 2026-02-25T08:40:06.683 INFO:teuthology.orchestra.run.trial023.stdout:ceph noarch packages 55 kB/s | 24 kB 00:00 2026-02-25T08:40:06.737 INFO:teuthology.orchestra.run.trial030.stdout:ceph noarch packages 57 kB/s | 24 kB 00:00 2026-02-25T08:40:07.141 INFO:teuthology.orchestra.run.trial023.stdout:ceph source packages 4.8 kB/s | 2.1 kB 00:00 2026-02-25T08:40:07.186 INFO:teuthology.orchestra.run.trial030.stdout:ceph source packages 4.9 kB/s | 2.1 kB 00:00 2026-02-25T08:40:08.143 INFO:teuthology.orchestra.run.trial030.stdout:CentOS Stream 9 - BaseOS 9.5 MB/s | 8.9 MB 00:00 2026-02-25T08:40:08.549 INFO:teuthology.orchestra.run.trial023.stdout:CentOS Stream 9 - BaseOS 6.4 MB/s | 8.9 MB 00:01 2026-02-25T08:40:11.282 INFO:teuthology.orchestra.run.trial023.stdout:CentOS Stream 9 - AppStream 12 MB/s | 27 MB 00:02 2026-02-25T08:40:12.017 INFO:teuthology.orchestra.run.trial030.stdout:CentOS Stream 9 - AppStream 8.1 MB/s | 27 MB 00:03 2026-02-25T08:40:14.924 INFO:teuthology.orchestra.run.trial023.stdout:CentOS Stream 9 - CRB 6.4 MB/s | 8.0 MB 00:01 2026-02-25T08:40:19.672 INFO:teuthology.orchestra.run.trial023.stdout:CentOS Stream 9 - Extras packages 5.0 kB/s | 20 kB 00:04 2026-02-25T08:40:20.531 INFO:teuthology.orchestra.run.trial023.stdout:Extra Packages for Enterprise Linux 26 MB/s | 20 MB 00:00 2026-02-25T08:40:23.561 INFO:teuthology.orchestra.run.trial030.stdout:CentOS Stream 9 - CRB 889 kB/s | 8.0 MB 00:09 2026-02-25T08:40:23.791 INFO:teuthology.orchestra.run.trial023.stdout:lab-extras 2.3 MB/s | 50 kB 00:00 2026-02-25T08:40:24.978 INFO:teuthology.orchestra.run.trial023.stdout:Package ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:24.996 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:40:24.996 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:40:24.997 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:40:25.021 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install ceph-test 2026-02-25T08:40:25.337 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:00:02 ago on Wed 25 Feb 2026 08:40:23 AM UTC. 2026-02-25T08:40:25.402 INFO:teuthology.orchestra.run.trial023.stdout:Package ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:25.420 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:40:25.421 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:40:25.421 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:40:25.443 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install ceph 2026-02-25T08:40:25.641 INFO:teuthology.orchestra.run.trial030.stdout:CentOS Stream 9 - Extras packages 15 kB/s | 20 kB 00:01 2026-02-25T08:40:25.755 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:00:02 ago on Wed 25 Feb 2026 08:40:23 AM UTC. 2026-02-25T08:40:25.820 INFO:teuthology.orchestra.run.trial023.stdout:Package ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:25.838 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:40:25.839 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:40:25.839 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:40:25.860 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install ceph-base 2026-02-25T08:40:26.178 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:00:03 ago on Wed 25 Feb 2026 08:40:23 AM UTC. 2026-02-25T08:40:26.244 INFO:teuthology.orchestra.run.trial023.stdout:Package ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:26.261 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:40:26.262 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:40:26.262 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:40:26.284 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install cephadm 2026-02-25T08:40:26.603 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:00:03 ago on Wed 25 Feb 2026 08:40:23 AM UTC. 2026-02-25T08:40:26.668 INFO:teuthology.orchestra.run.trial023.stdout:Package cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:26.686 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:40:26.687 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:40:26.687 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:40:26.709 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install ceph-immutable-object-cache 2026-02-25T08:40:27.025 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:00:04 ago on Wed 25 Feb 2026 08:40:23 AM UTC. 2026-02-25T08:40:27.030 INFO:teuthology.orchestra.run.trial030.stdout:Extra Packages for Enterprise Linux 15 MB/s | 20 MB 00:01 2026-02-25T08:40:27.090 INFO:teuthology.orchestra.run.trial023.stdout:Package ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:27.108 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:40:27.108 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:40:27.108 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:40:27.130 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install ceph-mgr 2026-02-25T08:40:27.448 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:00:04 ago on Wed 25 Feb 2026 08:40:23 AM UTC. 2026-02-25T08:40:27.513 INFO:teuthology.orchestra.run.trial023.stdout:Package ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:27.531 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:40:27.532 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:40:27.532 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:40:27.553 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install ceph-mgr-dashboard 2026-02-25T08:40:27.870 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:00:04 ago on Wed 25 Feb 2026 08:40:23 AM UTC. 2026-02-25T08:40:27.936 INFO:teuthology.orchestra.run.trial023.stdout:Package ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:27.954 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:40:27.955 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:40:27.955 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:40:27.976 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-25T08:40:28.294 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:00:05 ago on Wed 25 Feb 2026 08:40:23 AM UTC. 2026-02-25T08:40:28.359 INFO:teuthology.orchestra.run.trial023.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:28.377 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:40:28.377 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:40:28.377 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:40:28.399 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install ceph-mgr-rook 2026-02-25T08:40:28.718 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:00:05 ago on Wed 25 Feb 2026 08:40:23 AM UTC. 2026-02-25T08:40:28.783 INFO:teuthology.orchestra.run.trial023.stdout:Package ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:28.801 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:40:28.801 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:40:28.802 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:40:28.823 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install ceph-mgr-cephadm 2026-02-25T08:40:29.139 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:00:06 ago on Wed 25 Feb 2026 08:40:23 AM UTC. 2026-02-25T08:40:29.206 INFO:teuthology.orchestra.run.trial023.stdout:Package ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:29.224 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:40:29.225 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:40:29.225 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:40:29.247 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install ceph-osd 2026-02-25T08:40:29.570 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:00:06 ago on Wed 25 Feb 2026 08:40:23 AM UTC. 2026-02-25T08:40:29.635 INFO:teuthology.orchestra.run.trial023.stdout:Package ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:29.652 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:40:29.653 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:40:29.653 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:40:29.675 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install ceph-osd-classic 2026-02-25T08:40:29.992 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:00:06 ago on Wed 25 Feb 2026 08:40:23 AM UTC. 2026-02-25T08:40:30.058 INFO:teuthology.orchestra.run.trial023.stdout:Package ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:30.078 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:40:30.078 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:40:30.078 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:40:30.100 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install ceph-fuse 2026-02-25T08:40:30.185 INFO:teuthology.orchestra.run.trial030.stdout:lab-extras 2.8 MB/s | 50 kB 00:00 2026-02-25T08:40:30.417 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:00:07 ago on Wed 25 Feb 2026 08:40:23 AM UTC. 2026-02-25T08:40:30.483 INFO:teuthology.orchestra.run.trial023.stdout:Package ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:30.501 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:40:30.501 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:40:30.501 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:40:30.523 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install ceph-volume 2026-02-25T08:40:30.841 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:00:07 ago on Wed 25 Feb 2026 08:40:23 AM UTC. 2026-02-25T08:40:30.908 INFO:teuthology.orchestra.run.trial023.stdout:Package ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:30.926 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:40:30.926 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:40:30.926 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:40:30.948 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install librados-devel 2026-02-25T08:40:31.270 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:00:08 ago on Wed 25 Feb 2026 08:40:23 AM UTC. 2026-02-25T08:40:31.335 INFO:teuthology.orchestra.run.trial023.stdout:Package librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:31.353 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:40:31.353 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:40:31.354 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:40:31.366 INFO:teuthology.orchestra.run.trial030.stdout:Package ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:31.375 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install libcephfs2 2026-02-25T08:40:31.384 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:40:31.385 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:40:31.385 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:40:31.408 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install ceph-test 2026-02-25T08:40:31.692 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:00:08 ago on Wed 25 Feb 2026 08:40:23 AM UTC. 2026-02-25T08:40:31.733 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:00:01 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:31.758 INFO:teuthology.orchestra.run.trial023.stdout:Package libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:31.776 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:40:31.777 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:40:31.777 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:40:31.799 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install libcephfs-devel 2026-02-25T08:40:31.800 INFO:teuthology.orchestra.run.trial030.stdout:Package ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:31.816 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:40:31.817 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:40:31.817 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:40:31.839 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install ceph 2026-02-25T08:40:32.116 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:00:09 ago on Wed 25 Feb 2026 08:40:23 AM UTC. 2026-02-25T08:40:32.155 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:00:02 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:32.181 INFO:teuthology.orchestra.run.trial023.stdout:Package libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:32.199 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:40:32.200 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:40:32.200 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:40:32.221 INFO:teuthology.orchestra.run.trial030.stdout:Package ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:32.222 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install librados2 2026-02-25T08:40:32.238 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:40:32.239 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:40:32.239 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:40:32.260 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install ceph-base 2026-02-25T08:40:32.540 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:00:09 ago on Wed 25 Feb 2026 08:40:23 AM UTC. 2026-02-25T08:40:32.574 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:00:02 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:32.606 INFO:teuthology.orchestra.run.trial023.stdout:Package librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:32.623 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:40:32.624 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:40:32.624 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:40:32.639 INFO:teuthology.orchestra.run.trial030.stdout:Package ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:32.646 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install librbd1 2026-02-25T08:40:32.657 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:40:32.658 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:40:32.658 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:40:32.678 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install cephadm 2026-02-25T08:40:32.963 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:00:09 ago on Wed 25 Feb 2026 08:40:23 AM UTC. 2026-02-25T08:40:32.996 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:00:02 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:33.029 INFO:teuthology.orchestra.run.trial023.stdout:Package librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:33.046 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:40:33.047 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:40:33.047 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:40:33.061 INFO:teuthology.orchestra.run.trial030.stdout:Package cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:33.069 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install python3-rados 2026-02-25T08:40:33.079 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:40:33.079 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:40:33.079 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:40:33.100 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install ceph-immutable-object-cache 2026-02-25T08:40:33.391 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:00:10 ago on Wed 25 Feb 2026 08:40:23 AM UTC. 2026-02-25T08:40:33.424 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:00:03 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:33.457 INFO:teuthology.orchestra.run.trial023.stdout:Package python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:33.475 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:40:33.476 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:40:33.476 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:40:33.489 INFO:teuthology.orchestra.run.trial030.stdout:Package ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:33.498 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install python3-rgw 2026-02-25T08:40:33.506 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:40:33.507 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:40:33.507 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:40:33.529 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install ceph-mgr 2026-02-25T08:40:33.812 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:00:10 ago on Wed 25 Feb 2026 08:40:23 AM UTC. 2026-02-25T08:40:33.847 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:00:03 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:33.879 INFO:teuthology.orchestra.run.trial023.stdout:Package python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:33.897 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:40:33.898 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:40:33.898 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:40:33.912 INFO:teuthology.orchestra.run.trial030.stdout:Package ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:33.919 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install python3-cephfs 2026-02-25T08:40:33.930 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:40:33.930 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:40:33.931 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:40:33.952 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install ceph-mgr-dashboard 2026-02-25T08:40:34.235 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:00:11 ago on Wed 25 Feb 2026 08:40:23 AM UTC. 2026-02-25T08:40:34.281 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:00:04 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:34.300 INFO:teuthology.orchestra.run.trial023.stdout:Package python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:34.318 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:40:34.318 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:40:34.318 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:40:34.341 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install python3-rbd 2026-02-25T08:40:34.346 INFO:teuthology.orchestra.run.trial030.stdout:Package ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:34.364 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:40:34.364 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:40:34.365 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:40:34.387 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-25T08:40:34.656 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:00:11 ago on Wed 25 Feb 2026 08:40:23 AM UTC. 2026-02-25T08:40:34.709 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:00:04 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:34.722 INFO:teuthology.orchestra.run.trial023.stdout:Package python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:34.739 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:40:34.740 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:40:34.740 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:40:34.762 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install rbd-fuse 2026-02-25T08:40:34.774 INFO:teuthology.orchestra.run.trial030.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:34.792 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:40:34.792 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:40:34.792 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:40:34.814 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install ceph-mgr-rook 2026-02-25T08:40:35.081 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:00:12 ago on Wed 25 Feb 2026 08:40:23 AM UTC. 2026-02-25T08:40:35.132 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:00:05 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:35.149 INFO:teuthology.orchestra.run.trial023.stdout:Package rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:35.166 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:40:35.167 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:40:35.167 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:40:35.190 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install rbd-mirror 2026-02-25T08:40:35.200 INFO:teuthology.orchestra.run.trial030.stdout:Package ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:35.217 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:40:35.218 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:40:35.218 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:40:35.240 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install ceph-mgr-cephadm 2026-02-25T08:40:35.505 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:00:12 ago on Wed 25 Feb 2026 08:40:23 AM UTC. 2026-02-25T08:40:35.554 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:00:05 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:35.571 INFO:teuthology.orchestra.run.trial023.stdout:Package rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:35.589 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:40:35.589 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:40:35.589 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:40:35.611 DEBUG:teuthology.orchestra.run.trial023:> sudo yum -y install rbd-nbd 2026-02-25T08:40:35.620 INFO:teuthology.orchestra.run.trial030.stdout:Package ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:35.637 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:40:35.638 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:40:35.638 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:40:35.659 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install ceph-osd 2026-02-25T08:40:35.928 INFO:teuthology.orchestra.run.trial023.stdout:Last metadata expiration check: 0:00:12 ago on Wed 25 Feb 2026 08:40:23 AM UTC. 2026-02-25T08:40:35.973 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:00:05 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:35.995 INFO:teuthology.orchestra.run.trial023.stdout:Package rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:36.012 INFO:teuthology.orchestra.run.trial023.stdout:Dependencies resolved. 2026-02-25T08:40:36.013 INFO:teuthology.orchestra.run.trial023.stdout:Nothing to do. 2026-02-25T08:40:36.013 INFO:teuthology.orchestra.run.trial023.stdout:Complete! 2026-02-25T08:40:36.038 DEBUG:teuthology.parallel:result is None 2026-02-25T08:40:36.039 INFO:teuthology.orchestra.run.trial030.stdout:Package ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:36.056 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:40:36.057 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:40:36.057 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:40:36.078 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install ceph-osd-classic 2026-02-25T08:40:36.399 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:00:06 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:36.465 INFO:teuthology.orchestra.run.trial030.stdout:Package ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:36.483 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:40:36.484 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:40:36.484 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:40:36.505 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install ceph-fuse 2026-02-25T08:40:36.822 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:00:06 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:36.887 INFO:teuthology.orchestra.run.trial030.stdout:Package ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:36.905 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:40:36.905 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:40:36.905 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:40:36.928 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install ceph-volume 2026-02-25T08:40:37.246 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:00:07 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:37.312 INFO:teuthology.orchestra.run.trial030.stdout:Package ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:37.329 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:40:37.330 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:40:37.330 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:40:37.352 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install librados-devel 2026-02-25T08:40:37.669 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:00:07 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:37.734 INFO:teuthology.orchestra.run.trial030.stdout:Package librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:37.752 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:40:37.753 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:40:37.753 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:40:37.774 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install libcephfs2 2026-02-25T08:40:38.098 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:00:08 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:38.163 INFO:teuthology.orchestra.run.trial030.stdout:Package libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:38.181 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:40:38.181 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:40:38.182 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:40:38.203 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install libcephfs-devel 2026-02-25T08:40:38.522 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:00:08 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:38.588 INFO:teuthology.orchestra.run.trial030.stdout:Package libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:38.605 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:40:38.606 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:40:38.606 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:40:38.628 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install librados2 2026-02-25T08:40:38.949 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:00:08 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:39.014 INFO:teuthology.orchestra.run.trial030.stdout:Package librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:39.033 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:40:39.033 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:40:39.034 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:40:39.056 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install librbd1 2026-02-25T08:40:39.382 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:00:09 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:39.447 INFO:teuthology.orchestra.run.trial030.stdout:Package librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:39.465 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:40:39.465 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:40:39.465 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:40:39.488 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install python3-rados 2026-02-25T08:40:39.809 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:00:09 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:39.875 INFO:teuthology.orchestra.run.trial030.stdout:Package python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:39.892 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:40:39.893 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:40:39.893 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:40:39.914 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install python3-rgw 2026-02-25T08:40:40.231 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:00:10 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:40.296 INFO:teuthology.orchestra.run.trial030.stdout:Package python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:40.313 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:40:40.314 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:40:40.314 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:40:40.336 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install python3-cephfs 2026-02-25T08:40:40.652 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:00:10 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:40.719 INFO:teuthology.orchestra.run.trial030.stdout:Package python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:40.736 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:40:40.737 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:40:40.737 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:40:40.760 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install python3-rbd 2026-02-25T08:40:41.077 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:00:11 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:41.144 INFO:teuthology.orchestra.run.trial030.stdout:Package python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:41.162 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:40:41.162 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:40:41.162 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:40:41.183 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install rbd-fuse 2026-02-25T08:40:41.501 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:00:11 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:41.567 INFO:teuthology.orchestra.run.trial030.stdout:Package rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:41.585 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:40:41.585 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:40:41.585 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:40:41.607 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install rbd-mirror 2026-02-25T08:40:41.924 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:00:11 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:41.989 INFO:teuthology.orchestra.run.trial030.stdout:Package rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:42.007 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:40:42.008 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:40:42.008 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:40:42.029 DEBUG:teuthology.orchestra.run.trial030:> sudo yum -y install rbd-nbd 2026-02-25T08:40:42.347 INFO:teuthology.orchestra.run.trial030.stdout:Last metadata expiration check: 0:00:12 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:42.414 INFO:teuthology.orchestra.run.trial030.stdout:Package rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:42.432 INFO:teuthology.orchestra.run.trial030.stdout:Dependencies resolved. 2026-02-25T08:40:42.432 INFO:teuthology.orchestra.run.trial030.stdout:Nothing to do. 2026-02-25T08:40:42.432 INFO:teuthology.orchestra.run.trial030.stdout:Complete! 2026-02-25T08:40:42.458 DEBUG:teuthology.parallel:result is None 2026-02-25T08:40:42.459 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:40:42.630 DEBUG:teuthology.orchestra.run.trial023:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-25T08:40:42.650 INFO:teuthology.orchestra.run.trial023.stdout:20.3.0-5338.g6aeb6975.el9 2026-02-25T08:40:42.651 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5338.g6aeb6975.el9 2026-02-25T08:40:42.651 INFO:teuthology.task.install:The correct ceph version 20.3.0-5338.g6aeb6975 is installed. 2026-02-25T08:40:42.654 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:40:42.813 DEBUG:teuthology.orchestra.run.trial030:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-25T08:40:42.834 INFO:teuthology.orchestra.run.trial030.stdout:20.3.0-5338.g6aeb6975.el9 2026-02-25T08:40:42.834 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5338.g6aeb6975.el9 2026-02-25T08:40:42.834 INFO:teuthology.task.install:The correct ceph version 20.3.0-5338.g6aeb6975 is installed. 2026-02-25T08:40:42.835 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2026-02-25T08:40:42.835 INFO:teuthology.run_tasks:Running task ceph... 2026-02-25T08:40:42.908 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-25T08:40:42.908 DEBUG:teuthology.orchestra.run.trial023:> sudo chmod 777 /var/log/ceph 2026-02-25T08:40:42.910 DEBUG:teuthology.orchestra.run.trial030:> sudo chmod 777 /var/log/ceph 2026-02-25T08:40:42.934 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-25T08:40:42.935 DEBUG:teuthology.orchestra.run.trial023:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-25T08:40:42.975 DEBUG:teuthology.orchestra.run.trial030:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-25T08:40:42.997 INFO:tasks.ceph:Creating extra log directories... 2026-02-25T08:40:42.997 DEBUG:teuthology.orchestra.run.trial023:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-25T08:40:43.038 DEBUG:teuthology.orchestra.run.trial030:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-25T08:40:43.076 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-25T08:40:43.076 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd target transaction size': 5, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Degraded data redundancy', 'is down', 'osds down', 'pg .*? is .*?degraded.*?, acting', 'pg .*? is stuck', 'pg degraded', 'PG_BACKFILL_FULL', 'Low space hindering backfill .*? backfill_toofull', 'OSD_ROOT_DOWN', 'pgs degraded', 'pgs undersized', 'is active.*backfill_toofull.*'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-25T08:40:43.076 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69065', 'branch': 'wip-logstoremanager', 'description': 'crimson-rados/thrash/{0-crimson_install 0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} clusters/{crimson-fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm} thrashers/{default} thrashosds-health workloads/snaps-few-objects}', 'email': None, 'first_in_suite': False, 'flavor': 'debug', 'job_id': '69065', 'kernel': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'trial', 'name': 'matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-logstoremanager'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd target transaction size': 5, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Degraded data redundancy', 'is down', 'osds down', 'pg .*? is .*?degraded.*?, acting', 'pg .*? is stuck', 'pg degraded', 'PG_BACKFILL_FULL', 'Low space hindering backfill .*? backfill_toofull', 'OSD_ROOT_DOWN', 'pgs degraded', 'pgs undersized', 'is active.*backfill_toofull.*'], 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'c32469671c6b0222f786dd36c34eb702f780a11d'}}, 'owner': 'scheduled_matan@soko04.front.sepia.ceph.com', 'priority': 70, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'], ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 869, 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': 'c32469671c6b0222f786dd36c34eb702f780a11d', 'targets': {'trial023.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPnROBW7BbzJe4h6dtZU3g748aCEt0GJxM7VF4d+cj+2EPpPYDUSYz9DHlYNwhr/BpHN+2GfrLzLQfif2W2ABr4=', 'trial030.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGREmmmHSQHT+Z/Iwdj68qbUv3TPtliFzc5BI+FC4YfWOexKFBz7/N0pQrz5NA70I/5YxjJjgdGObk99vDbrzlA='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}}, {'pexec': {'all': ['sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson']}}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd target transaction size': 5, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Degraded data redundancy', 'is down', 'osds down', 'pg .*? is .*?degraded.*?, acting', 'pg .*? is stuck', 'pg degraded', 'PG_BACKFILL_FULL', 'Low space hindering backfill .*? backfill_toofull', 'OSD_ROOT_DOWN', 'pgs degraded', 'pgs undersized', 'is active.*backfill_toofull.*'], 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819', 'cluster': 'ceph'}}, {'thrashosds': {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_trim_stale_osdmaps': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400}}, {'rados': {'clients': ['client.0'], 'objects': 50, 'op_weights': {'copy_from': 0, 'delete': 50, 'read': 100, 'rollback': 0, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'fa17720d0088c3ac28e473468bfc79eeaff5cd38', 'timestamp': '2026-02-25_07:55:35', 'tube': 'trial', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699'} 2026-02-25T08:40:43.077 DEBUG:teuthology.orchestra.run.trial023:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-25T08:40:43.102 DEBUG:teuthology.orchestra.run.trial030:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-25T08:40:43.119 DEBUG:teuthology.orchestra.run.trial023:> sudo install -d -m0777 -- /var/run/ceph 2026-02-25T08:40:43.158 DEBUG:teuthology.orchestra.run.trial030:> sudo install -d -m0777 -- /var/run/ceph 2026-02-25T08:40:43.182 DEBUG:teuthology.orchestra.run.trial023:> set -ex 2026-02-25T08:40:43.182 DEBUG:teuthology.orchestra.run.trial023:> dd if=/scratch_devs of=/dev/stdout 2026-02-25T08:40:43.237 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-25T08:40:43.237 DEBUG:teuthology.orchestra.run.trial023:> stat /dev/vg_nvme/lv_1 2026-02-25T08:40:43.293 INFO:teuthology.orchestra.run.trial023.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-25T08:40:43.294 INFO:teuthology.orchestra.run.trial023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T08:40:43.294 INFO:teuthology.orchestra.run.trial023.stdout:Device: 6h/6d Inode: 1301 Links: 1 2026-02-25T08:40:43.294 INFO:teuthology.orchestra.run.trial023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T08:40:43.294 INFO:teuthology.orchestra.run.trial023.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T08:40:43.294 INFO:teuthology.orchestra.run.trial023.stdout:Access: 2026-02-25 08:18:21.093748567 +0000 2026-02-25T08:40:43.294 INFO:teuthology.orchestra.run.trial023.stdout:Modify: 2026-02-25 08:07:49.599333189 +0000 2026-02-25T08:40:43.294 INFO:teuthology.orchestra.run.trial023.stdout:Change: 2026-02-25 08:07:49.599333189 +0000 2026-02-25T08:40:43.294 INFO:teuthology.orchestra.run.trial023.stdout: Birth: 2026-02-25 08:07:49.599333189 +0000 2026-02-25T08:40:43.295 DEBUG:teuthology.orchestra.run.trial023:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-25T08:40:43.356 INFO:teuthology.orchestra.run.trial023.stderr:1+0 records in 2026-02-25T08:40:43.356 INFO:teuthology.orchestra.run.trial023.stderr:1+0 records out 2026-02-25T08:40:43.357 INFO:teuthology.orchestra.run.trial023.stderr:512 bytes copied, 0.000150018 s, 3.4 MB/s 2026-02-25T08:40:43.358 DEBUG:teuthology.orchestra.run.trial023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-25T08:40:43.415 DEBUG:teuthology.orchestra.run.trial023:> stat /dev/vg_nvme/lv_2 2026-02-25T08:40:43.471 INFO:teuthology.orchestra.run.trial023.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-25T08:40:43.471 INFO:teuthology.orchestra.run.trial023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T08:40:43.472 INFO:teuthology.orchestra.run.trial023.stdout:Device: 6h/6d Inode: 1302 Links: 1 2026-02-25T08:40:43.472 INFO:teuthology.orchestra.run.trial023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T08:40:43.472 INFO:teuthology.orchestra.run.trial023.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T08:40:43.472 INFO:teuthology.orchestra.run.trial023.stdout:Access: 2026-02-25 08:18:21.094748545 +0000 2026-02-25T08:40:43.472 INFO:teuthology.orchestra.run.trial023.stdout:Modify: 2026-02-25 08:07:49.599333189 +0000 2026-02-25T08:40:43.472 INFO:teuthology.orchestra.run.trial023.stdout:Change: 2026-02-25 08:07:49.599333189 +0000 2026-02-25T08:40:43.472 INFO:teuthology.orchestra.run.trial023.stdout: Birth: 2026-02-25 08:07:49.599333189 +0000 2026-02-25T08:40:43.472 DEBUG:teuthology.orchestra.run.trial023:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-25T08:40:43.534 INFO:teuthology.orchestra.run.trial023.stderr:1+0 records in 2026-02-25T08:40:43.534 INFO:teuthology.orchestra.run.trial023.stderr:1+0 records out 2026-02-25T08:40:43.534 INFO:teuthology.orchestra.run.trial023.stderr:512 bytes copied, 0.000144618 s, 3.5 MB/s 2026-02-25T08:40:43.535 DEBUG:teuthology.orchestra.run.trial023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-25T08:40:43.592 DEBUG:teuthology.orchestra.run.trial023:> stat /dev/vg_nvme/lv_3 2026-02-25T08:40:43.648 INFO:teuthology.orchestra.run.trial023.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-25T08:40:43.649 INFO:teuthology.orchestra.run.trial023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T08:40:43.649 INFO:teuthology.orchestra.run.trial023.stdout:Device: 6h/6d Inode: 1318 Links: 1 2026-02-25T08:40:43.649 INFO:teuthology.orchestra.run.trial023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T08:40:43.649 INFO:teuthology.orchestra.run.trial023.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T08:40:43.649 INFO:teuthology.orchestra.run.trial023.stdout:Access: 2026-02-25 08:18:21.094748545 +0000 2026-02-25T08:40:43.649 INFO:teuthology.orchestra.run.trial023.stdout:Modify: 2026-02-25 08:07:49.599333189 +0000 2026-02-25T08:40:43.649 INFO:teuthology.orchestra.run.trial023.stdout:Change: 2026-02-25 08:07:49.599333189 +0000 2026-02-25T08:40:43.649 INFO:teuthology.orchestra.run.trial023.stdout: Birth: 2026-02-25 08:07:49.599333189 +0000 2026-02-25T08:40:43.650 DEBUG:teuthology.orchestra.run.trial023:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-25T08:40:43.710 INFO:teuthology.orchestra.run.trial023.stderr:1+0 records in 2026-02-25T08:40:43.710 INFO:teuthology.orchestra.run.trial023.stderr:1+0 records out 2026-02-25T08:40:43.711 INFO:teuthology.orchestra.run.trial023.stderr:512 bytes copied, 0.00016178 s, 3.2 MB/s 2026-02-25T08:40:43.712 DEBUG:teuthology.orchestra.run.trial023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-25T08:40:43.769 DEBUG:teuthology.orchestra.run.trial023:> stat /dev/vg_nvme/lv_4 2026-02-25T08:40:43.826 INFO:teuthology.orchestra.run.trial023.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-25T08:40:43.826 INFO:teuthology.orchestra.run.trial023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T08:40:43.826 INFO:teuthology.orchestra.run.trial023.stdout:Device: 6h/6d Inode: 1296 Links: 1 2026-02-25T08:40:43.826 INFO:teuthology.orchestra.run.trial023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T08:40:43.826 INFO:teuthology.orchestra.run.trial023.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T08:40:43.826 INFO:teuthology.orchestra.run.trial023.stdout:Access: 2026-02-25 08:18:21.094748545 +0000 2026-02-25T08:40:43.826 INFO:teuthology.orchestra.run.trial023.stdout:Modify: 2026-02-25 08:07:49.599333189 +0000 2026-02-25T08:40:43.826 INFO:teuthology.orchestra.run.trial023.stdout:Change: 2026-02-25 08:07:49.599333189 +0000 2026-02-25T08:40:43.826 INFO:teuthology.orchestra.run.trial023.stdout: Birth: 2026-02-25 08:07:49.599333189 +0000 2026-02-25T08:40:43.827 DEBUG:teuthology.orchestra.run.trial023:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-25T08:40:43.888 INFO:teuthology.orchestra.run.trial023.stderr:1+0 records in 2026-02-25T08:40:43.888 INFO:teuthology.orchestra.run.trial023.stderr:1+0 records out 2026-02-25T08:40:43.888 INFO:teuthology.orchestra.run.trial023.stderr:512 bytes copied, 0.000186275 s, 2.7 MB/s 2026-02-25T08:40:43.889 DEBUG:teuthology.orchestra.run.trial023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-25T08:40:43.946 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2026-02-25T08:40:43.946 DEBUG:teuthology.orchestra.run.trial030:> set -ex 2026-02-25T08:40:43.946 DEBUG:teuthology.orchestra.run.trial030:> dd if=/scratch_devs of=/dev/stdout 2026-02-25T08:40:43.965 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-25T08:40:43.965 DEBUG:teuthology.orchestra.run.trial030:> stat /dev/vg_nvme/lv_1 2026-02-25T08:40:44.021 INFO:teuthology.orchestra.run.trial030.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-25T08:40:44.021 INFO:teuthology.orchestra.run.trial030.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T08:40:44.021 INFO:teuthology.orchestra.run.trial030.stdout:Device: 6h/6d Inode: 1299 Links: 1 2026-02-25T08:40:44.021 INFO:teuthology.orchestra.run.trial030.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T08:40:44.021 INFO:teuthology.orchestra.run.trial030.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T08:40:44.021 INFO:teuthology.orchestra.run.trial030.stdout:Access: 2026-02-25 08:21:55.690162554 +0000 2026-02-25T08:40:44.021 INFO:teuthology.orchestra.run.trial030.stdout:Modify: 2026-02-25 08:07:49.321963631 +0000 2026-02-25T08:40:44.022 INFO:teuthology.orchestra.run.trial030.stdout:Change: 2026-02-25 08:07:49.321963631 +0000 2026-02-25T08:40:44.022 INFO:teuthology.orchestra.run.trial030.stdout: Birth: 2026-02-25 08:07:49.321963631 +0000 2026-02-25T08:40:44.022 DEBUG:teuthology.orchestra.run.trial030:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-25T08:40:44.084 INFO:teuthology.orchestra.run.trial030.stderr:1+0 records in 2026-02-25T08:40:44.085 INFO:teuthology.orchestra.run.trial030.stderr:1+0 records out 2026-02-25T08:40:44.085 INFO:teuthology.orchestra.run.trial030.stderr:512 bytes copied, 0.000173682 s, 2.9 MB/s 2026-02-25T08:40:44.086 DEBUG:teuthology.orchestra.run.trial030:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-25T08:40:44.144 DEBUG:teuthology.orchestra.run.trial030:> stat /dev/vg_nvme/lv_2 2026-02-25T08:40:44.200 INFO:teuthology.orchestra.run.trial030.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-25T08:40:44.200 INFO:teuthology.orchestra.run.trial030.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T08:40:44.200 INFO:teuthology.orchestra.run.trial030.stdout:Device: 6h/6d Inode: 1310 Links: 1 2026-02-25T08:40:44.200 INFO:teuthology.orchestra.run.trial030.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T08:40:44.200 INFO:teuthology.orchestra.run.trial030.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T08:40:44.200 INFO:teuthology.orchestra.run.trial030.stdout:Access: 2026-02-25 08:21:55.690162554 +0000 2026-02-25T08:40:44.200 INFO:teuthology.orchestra.run.trial030.stdout:Modify: 2026-02-25 08:07:49.322963603 +0000 2026-02-25T08:40:44.200 INFO:teuthology.orchestra.run.trial030.stdout:Change: 2026-02-25 08:07:49.322963603 +0000 2026-02-25T08:40:44.201 INFO:teuthology.orchestra.run.trial030.stdout: Birth: 2026-02-25 08:07:49.322963603 +0000 2026-02-25T08:40:44.201 DEBUG:teuthology.orchestra.run.trial030:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-25T08:40:44.262 INFO:teuthology.orchestra.run.trial030.stderr:1+0 records in 2026-02-25T08:40:44.262 INFO:teuthology.orchestra.run.trial030.stderr:1+0 records out 2026-02-25T08:40:44.262 INFO:teuthology.orchestra.run.trial030.stderr:512 bytes copied, 0.000141092 s, 3.6 MB/s 2026-02-25T08:40:44.263 DEBUG:teuthology.orchestra.run.trial030:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-25T08:40:44.320 DEBUG:teuthology.orchestra.run.trial030:> stat /dev/vg_nvme/lv_3 2026-02-25T08:40:44.377 INFO:teuthology.orchestra.run.trial030.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-25T08:40:44.377 INFO:teuthology.orchestra.run.trial030.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T08:40:44.377 INFO:teuthology.orchestra.run.trial030.stdout:Device: 6h/6d Inode: 1294 Links: 1 2026-02-25T08:40:44.377 INFO:teuthology.orchestra.run.trial030.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T08:40:44.377 INFO:teuthology.orchestra.run.trial030.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T08:40:44.377 INFO:teuthology.orchestra.run.trial030.stdout:Access: 2026-02-25 08:21:55.690162554 +0000 2026-02-25T08:40:44.377 INFO:teuthology.orchestra.run.trial030.stdout:Modify: 2026-02-25 08:07:49.321963631 +0000 2026-02-25T08:40:44.377 INFO:teuthology.orchestra.run.trial030.stdout:Change: 2026-02-25 08:07:49.321963631 +0000 2026-02-25T08:40:44.378 INFO:teuthology.orchestra.run.trial030.stdout: Birth: 2026-02-25 08:07:49.321963631 +0000 2026-02-25T08:40:44.378 DEBUG:teuthology.orchestra.run.trial030:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-25T08:40:44.441 INFO:teuthology.orchestra.run.trial030.stderr:1+0 records in 2026-02-25T08:40:44.441 INFO:teuthology.orchestra.run.trial030.stderr:1+0 records out 2026-02-25T08:40:44.441 INFO:teuthology.orchestra.run.trial030.stderr:512 bytes copied, 0.000218235 s, 2.3 MB/s 2026-02-25T08:40:44.442 DEBUG:teuthology.orchestra.run.trial030:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-25T08:40:44.499 DEBUG:teuthology.orchestra.run.trial030:> stat /dev/vg_nvme/lv_4 2026-02-25T08:40:44.556 INFO:teuthology.orchestra.run.trial030.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-25T08:40:44.556 INFO:teuthology.orchestra.run.trial030.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T08:40:44.557 INFO:teuthology.orchestra.run.trial030.stdout:Device: 6h/6d Inode: 1298 Links: 1 2026-02-25T08:40:44.557 INFO:teuthology.orchestra.run.trial030.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T08:40:44.557 INFO:teuthology.orchestra.run.trial030.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T08:40:44.557 INFO:teuthology.orchestra.run.trial030.stdout:Access: 2026-02-25 08:21:55.691162523 +0000 2026-02-25T08:40:44.557 INFO:teuthology.orchestra.run.trial030.stdout:Modify: 2026-02-25 08:07:49.321963631 +0000 2026-02-25T08:40:44.557 INFO:teuthology.orchestra.run.trial030.stdout:Change: 2026-02-25 08:07:49.321963631 +0000 2026-02-25T08:40:44.557 INFO:teuthology.orchestra.run.trial030.stdout: Birth: 2026-02-25 08:07:49.321963631 +0000 2026-02-25T08:40:44.557 DEBUG:teuthology.orchestra.run.trial030:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-25T08:40:44.620 INFO:teuthology.orchestra.run.trial030.stderr:1+0 records in 2026-02-25T08:40:44.620 INFO:teuthology.orchestra.run.trial030.stderr:1+0 records out 2026-02-25T08:40:44.620 INFO:teuthology.orchestra.run.trial030.stderr:512 bytes copied, 0.000228946 s, 2.2 MB/s 2026-02-25T08:40:44.621 DEBUG:teuthology.orchestra.run.trial030:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-25T08:40:44.678 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2026-02-25T08:40:44.678 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial023.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@trial030.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2026-02-25T08:40:44.678 INFO:tasks.ceph:Generating config... 2026-02-25T08:40:44.681 INFO:tasks.ceph:[osd] debug monc = 20 2026-02-25T08:40:44.681 INFO:tasks.ceph:[osd] crimson cpu num = 4 2026-02-25T08:40:44.681 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2026-02-25T08:40:44.681 INFO:tasks.ceph:[osd] debug ms = 20 2026-02-25T08:40:44.681 INFO:tasks.ceph:[osd] debug osd = 20 2026-02-25T08:40:44.681 INFO:tasks.ceph:[osd] debug seastore = 20 2026-02-25T08:40:44.682 INFO:tasks.ceph:[osd] debug seastore backref = 20 2026-02-25T08:40:44.682 INFO:tasks.ceph:[osd] debug seastore cache = 20 2026-02-25T08:40:44.682 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2026-02-25T08:40:44.682 INFO:tasks.ceph:[osd] debug seastore device = 20 2026-02-25T08:40:44.682 INFO:tasks.ceph:[osd] debug seastore epm = 20 2026-02-25T08:40:44.682 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2026-02-25T08:40:44.682 INFO:tasks.ceph:[osd] debug seastore journal = 20 2026-02-25T08:40:44.682 INFO:tasks.ceph:[osd] debug seastore lba = 20 2026-02-25T08:40:44.682 INFO:tasks.ceph:[osd] debug seastore odata = 20 2026-02-25T08:40:44.682 INFO:tasks.ceph:[osd] debug seastore omap = 20 2026-02-25T08:40:44.682 INFO:tasks.ceph:[osd] debug seastore onode = 20 2026-02-25T08:40:44.682 INFO:tasks.ceph:[osd] debug seastore t = 20 2026-02-25T08:40:44.682 INFO:tasks.ceph:[osd] debug seastore tm = 20 2026-02-25T08:40:44.683 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2026-02-25T08:40:44.683 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2026-02-25T08:40:44.683 INFO:tasks.ceph:[osd] osd delete sleep = 1 2026-02-25T08:40:44.683 INFO:tasks.ceph:[osd] osd max backfills = 3 2026-02-25T08:40:44.683 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2026-02-25T08:40:44.683 INFO:tasks.ceph:[osd] osd objectstore = seastore 2026-02-25T08:40:44.683 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2026-02-25T08:40:44.683 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2026-02-25T08:40:44.683 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2026-02-25T08:40:44.683 INFO:tasks.ceph:[osd] osd snap trim sleep = 2 2026-02-25T08:40:44.683 INFO:tasks.ceph:[osd] osd target transaction size = 5 2026-02-25T08:40:44.683 INFO:tasks.ceph:[osd] seastore cachepin size pershard = 64M 2026-02-25T08:40:44.684 INFO:tasks.ceph:[osd] seastore main device type = RANDOM_BLOCK_SSD 2026-02-25T08:40:44.684 INFO:tasks.ceph:[osd] seastore max concurrent transactions = 8 2026-02-25T08:40:44.684 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2026-02-25T08:40:44.684 INFO:tasks.ceph:[global] ms client mode = crc 2026-02-25T08:40:44.684 INFO:tasks.ceph:[global] ms cluster mode = crc 2026-02-25T08:40:44.684 INFO:tasks.ceph:[global] ms mon client mode = crc 2026-02-25T08:40:44.684 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2026-02-25T08:40:44.684 INFO:tasks.ceph:[global] ms mon service mode = crc 2026-02-25T08:40:44.684 INFO:tasks.ceph:[global] ms service mode = crc 2026-02-25T08:40:44.684 INFO:tasks.ceph:[global] osd_max_pg_log_entries = 2 2026-02-25T08:40:44.684 INFO:tasks.ceph:[global] osd_min_pg_log_entries = 1 2026-02-25T08:40:44.684 INFO:tasks.ceph:[global] osd_pg_log_trim_min = 0 2026-02-25T08:40:44.684 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2026-02-25T08:40:44.685 INFO:tasks.ceph:[global] osd_pool_default_size = 3 2026-02-25T08:40:44.685 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-25T08:40:44.685 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-25T08:40:44.685 INFO:tasks.ceph:[mon] debug mon = 20 2026-02-25T08:40:44.685 INFO:tasks.ceph:[mon] debug ms = 1 2026-02-25T08:40:44.685 INFO:tasks.ceph:[mon] debug paxos = 20 2026-02-25T08:40:44.685 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2026-02-25T08:40:44.685 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2026-02-25T08:40:44.685 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2026-02-25T08:40:44.685 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2026-02-25T08:40:44.685 INFO:tasks.ceph:[mon] osd pool default crimson = True 2026-02-25T08:40:44.685 INFO:tasks.ceph:[mon] paxos service trim min = 10 2026-02-25T08:40:44.686 INFO:tasks.ceph:Setting up mon.a... 2026-02-25T08:40:44.686 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-25T08:40:44.729 INFO:teuthology.orchestra.run.trial023.stdout:creating /etc/ceph/ceph.keyring 2026-02-25T08:40:44.732 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-25T08:40:44.815 DEBUG:teuthology.orchestra.run.trial023:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-25T08:40:44.879 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.23')] 2026-02-25T08:40:44.880 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.23', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': 'true', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd target transaction size': 5, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2026-02-25T08:40:44.882 DEBUG:teuthology.orchestra.run.trial023:> set -ex 2026-02-25T08:40:44.882 DEBUG:teuthology.orchestra.run.trial023:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-25T08:40:44.935 DEBUG:teuthology.orchestra.run.trial023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.23 --print /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:40:45.015 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:40:45.014+0000 7f7b05432f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:45.016 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:40:45.014+0000 7f7b05432f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:45.016 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:40:45.014+0000 7f7b05432f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:45.016 INFO:teuthology.orchestra.run.trial023.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:40:45.016 INFO:teuthology.orchestra.run.trial023.stdout:monmaptool: generated fsid 542a2d4e-0dbc-4a03-9358-454614077093 2026-02-25T08:40:45.016 INFO:teuthology.orchestra.run.trial023.stdout:setting min_mon_release = tentacle 2026-02-25T08:40:45.016 INFO:teuthology.orchestra.run.trial023.stdout:epoch 0 2026-02-25T08:40:45.016 INFO:teuthology.orchestra.run.trial023.stdout:fsid 542a2d4e-0dbc-4a03-9358-454614077093 2026-02-25T08:40:45.016 INFO:teuthology.orchestra.run.trial023.stdout:last_changed 2026-02-25T08:40:45.015505+0000 2026-02-25T08:40:45.016 INFO:teuthology.orchestra.run.trial023.stdout:created 2026-02-25T08:40:45.015505+0000 2026-02-25T08:40:45.016 INFO:teuthology.orchestra.run.trial023.stdout:min_mon_release 20 (tentacle) 2026-02-25T08:40:45.016 INFO:teuthology.orchestra.run.trial023.stdout:election_strategy: 1 2026-02-25T08:40:45.016 INFO:teuthology.orchestra.run.trial023.stdout:0: [v2:10.20.193.23:3300/0,v1:10.20.193.23:6789/0] mon.a 2026-02-25T08:40:45.016 INFO:teuthology.orchestra.run.trial023.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2026-02-25T08:40:45.017 DEBUG:teuthology.orchestra.run.trial023:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-25T08:40:45.074 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 542a2d4e-0dbc-4a03-9358-454614077093... 2026-02-25T08:40:45.075 DEBUG:teuthology.orchestra.run.trial023:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-25T08:40:45.116 DEBUG:teuthology.orchestra.run.trial030:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-25T08:40:45.154 INFO:teuthology.orchestra.run.trial023.stdout:[global] 2026-02-25T08:40:45.154 INFO:teuthology.orchestra.run.trial023.stdout: chdir = "" 2026-02-25T08:40:45.155 INFO:teuthology.orchestra.run.trial023.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-25T08:40:45.155 INFO:teuthology.orchestra.run.trial023.stdout: auth supported = cephx 2026-02-25T08:40:45.155 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:40:45.155 INFO:teuthology.orchestra.run.trial023.stdout: filestore xattr use omap = true 2026-02-25T08:40:45.155 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:40:45.155 INFO:teuthology.orchestra.run.trial023.stdout: mon clock drift allowed = 1.000 2026-02-25T08:40:45.155 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:40:45.155 INFO:teuthology.orchestra.run.trial023.stdout: osd crush chooseleaf type = 0 2026-02-25T08:40:45.155 INFO:teuthology.orchestra.run.trial023.stdout: auth debug = true 2026-02-25T08:40:45.155 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:40:45.156 INFO:teuthology.orchestra.run.trial023.stdout: ms die on old message = true 2026-02-25T08:40:45.156 INFO:teuthology.orchestra.run.trial023.stdout: ms die on bug = true 2026-02-25T08:40:45.156 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:40:45.156 INFO:teuthology.orchestra.run.trial023.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-25T08:40:45.156 INFO:teuthology.orchestra.run.trial023.stdout: mon pg warn max object skew = 0 2026-02-25T08:40:45.156 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:40:45.156 INFO:teuthology.orchestra.run.trial023.stdout: # disable pg_autoscaler by default for new pools 2026-02-25T08:40:45.156 INFO:teuthology.orchestra.run.trial023.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-25T08:40:45.156 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:40:45.156 INFO:teuthology.orchestra.run.trial023.stdout: osd pool default size = 2 2026-02-25T08:40:45.156 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:40:45.156 INFO:teuthology.orchestra.run.trial023.stdout: mon osd allow primary affinity = true 2026-02-25T08:40:45.157 INFO:teuthology.orchestra.run.trial023.stdout: mon osd allow pg remap = true 2026-02-25T08:40:45.157 INFO:teuthology.orchestra.run.trial023.stdout: mon warn on legacy crush tunables = false 2026-02-25T08:40:45.157 INFO:teuthology.orchestra.run.trial023.stdout: mon warn on crush straw calc version zero = false 2026-02-25T08:40:45.157 INFO:teuthology.orchestra.run.trial023.stdout: mon warn on no sortbitwise = false 2026-02-25T08:40:45.157 INFO:teuthology.orchestra.run.trial023.stdout: mon warn on osd down out interval zero = false 2026-02-25T08:40:45.157 INFO:teuthology.orchestra.run.trial023.stdout: mon warn on too few osds = false 2026-02-25T08:40:45.157 INFO:teuthology.orchestra.run.trial023.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-25T08:40:45.157 INFO:teuthology.orchestra.run.trial023.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-25T08:40:45.157 INFO:teuthology.orchestra.run.trial023.stdout: mon_allow_pool_size_one = true 2026-02-25T08:40:45.157 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:40:45.157 INFO:teuthology.orchestra.run.trial023.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-25T08:40:45.157 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:40:45.158 INFO:teuthology.orchestra.run.trial023.stdout: osd default data pool replay window = 5 2026-02-25T08:40:45.158 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:40:45.158 INFO:teuthology.orchestra.run.trial023.stdout: mon allow pool delete = true 2026-02-25T08:40:45.158 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:40:45.158 INFO:teuthology.orchestra.run.trial023.stdout: mon cluster log file level = debug 2026-02-25T08:40:45.158 INFO:teuthology.orchestra.run.trial023.stdout: debug asserts on shutdown = true 2026-02-25T08:40:45.158 INFO:teuthology.orchestra.run.trial023.stdout: mon health detail to clog = false 2026-02-25T08:40:45.158 INFO:teuthology.orchestra.run.trial023.stdout: mon host = 10.20.193.23 2026-02-25T08:40:45.158 INFO:teuthology.orchestra.run.trial023.stdout: enable experimental unrecoverable data corrupting features = crimson 2026-02-25T08:40:45.158 INFO:teuthology.orchestra.run.trial023.stdout: ms client mode = crc 2026-02-25T08:40:45.158 INFO:teuthology.orchestra.run.trial023.stdout: ms cluster mode = crc 2026-02-25T08:40:45.158 INFO:teuthology.orchestra.run.trial023.stdout: ms mon client mode = crc 2026-02-25T08:40:45.159 INFO:teuthology.orchestra.run.trial023.stdout: ms mon cluster mode = crc 2026-02-25T08:40:45.159 INFO:teuthology.orchestra.run.trial023.stdout: ms mon service mode = crc 2026-02-25T08:40:45.159 INFO:teuthology.orchestra.run.trial023.stdout: ms service mode = crc 2026-02-25T08:40:45.159 INFO:teuthology.orchestra.run.trial023.stdout: osd_max_pg_log_entries = 2 2026-02-25T08:40:45.159 INFO:teuthology.orchestra.run.trial023.stdout: osd_min_pg_log_entries = 1 2026-02-25T08:40:45.159 INFO:teuthology.orchestra.run.trial023.stdout: osd_pg_log_trim_min = 0 2026-02-25T08:40:45.159 INFO:teuthology.orchestra.run.trial023.stdout: osd_pool_default_min_size = 2 2026-02-25T08:40:45.159 INFO:teuthology.orchestra.run.trial023.stdout: osd_pool_default_size = 3 2026-02-25T08:40:45.159 INFO:teuthology.orchestra.run.trial023.stdout: fsid = 542a2d4e-0dbc-4a03-9358-454614077093 2026-02-25T08:40:45.159 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:40:45.159 INFO:teuthology.orchestra.run.trial023.stdout:[osd] 2026-02-25T08:40:45.159 INFO:teuthology.orchestra.run.trial023.stdout: osd journal size = 100 2026-02-25T08:40:45.159 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:40:45.160 INFO:teuthology.orchestra.run.trial023.stdout: osd scrub load threshold = 5.0 2026-02-25T08:40:45.160 INFO:teuthology.orchestra.run.trial023.stdout: osd scrub max interval = 120 2026-02-25T08:40:45.160 INFO:teuthology.orchestra.run.trial023.stdout: osd mclock profile = high_recovery_ops 2026-02-25T08:40:45.160 INFO:teuthology.orchestra.run.trial023.stdout: osd mclock skip benchmark = true 2026-02-25T08:40:45.160 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:40:45.160 INFO:teuthology.orchestra.run.trial023.stdout: osd recover clone overlap = true 2026-02-25T08:40:45.160 INFO:teuthology.orchestra.run.trial023.stdout: osd recovery max chunk = 1048576 2026-02-25T08:40:45.160 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:40:45.160 INFO:teuthology.orchestra.run.trial023.stdout: osd debug shutdown = true 2026-02-25T08:40:45.160 INFO:teuthology.orchestra.run.trial023.stdout: osd debug op order = true 2026-02-25T08:40:45.160 INFO:teuthology.orchestra.run.trial023.stdout: osd debug verify stray on activate = true 2026-02-25T08:40:45.160 INFO:teuthology.orchestra.run.trial023.stdout: osd debug trim objects = true 2026-02-25T08:40:45.160 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:40:45.161 INFO:teuthology.orchestra.run.trial023.stdout: osd open classes on start = true 2026-02-25T08:40:45.161 INFO:teuthology.orchestra.run.trial023.stdout: osd debug pg log writeout = true 2026-02-25T08:40:45.161 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:40:45.161 INFO:teuthology.orchestra.run.trial023.stdout: osd deep scrub update digest min age = 30 2026-02-25T08:40:45.161 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:40:45.161 INFO:teuthology.orchestra.run.trial023.stdout: osd map max advance = 10 2026-02-25T08:40:45.161 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:40:45.161 INFO:teuthology.orchestra.run.trial023.stdout: journal zero on create = true 2026-02-25T08:40:45.161 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:40:45.161 INFO:teuthology.orchestra.run.trial023.stdout: filestore ondisk finisher threads = 3 2026-02-25T08:40:45.161 INFO:teuthology.orchestra.run.trial023.stdout: filestore apply finisher threads = 3 2026-02-25T08:40:45.161 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:40:45.162 INFO:teuthology.orchestra.run.trial023.stdout: bdev debug aio = true 2026-02-25T08:40:45.162 INFO:teuthology.orchestra.run.trial023.stdout: osd debug misdirected ops = true 2026-02-25T08:40:45.162 INFO:teuthology.orchestra.run.trial023.stdout: debug monc = 20 2026-02-25T08:40:45.162 INFO:teuthology.orchestra.run.trial023.stdout: crimson cpu num = 4 2026-02-25T08:40:45.162 INFO:teuthology.orchestra.run.trial023.stdout: crimson osd obc lru size = 10 2026-02-25T08:40:45.162 INFO:teuthology.orchestra.run.trial023.stdout: debug ms = 20 2026-02-25T08:40:45.162 INFO:teuthology.orchestra.run.trial023.stdout: debug osd = 20 2026-02-25T08:40:45.162 INFO:teuthology.orchestra.run.trial023.stdout: debug seastore = 20 2026-02-25T08:40:45.162 INFO:teuthology.orchestra.run.trial023.stdout: debug seastore backref = 20 2026-02-25T08:40:45.162 INFO:teuthology.orchestra.run.trial023.stdout: debug seastore cache = 20 2026-02-25T08:40:45.162 INFO:teuthology.orchestra.run.trial023.stdout: debug seastore cleaner = 20 2026-02-25T08:40:45.162 INFO:teuthology.orchestra.run.trial023.stdout: debug seastore device = 20 2026-02-25T08:40:45.162 INFO:teuthology.orchestra.run.trial023.stdout: debug seastore epm = 20 2026-02-25T08:40:45.163 INFO:teuthology.orchestra.run.trial023.stdout: debug seastore fixedkv tree = 20 2026-02-25T08:40:45.163 INFO:teuthology.orchestra.run.trial023.stdout: debug seastore journal = 20 2026-02-25T08:40:45.163 INFO:teuthology.orchestra.run.trial023.stdout: debug seastore lba = 20 2026-02-25T08:40:45.163 INFO:teuthology.orchestra.run.trial023.stdout: debug seastore odata = 20 2026-02-25T08:40:45.163 INFO:teuthology.orchestra.run.trial023.stdout: debug seastore omap = 20 2026-02-25T08:40:45.163 INFO:teuthology.orchestra.run.trial023.stdout: debug seastore onode = 20 2026-02-25T08:40:45.163 INFO:teuthology.orchestra.run.trial023.stdout: debug seastore t = 20 2026-02-25T08:40:45.163 INFO:teuthology.orchestra.run.trial023.stdout: debug seastore tm = 20 2026-02-25T08:40:45.163 INFO:teuthology.orchestra.run.trial023.stdout: osd blocked scrub grace period = 3600 2026-02-25T08:40:45.163 INFO:teuthology.orchestra.run.trial023.stdout: osd debug reject backfill probability = 0.3 2026-02-25T08:40:45.163 INFO:teuthology.orchestra.run.trial023.stdout: osd delete sleep = 1 2026-02-25T08:40:45.163 INFO:teuthology.orchestra.run.trial023.stdout: osd max backfills = 3 2026-02-25T08:40:45.163 INFO:teuthology.orchestra.run.trial023.stdout: osd max markdown count = 1000 2026-02-25T08:40:45.164 INFO:teuthology.orchestra.run.trial023.stdout: osd objectstore = seastore 2026-02-25T08:40:45.164 INFO:teuthology.orchestra.run.trial023.stdout: osd scrub min interval = 60 2026-02-25T08:40:45.164 INFO:teuthology.orchestra.run.trial023.stdout: osd shutdown pgref assert = True 2026-02-25T08:40:45.164 INFO:teuthology.orchestra.run.trial023.stdout: osd snap trim sleep = 2 2026-02-25T08:40:45.164 INFO:teuthology.orchestra.run.trial023.stdout: osd target transaction size = 5 2026-02-25T08:40:45.164 INFO:teuthology.orchestra.run.trial023.stdout: seastore cachepin size pershard = 64M 2026-02-25T08:40:45.164 INFO:teuthology.orchestra.run.trial023.stdout: seastore main device type = RANDOM_BLOCK_SSD 2026-02-25T08:40:45.164 INFO:teuthology.orchestra.run.trial023.stdout: seastore max concurrent transactions = 8 2026-02-25T08:40:45.164 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:40:45.164 INFO:teuthology.orchestra.run.trial023.stdout:[mgr] 2026-02-25T08:40:45.164 INFO:teuthology.orchestra.run.trial023.stdout: debug ms = 1 2026-02-25T08:40:45.164 INFO:teuthology.orchestra.run.trial023.stdout: debug mgr = 20 2026-02-25T08:40:45.164 INFO:teuthology.orchestra.run.trial023.stdout: debug mon = 20 2026-02-25T08:40:45.165 INFO:teuthology.orchestra.run.trial023.stdout: debug auth = 20 2026-02-25T08:40:45.165 INFO:teuthology.orchestra.run.trial023.stdout: mon reweight min pgs per osd = 4 2026-02-25T08:40:45.165 INFO:teuthology.orchestra.run.trial023.stdout: mon reweight min bytes per osd = 10 2026-02-25T08:40:45.165 INFO:teuthology.orchestra.run.trial023.stdout: mgr/telemetry/nag = false 2026-02-25T08:40:45.165 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:40:45.165 INFO:teuthology.orchestra.run.trial023.stdout:[mon] 2026-02-25T08:40:45.165 INFO:teuthology.orchestra.run.trial023.stdout: debug ms = 1 2026-02-25T08:40:45.165 INFO:teuthology.orchestra.run.trial023.stdout: debug mon = 20 2026-02-25T08:40:45.165 INFO:teuthology.orchestra.run.trial023.stdout: debug paxos = 20 2026-02-25T08:40:45.165 INFO:teuthology.orchestra.run.trial023.stdout: debug auth = 20 2026-02-25T08:40:45.165 INFO:teuthology.orchestra.run.trial023.stdout: mon data avail warn = 5 2026-02-25T08:40:45.165 INFO:teuthology.orchestra.run.trial023.stdout: mon mgr mkfs grace = 240 2026-02-25T08:40:45.165 INFO:teuthology.orchestra.run.trial023.stdout: mon reweight min pgs per osd = 4 2026-02-25T08:40:45.166 INFO:teuthology.orchestra.run.trial023.stdout: mon osd reporter subtree level = osd 2026-02-25T08:40:45.166 INFO:teuthology.orchestra.run.trial023.stdout: mon osd prime pg temp = true 2026-02-25T08:40:45.166 INFO:teuthology.orchestra.run.trial023.stdout: mon reweight min bytes per osd = 10 2026-02-25T08:40:45.166 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:40:45.166 INFO:teuthology.orchestra.run.trial023.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-25T08:40:45.166 INFO:teuthology.orchestra.run.trial023.stdout: auth mon ticket ttl = 660 # 11m 2026-02-25T08:40:45.166 INFO:teuthology.orchestra.run.trial023.stdout: auth service ticket ttl = 240 # 4m 2026-02-25T08:40:45.166 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:40:45.166 INFO:teuthology.orchestra.run.trial023.stdout: # don't complain about insecure global_id in the test suite 2026-02-25T08:40:45.166 INFO:teuthology.orchestra.run.trial023.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-25T08:40:45.166 INFO:teuthology.orchestra.run.trial023.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-25T08:40:45.166 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:40:45.167 INFO:teuthology.orchestra.run.trial023.stdout: # 1m isn't quite enough 2026-02-25T08:40:45.167 INFO:teuthology.orchestra.run.trial023.stdout: mon_down_mkfs_grace = 2m 2026-02-25T08:40:45.167 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:40:45.167 INFO:teuthology.orchestra.run.trial023.stdout: mon_warn_on_filestore_osds = false 2026-02-25T08:40:45.167 INFO:teuthology.orchestra.run.trial023.stdout: mon min osdmap epochs = 50 2026-02-25T08:40:45.167 INFO:teuthology.orchestra.run.trial023.stdout: mon osdmap full prune interval = 2 2026-02-25T08:40:45.167 INFO:teuthology.orchestra.run.trial023.stdout: mon osdmap full prune min = 15 2026-02-25T08:40:45.167 INFO:teuthology.orchestra.run.trial023.stdout: mon osdmap full prune txsize = 2 2026-02-25T08:40:45.167 INFO:teuthology.orchestra.run.trial023.stdout: osd pool default crimson = True 2026-02-25T08:40:45.167 INFO:teuthology.orchestra.run.trial023.stdout: paxos service trim min = 10 2026-02-25T08:40:45.167 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:40:45.167 INFO:teuthology.orchestra.run.trial023.stdout:[client] 2026-02-25T08:40:45.167 INFO:teuthology.orchestra.run.trial023.stdout: rgw cache enabled = true 2026-02-25T08:40:45.168 INFO:teuthology.orchestra.run.trial023.stdout: rgw enable ops log = true 2026-02-25T08:40:45.168 INFO:teuthology.orchestra.run.trial023.stdout: rgw enable usage log = true 2026-02-25T08:40:45.168 INFO:teuthology.orchestra.run.trial023.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-25T08:40:45.168 INFO:teuthology.orchestra.run.trial023.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-25T08:40:45.168 INFO:teuthology.orchestra.run.trial023.stdout:[mon.a] 2026-02-25T08:40:45.170 INFO:teuthology.orchestra.run.trial030.stdout:[global] 2026-02-25T08:40:45.170 INFO:teuthology.orchestra.run.trial030.stdout: chdir = "" 2026-02-25T08:40:45.170 INFO:teuthology.orchestra.run.trial030.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-25T08:40:45.170 INFO:teuthology.orchestra.run.trial030.stdout: auth supported = cephx 2026-02-25T08:40:45.170 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:40:45.170 INFO:teuthology.orchestra.run.trial030.stdout: filestore xattr use omap = true 2026-02-25T08:40:45.170 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:40:45.170 INFO:teuthology.orchestra.run.trial030.stdout: mon clock drift allowed = 1.000 2026-02-25T08:40:45.170 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:40:45.171 INFO:teuthology.orchestra.run.trial030.stdout: osd crush chooseleaf type = 0 2026-02-25T08:40:45.171 INFO:teuthology.orchestra.run.trial030.stdout: auth debug = true 2026-02-25T08:40:45.171 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:40:45.171 INFO:teuthology.orchestra.run.trial030.stdout: ms die on old message = true 2026-02-25T08:40:45.171 INFO:teuthology.orchestra.run.trial030.stdout: ms die on bug = true 2026-02-25T08:40:45.171 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:40:45.171 INFO:teuthology.orchestra.run.trial030.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-25T08:40:45.171 INFO:teuthology.orchestra.run.trial030.stdout: mon pg warn max object skew = 0 2026-02-25T08:40:45.171 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:40:45.171 INFO:teuthology.orchestra.run.trial030.stdout: # disable pg_autoscaler by default for new pools 2026-02-25T08:40:45.171 INFO:teuthology.orchestra.run.trial030.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-25T08:40:45.171 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:40:45.172 INFO:teuthology.orchestra.run.trial030.stdout: osd pool default size = 2 2026-02-25T08:40:45.172 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:40:45.172 INFO:teuthology.orchestra.run.trial030.stdout: mon osd allow primary affinity = true 2026-02-25T08:40:45.172 INFO:teuthology.orchestra.run.trial030.stdout: mon osd allow pg remap = true 2026-02-25T08:40:45.172 INFO:teuthology.orchestra.run.trial030.stdout: mon warn on legacy crush tunables = false 2026-02-25T08:40:45.172 INFO:teuthology.orchestra.run.trial030.stdout: mon warn on crush straw calc version zero = false 2026-02-25T08:40:45.172 INFO:teuthology.orchestra.run.trial030.stdout: mon warn on no sortbitwise = false 2026-02-25T08:40:45.172 INFO:teuthology.orchestra.run.trial030.stdout: mon warn on osd down out interval zero = false 2026-02-25T08:40:45.172 INFO:teuthology.orchestra.run.trial030.stdout: mon warn on too few osds = false 2026-02-25T08:40:45.172 INFO:teuthology.orchestra.run.trial030.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-25T08:40:45.172 INFO:teuthology.orchestra.run.trial030.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-25T08:40:45.172 INFO:teuthology.orchestra.run.trial030.stdout: mon_allow_pool_size_one = true 2026-02-25T08:40:45.172 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:40:45.173 INFO:teuthology.orchestra.run.trial030.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-25T08:40:45.173 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:40:45.173 INFO:teuthology.orchestra.run.trial030.stdout: osd default data pool replay window = 5 2026-02-25T08:40:45.173 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:40:45.173 INFO:teuthology.orchestra.run.trial030.stdout: mon allow pool delete = true 2026-02-25T08:40:45.173 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:40:45.173 INFO:teuthology.orchestra.run.trial030.stdout: mon cluster log file level = debug 2026-02-25T08:40:45.173 INFO:teuthology.orchestra.run.trial030.stdout: debug asserts on shutdown = true 2026-02-25T08:40:45.173 INFO:teuthology.orchestra.run.trial030.stdout: mon health detail to clog = false 2026-02-25T08:40:45.173 INFO:teuthology.orchestra.run.trial030.stdout: mon host = 10.20.193.23 2026-02-25T08:40:45.173 INFO:teuthology.orchestra.run.trial030.stdout: enable experimental unrecoverable data corrupting features = crimson 2026-02-25T08:40:45.173 INFO:teuthology.orchestra.run.trial030.stdout: ms client mode = crc 2026-02-25T08:40:45.174 INFO:teuthology.orchestra.run.trial030.stdout: ms cluster mode = crc 2026-02-25T08:40:45.174 INFO:teuthology.orchestra.run.trial030.stdout: ms mon client mode = crc 2026-02-25T08:40:45.174 INFO:teuthology.orchestra.run.trial030.stdout: ms mon cluster mode = crc 2026-02-25T08:40:45.174 INFO:teuthology.orchestra.run.trial030.stdout: ms mon service mode = crc 2026-02-25T08:40:45.174 INFO:teuthology.orchestra.run.trial030.stdout: ms service mode = crc 2026-02-25T08:40:45.174 INFO:teuthology.orchestra.run.trial030.stdout: osd_max_pg_log_entries = 2 2026-02-25T08:40:45.174 INFO:teuthology.orchestra.run.trial030.stdout: osd_min_pg_log_entries = 1 2026-02-25T08:40:45.174 INFO:teuthology.orchestra.run.trial030.stdout: osd_pg_log_trim_min = 0 2026-02-25T08:40:45.174 INFO:teuthology.orchestra.run.trial030.stdout: osd_pool_default_min_size = 2 2026-02-25T08:40:45.174 INFO:teuthology.orchestra.run.trial030.stdout: osd_pool_default_size = 3 2026-02-25T08:40:45.174 INFO:teuthology.orchestra.run.trial030.stdout: fsid = 542a2d4e-0dbc-4a03-9358-454614077093 2026-02-25T08:40:45.174 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:40:45.174 INFO:teuthology.orchestra.run.trial030.stdout:[osd] 2026-02-25T08:40:45.174 INFO:teuthology.orchestra.run.trial030.stdout: osd journal size = 100 2026-02-25T08:40:45.175 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:40:45.175 INFO:teuthology.orchestra.run.trial030.stdout: osd scrub load threshold = 5.0 2026-02-25T08:40:45.175 INFO:teuthology.orchestra.run.trial030.stdout: osd scrub max interval = 120 2026-02-25T08:40:45.175 INFO:teuthology.orchestra.run.trial030.stdout: osd mclock profile = high_recovery_ops 2026-02-25T08:40:45.175 INFO:teuthology.orchestra.run.trial030.stdout: osd mclock skip benchmark = true 2026-02-25T08:40:45.175 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:40:45.175 INFO:teuthology.orchestra.run.trial030.stdout: osd recover clone overlap = true 2026-02-25T08:40:45.175 INFO:teuthology.orchestra.run.trial030.stdout: osd recovery max chunk = 1048576 2026-02-25T08:40:45.175 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:40:45.175 INFO:teuthology.orchestra.run.trial030.stdout: osd debug shutdown = true 2026-02-25T08:40:45.175 INFO:teuthology.orchestra.run.trial030.stdout: osd debug op order = true 2026-02-25T08:40:45.175 INFO:teuthology.orchestra.run.trial030.stdout: osd debug verify stray on activate = true 2026-02-25T08:40:45.176 INFO:teuthology.orchestra.run.trial030.stdout: osd debug trim objects = true 2026-02-25T08:40:45.176 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:40:45.176 INFO:teuthology.orchestra.run.trial030.stdout: osd open classes on start = true 2026-02-25T08:40:45.176 INFO:teuthology.orchestra.run.trial030.stdout: osd debug pg log writeout = true 2026-02-25T08:40:45.176 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:40:45.176 INFO:teuthology.orchestra.run.trial030.stdout: osd deep scrub update digest min age = 30 2026-02-25T08:40:45.176 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:40:45.176 INFO:teuthology.orchestra.run.trial030.stdout: osd map max advance = 10 2026-02-25T08:40:45.176 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:40:45.176 INFO:teuthology.orchestra.run.trial030.stdout: journal zero on create = true 2026-02-25T08:40:45.176 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:40:45.176 INFO:teuthology.orchestra.run.trial030.stdout: filestore ondisk finisher threads = 3 2026-02-25T08:40:45.176 INFO:teuthology.orchestra.run.trial030.stdout: filestore apply finisher threads = 3 2026-02-25T08:40:45.177 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:40:45.177 INFO:teuthology.orchestra.run.trial030.stdout: bdev debug aio = true 2026-02-25T08:40:45.177 INFO:teuthology.orchestra.run.trial030.stdout: osd debug misdirected ops = true 2026-02-25T08:40:45.177 INFO:teuthology.orchestra.run.trial030.stdout: debug monc = 20 2026-02-25T08:40:45.177 INFO:teuthology.orchestra.run.trial030.stdout: crimson cpu num = 4 2026-02-25T08:40:45.177 INFO:teuthology.orchestra.run.trial030.stdout: crimson osd obc lru size = 10 2026-02-25T08:40:45.177 INFO:teuthology.orchestra.run.trial030.stdout: debug ms = 20 2026-02-25T08:40:45.177 INFO:teuthology.orchestra.run.trial030.stdout: debug osd = 20 2026-02-25T08:40:45.177 INFO:teuthology.orchestra.run.trial030.stdout: debug seastore = 20 2026-02-25T08:40:45.177 INFO:teuthology.orchestra.run.trial030.stdout: debug seastore backref = 20 2026-02-25T08:40:45.177 INFO:teuthology.orchestra.run.trial030.stdout: debug seastore cache = 20 2026-02-25T08:40:45.177 INFO:teuthology.orchestra.run.trial030.stdout: debug seastore cleaner = 20 2026-02-25T08:40:45.177 INFO:teuthology.orchestra.run.trial030.stdout: debug seastore device = 20 2026-02-25T08:40:45.177 INFO:teuthology.orchestra.run.trial030.stdout: debug seastore epm = 20 2026-02-25T08:40:45.178 INFO:teuthology.orchestra.run.trial030.stdout: debug seastore fixedkv tree = 20 2026-02-25T08:40:45.178 INFO:teuthology.orchestra.run.trial030.stdout: debug seastore journal = 20 2026-02-25T08:40:45.178 INFO:teuthology.orchestra.run.trial030.stdout: debug seastore lba = 20 2026-02-25T08:40:45.178 INFO:teuthology.orchestra.run.trial030.stdout: debug seastore odata = 20 2026-02-25T08:40:45.178 INFO:teuthology.orchestra.run.trial030.stdout: debug seastore omap = 20 2026-02-25T08:40:45.178 INFO:teuthology.orchestra.run.trial030.stdout: debug seastore onode = 20 2026-02-25T08:40:45.178 INFO:teuthology.orchestra.run.trial030.stdout: debug seastore t = 20 2026-02-25T08:40:45.178 INFO:teuthology.orchestra.run.trial030.stdout: debug seastore tm = 20 2026-02-25T08:40:45.178 INFO:teuthology.orchestra.run.trial030.stdout: osd blocked scrub grace period = 3600 2026-02-25T08:40:45.178 INFO:teuthology.orchestra.run.trial030.stdout: osd debug reject backfill probability = 0.3 2026-02-25T08:40:45.178 INFO:teuthology.orchestra.run.trial030.stdout: osd delete sleep = 1 2026-02-25T08:40:45.178 INFO:teuthology.orchestra.run.trial030.stdout: osd max backfills = 3 2026-02-25T08:40:45.178 INFO:teuthology.orchestra.run.trial030.stdout: osd max markdown count = 1000 2026-02-25T08:40:45.179 INFO:teuthology.orchestra.run.trial030.stdout: osd objectstore = seastore 2026-02-25T08:40:45.179 INFO:teuthology.orchestra.run.trial030.stdout: osd scrub min interval = 60 2026-02-25T08:40:45.179 INFO:teuthology.orchestra.run.trial030.stdout: osd shutdown pgref assert = True 2026-02-25T08:40:45.179 INFO:teuthology.orchestra.run.trial030.stdout: osd snap trim sleep = 2 2026-02-25T08:40:45.179 INFO:teuthology.orchestra.run.trial030.stdout: osd target transaction size = 5 2026-02-25T08:40:45.179 INFO:teuthology.orchestra.run.trial030.stdout: seastore cachepin size pershard = 64M 2026-02-25T08:40:45.179 INFO:teuthology.orchestra.run.trial030.stdout: seastore main device type = RANDOM_BLOCK_SSD 2026-02-25T08:40:45.179 INFO:teuthology.orchestra.run.trial030.stdout: seastore max concurrent transactions = 8 2026-02-25T08:40:45.179 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:40:45.179 INFO:teuthology.orchestra.run.trial030.stdout:[mgr] 2026-02-25T08:40:45.179 INFO:teuthology.orchestra.run.trial030.stdout: debug ms = 1 2026-02-25T08:40:45.179 INFO:teuthology.orchestra.run.trial030.stdout: debug mgr = 20 2026-02-25T08:40:45.179 INFO:teuthology.orchestra.run.trial030.stdout: debug mon = 20 2026-02-25T08:40:45.180 INFO:teuthology.orchestra.run.trial030.stdout: debug auth = 20 2026-02-25T08:40:45.180 INFO:teuthology.orchestra.run.trial030.stdout: mon reweight min pgs per osd = 4 2026-02-25T08:40:45.180 INFO:teuthology.orchestra.run.trial030.stdout: mon reweight min bytes per osd = 10 2026-02-25T08:40:45.180 INFO:teuthology.orchestra.run.trial030.stdout: mgr/telemetry/nag = false 2026-02-25T08:40:45.180 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:40:45.180 INFO:teuthology.orchestra.run.trial030.stdout:[mon] 2026-02-25T08:40:45.180 INFO:teuthology.orchestra.run.trial030.stdout: debug ms = 1 2026-02-25T08:40:45.180 INFO:teuthology.orchestra.run.trial030.stdout: debug mon = 20 2026-02-25T08:40:45.180 INFO:teuthology.orchestra.run.trial030.stdout: debug paxos = 20 2026-02-25T08:40:45.180 INFO:teuthology.orchestra.run.trial030.stdout: debug auth = 20 2026-02-25T08:40:45.180 INFO:teuthology.orchestra.run.trial030.stdout: mon data avail warn = 5 2026-02-25T08:40:45.180 INFO:teuthology.orchestra.run.trial030.stdout: mon mgr mkfs grace = 240 2026-02-25T08:40:45.180 INFO:teuthology.orchestra.run.trial030.stdout: mon reweight min pgs per osd = 4 2026-02-25T08:40:45.181 INFO:teuthology.orchestra.run.trial030.stdout: mon osd reporter subtree level = osd 2026-02-25T08:40:45.181 INFO:teuthology.orchestra.run.trial030.stdout: mon osd prime pg temp = true 2026-02-25T08:40:45.181 INFO:teuthology.orchestra.run.trial030.stdout: mon reweight min bytes per osd = 10 2026-02-25T08:40:45.181 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:40:45.181 INFO:teuthology.orchestra.run.trial030.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-25T08:40:45.181 INFO:teuthology.orchestra.run.trial030.stdout: auth mon ticket ttl = 660 # 11m 2026-02-25T08:40:45.181 INFO:teuthology.orchestra.run.trial030.stdout: auth service ticket ttl = 240 # 4m 2026-02-25T08:40:45.181 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:40:45.181 INFO:teuthology.orchestra.run.trial030.stdout: # don't complain about insecure global_id in the test suite 2026-02-25T08:40:45.181 INFO:teuthology.orchestra.run.trial030.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-25T08:40:45.181 INFO:teuthology.orchestra.run.trial030.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-25T08:40:45.181 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:40:45.181 INFO:teuthology.orchestra.run.trial030.stdout: # 1m isn't quite enough 2026-02-25T08:40:45.182 INFO:teuthology.orchestra.run.trial030.stdout: mon_down_mkfs_grace = 2m 2026-02-25T08:40:45.182 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:40:45.182 INFO:teuthology.orchestra.run.trial030.stdout: mon_warn_on_filestore_osds = false 2026-02-25T08:40:45.182 INFO:teuthology.orchestra.run.trial030.stdout: mon min osdmap epochs = 50 2026-02-25T08:40:45.182 INFO:teuthology.orchestra.run.trial030.stdout: mon osdmap full prune interval = 2 2026-02-25T08:40:45.182 INFO:teuthology.orchestra.run.trial030.stdout: mon osdmap full prune min = 15 2026-02-25T08:40:45.182 INFO:teuthology.orchestra.run.trial030.stdout: mon osdmap full prune txsize = 2 2026-02-25T08:40:45.182 INFO:teuthology.orchestra.run.trial030.stdout: osd pool default crimson = True 2026-02-25T08:40:45.182 INFO:teuthology.orchestra.run.trial030.stdout: paxos service trim min = 10 2026-02-25T08:40:45.182 INFO:teuthology.orchestra.run.trial030.stdout: 2026-02-25T08:40:45.182 INFO:teuthology.orchestra.run.trial030.stdout:[client] 2026-02-25T08:40:45.182 INFO:teuthology.orchestra.run.trial030.stdout: rgw cache enabled = true 2026-02-25T08:40:45.182 INFO:teuthology.orchestra.run.trial030.stdout: rgw enable ops log = true 2026-02-25T08:40:45.183 INFO:teuthology.orchestra.run.trial030.stdout: rgw enable usage log = true 2026-02-25T08:40:45.183 INFO:teuthology.orchestra.run.trial030.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-25T08:40:45.183 INFO:teuthology.orchestra.run.trial030.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-25T08:40:45.183 INFO:teuthology.orchestra.run.trial030.stdout:[mon.a] 2026-02-25T08:40:45.183 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-25T08:40:45.183 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-25T08:40:45.260 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-25T08:40:45.260 DEBUG:teuthology.orchestra.run.trial023:> set -ex 2026-02-25T08:40:45.260 DEBUG:teuthology.orchestra.run.trial023:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-25T08:40:45.277 DEBUG:teuthology.orchestra.run.trial023:> set -ex 2026-02-25T08:40:45.277 DEBUG:teuthology.orchestra.run.trial023:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-25T08:40:45.333 INFO:tasks.ceph:Sending monmap to node ubuntu@trial023.front.sepia.ceph.com 2026-02-25T08:40:45.334 DEBUG:teuthology.orchestra.run.trial023:> set -ex 2026-02-25T08:40:45.334 DEBUG:teuthology.orchestra.run.trial023:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-25T08:40:45.334 DEBUG:teuthology.orchestra.run.trial023:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-25T08:40:45.405 DEBUG:teuthology.orchestra.run.trial023:> set -ex 2026-02-25T08:40:45.406 DEBUG:teuthology.orchestra.run.trial023:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:40:45.461 INFO:tasks.ceph:Sending monmap to node ubuntu@trial030.front.sepia.ceph.com 2026-02-25T08:40:45.461 DEBUG:teuthology.orchestra.run.trial030:> set -ex 2026-02-25T08:40:45.461 DEBUG:teuthology.orchestra.run.trial030:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-25T08:40:45.461 DEBUG:teuthology.orchestra.run.trial030:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-25T08:40:45.496 DEBUG:teuthology.orchestra.run.trial030:> set -ex 2026-02-25T08:40:45.496 DEBUG:teuthology.orchestra.run.trial030:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:40:45.552 INFO:tasks.ceph:Setting up mon nodes... 2026-02-25T08:40:45.553 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-25T08:40:45.553 DEBUG:teuthology.orchestra.run.trial030:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-25T08:40:45.643 INFO:teuthology.orchestra.run.trial030.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-25T08:40:45.646 INFO:tasks.ceph:Setting up mds nodes... 2026-02-25T08:40:45.646 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-25T08:40:45.647 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-25T08:40:45.689 INFO:teuthology.orchestra.run.trial023.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-25T08:40:45.705 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2026-02-25T08:40:45.751 INFO:teuthology.orchestra.run.trial030.stdout:creating /etc/ceph/ceph.client.1.keyring 2026-02-25T08:40:45.764 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-25T08:40:45.765 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial023.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@trial030.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2026-02-25T08:40:45.765 DEBUG:teuthology.orchestra.run.trial023:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-25T08:40:45.790 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2026-02-25T08:40:45.790 INFO:tasks.ceph:role: osd.0 2026-02-25T08:40:45.790 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial023.front.sepia.ceph.com 2026-02-25T08:40:45.790 DEBUG:teuthology.orchestra.run.trial023:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-25T08:40:45.855 INFO:teuthology.orchestra.run.trial023.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-25T08:40:45.855 INFO:teuthology.orchestra.run.trial023.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-25T08:40:45.855 INFO:teuthology.orchestra.run.trial023.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-25T08:40:45.855 INFO:teuthology.orchestra.run.trial023.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-25T08:40:45.855 INFO:teuthology.orchestra.run.trial023.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-25T08:40:45.856 INFO:teuthology.orchestra.run.trial023.stdout: = sunit=0 swidth=0 blks 2026-02-25T08:40:45.856 INFO:teuthology.orchestra.run.trial023.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-25T08:40:45.856 INFO:teuthology.orchestra.run.trial023.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-25T08:40:45.856 INFO:teuthology.orchestra.run.trial023.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-25T08:40:45.856 INFO:teuthology.orchestra.run.trial023.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-25T08:40:46.433 INFO:teuthology.orchestra.run.trial023.stdout:Discarding blocks...Done. 2026-02-25T08:40:46.435 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial023.front.sepia.ceph.com -o noatime 2026-02-25T08:40:46.436 DEBUG:teuthology.orchestra.run.trial023:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-25T08:40:46.503 DEBUG:teuthology.orchestra.run.trial023:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-25T08:40:46.570 DEBUG:teuthology.orchestra.run.trial023:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-25T08:40:46.634 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2026-02-25T08:40:46.634 INFO:tasks.ceph:role: osd.1 2026-02-25T08:40:46.634 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial023.front.sepia.ceph.com 2026-02-25T08:40:46.634 DEBUG:teuthology.orchestra.run.trial023:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-25T08:40:46.699 INFO:teuthology.orchestra.run.trial023.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-25T08:40:46.699 INFO:teuthology.orchestra.run.trial023.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-25T08:40:46.699 INFO:teuthology.orchestra.run.trial023.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-25T08:40:46.699 INFO:teuthology.orchestra.run.trial023.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-25T08:40:46.699 INFO:teuthology.orchestra.run.trial023.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-25T08:40:46.699 INFO:teuthology.orchestra.run.trial023.stdout: = sunit=0 swidth=0 blks 2026-02-25T08:40:46.699 INFO:teuthology.orchestra.run.trial023.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-25T08:40:46.699 INFO:teuthology.orchestra.run.trial023.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-25T08:40:46.700 INFO:teuthology.orchestra.run.trial023.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-25T08:40:46.700 INFO:teuthology.orchestra.run.trial023.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-25T08:40:47.256 INFO:teuthology.orchestra.run.trial023.stdout:Discarding blocks...Done. 2026-02-25T08:40:47.258 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial023.front.sepia.ceph.com -o noatime 2026-02-25T08:40:47.259 DEBUG:teuthology.orchestra.run.trial023:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-25T08:40:47.326 DEBUG:teuthology.orchestra.run.trial023:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-25T08:40:47.393 DEBUG:teuthology.orchestra.run.trial023:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:40:47.545 INFO:teuthology.orchestra.run.trial023.stderr:INFO  osd - OSD::main: parsing early config 2026-02-25T08:40:47.599 INFO:teuthology.orchestra.run.trial023.stderr:INFO 2026-02-25 08:40:47,598 seastar - Reactor backend: linux-aio 2026-02-25T08:40:47.610 INFO:teuthology.orchestra.run.trial023.stderr:INFO 2026-02-25 08:40:47,609 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:47.610 INFO:teuthology.orchestra.run.trial023.stderr:INFO 2026-02-25 08:40:47,609 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:47.612 INFO:teuthology.orchestra.run.trial023.stderr:==40061==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:40:47.668 INFO:teuthology.orchestra.run.trial023.stderr:INFO 2026-02-25 08:40:47,667 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-25T08:40:47.668 INFO:teuthology.orchestra.run.trial023.stderr:INFO 2026-02-25 08:40:47,667 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-25T08:40:47.669 INFO:teuthology.orchestra.run.trial023.stderr:INFO 2026-02-25 08:40:47,667 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-25T08:40:47.669 INFO:teuthology.orchestra.run.trial023.stderr:INFO 2026-02-25 08:40:47,667 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-25T08:40:47.669 INFO:teuthology.orchestra.run.trial023.stderr:DEBUG 2026-02-25 08:40:47,667 [shard 0:main] seastar - reactor::drain 2026-02-25T08:40:49.748 INFO:teuthology.orchestra.run.trial023.stderr:----------------------------------------------------- 2026-02-25T08:40:49.749 INFO:teuthology.orchestra.run.trial023.stderr:Suppressions used: 2026-02-25T08:40:49.749 INFO:teuthology.orchestra.run.trial023.stderr: count bytes template 2026-02-25T08:40:49.749 INFO:teuthology.orchestra.run.trial023.stderr: 1 8 MallocExtension::Register 2026-02-25T08:40:49.749 INFO:teuthology.orchestra.run.trial023.stderr:----------------------------------------------------- 2026-02-25T08:40:49.749 INFO:teuthology.orchestra.run.trial023.stderr: 2026-02-25T08:40:49.768 INFO:teuthology.orchestra.run.trial023.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-25T08:40:49.768 INFO:teuthology.orchestra.run.trial023.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-25T08:40:49.816 INFO:teuthology.orchestra.run.trial023.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-25T08:40:49.816 INFO:teuthology.orchestra.run.trial023.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-25T08:40:49.822 INFO:teuthology.orchestra.run.trial023.stderr:INFO 2026-02-25 08:40:49,822 seastar - Reactor backend: linux-aio 2026-02-25T08:40:49.837 INFO:teuthology.orchestra.run.trial023.stderr:INFO 2026-02-25 08:40:49,837 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:49.838 INFO:teuthology.orchestra.run.trial023.stderr:INFO 2026-02-25 08:40:49,837 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:49.841 INFO:teuthology.orchestra.run.trial023.stderr:==40059==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:40:49.841 INFO:teuthology.orchestra.run.trial023.stderr:INFO 2026-02-25 08:40:49,840 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-25T08:40:49.900 INFO:teuthology.orchestra.run.trial023.stderr:DEBUG 2026-02-25 08:40:49,899 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-25T08:40:49.980 INFO:teuthology.orchestra.run.trial023.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xf0e7f /lib64/libasan.so.8+0xe908 /lib64/libasan.so.8+0xec776 0xcc26c1 0x318584f 0x3185bb5 0x3185caa 0x9f169a9 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:49.980 INFO:teuthology.orchestra.run.trial023.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x119106 /lib64/libasan.so.8+0xebbf /lib64/libasan.so.8+0xef905 0x3185724 0x31858fe 0x3185bb5 0x3185b7a 0x3185caa 0x6f4328c 0x7ba1b15 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:49.980 INFO:teuthology.orchestra.run.trial023.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x755a777 0x725c0b0 0x711b842 0x7ba1535 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:49.980 INFO:teuthology.orchestra.run.trial023.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87e1ff44 0xffffffff87e203b0 0xffffffff87c9525e 0xffffffff87fe7269 0xffffffff87fca7d5 0xffffffff87fd4bab 0xffffffff87fd500b 0xffffffff87c910d2 0xffffffff88950451 0xffffffff88a00ba2 2026-02-25T08:40:50.131 INFO:teuthology.orchestra.run.trial023.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x10069f /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x9e21748 0x9e21ba9 0x9e21e1c 0x9e220d2 0x9e22b81 0x9e2307f 0x9e236af 0x9e23819 0x9f03b89 0x9f1698c 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:50.131 INFO:teuthology.orchestra.run.trial023.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x104e70 /lib64/libasan.so.8+0xea9a /lib64/libasan.so.8+0xef905 0x3185724 0x31858fe 0x3185bb5 0x3185caa 0x6f4328c 0x7ba192e 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:50.131 INFO:teuthology.orchestra.run.trial023.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xe99a /lib64/libasan.so.8+0xec776 0xcc26c1 0x9ef6ecd 0x9f1372b 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:50.131 INFO:teuthology.orchestra.run.trial023.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff87c91052 0xffffffff88950451 0xffffffff88a00ba2 2026-02-25T08:40:52.258 INFO:teuthology.orchestra.run.trial023.stderr:----------------------------------------------------- 2026-02-25T08:40:52.258 INFO:teuthology.orchestra.run.trial023.stderr:Suppressions used: 2026-02-25T08:40:52.258 INFO:teuthology.orchestra.run.trial023.stderr: count bytes template 2026-02-25T08:40:52.258 INFO:teuthology.orchestra.run.trial023.stderr: 1 8 MallocExtension::Register 2026-02-25T08:40:52.258 INFO:teuthology.orchestra.run.trial023.stderr:----------------------------------------------------- 2026-02-25T08:40:52.258 INFO:teuthology.orchestra.run.trial023.stderr: 2026-02-25T08:40:52.289 DEBUG:teuthology.orchestra.run.trial023:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-25T08:40:52.315 DEBUG:teuthology.orchestra.run.trial023:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:40:52.466 INFO:teuthology.orchestra.run.trial023.stderr:INFO  osd - OSD::main: parsing early config 2026-02-25T08:40:52.519 INFO:teuthology.orchestra.run.trial023.stderr:INFO 2026-02-25 08:40:52,518 seastar - Reactor backend: linux-aio 2026-02-25T08:40:52.536 INFO:teuthology.orchestra.run.trial023.stderr:INFO 2026-02-25 08:40:52,536 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:52.536 INFO:teuthology.orchestra.run.trial023.stderr:INFO 2026-02-25 08:40:52,536 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:52.539 INFO:teuthology.orchestra.run.trial023.stderr:==40128==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:40:52.595 INFO:teuthology.orchestra.run.trial023.stderr:INFO 2026-02-25 08:40:52,594 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-25T08:40:52.595 INFO:teuthology.orchestra.run.trial023.stderr:INFO 2026-02-25 08:40:52,594 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-25T08:40:52.595 INFO:teuthology.orchestra.run.trial023.stderr:INFO 2026-02-25 08:40:52,594 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-25T08:40:52.595 INFO:teuthology.orchestra.run.trial023.stderr:INFO 2026-02-25 08:40:52,594 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-25T08:40:52.595 INFO:teuthology.orchestra.run.trial023.stderr:DEBUG 2026-02-25 08:40:52,594 [shard 0:main] seastar - reactor::drain 2026-02-25T08:40:54.657 INFO:teuthology.orchestra.run.trial023.stderr:----------------------------------------------------- 2026-02-25T08:40:54.657 INFO:teuthology.orchestra.run.trial023.stderr:Suppressions used: 2026-02-25T08:40:54.657 INFO:teuthology.orchestra.run.trial023.stderr: count bytes template 2026-02-25T08:40:54.657 INFO:teuthology.orchestra.run.trial023.stderr: 1 8 MallocExtension::Register 2026-02-25T08:40:54.657 INFO:teuthology.orchestra.run.trial023.stderr:----------------------------------------------------- 2026-02-25T08:40:54.657 INFO:teuthology.orchestra.run.trial023.stderr: 2026-02-25T08:40:54.677 INFO:teuthology.orchestra.run.trial023.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-25T08:40:54.677 INFO:teuthology.orchestra.run.trial023.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-25T08:40:54.725 INFO:teuthology.orchestra.run.trial023.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-25T08:40:54.725 INFO:teuthology.orchestra.run.trial023.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-25T08:40:54.731 INFO:teuthology.orchestra.run.trial023.stderr:INFO 2026-02-25 08:40:54,731 seastar - Reactor backend: linux-aio 2026-02-25T08:40:54.742 INFO:teuthology.orchestra.run.trial023.stderr:INFO 2026-02-25 08:40:54,741 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:54.742 INFO:teuthology.orchestra.run.trial023.stderr:INFO 2026-02-25 08:40:54,741 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:54.746 INFO:teuthology.orchestra.run.trial023.stderr:==40126==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:40:54.746 INFO:teuthology.orchestra.run.trial023.stderr:INFO 2026-02-25 08:40:54,745 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-25T08:40:54.805 INFO:teuthology.orchestra.run.trial023.stderr:DEBUG 2026-02-25 08:40:54,804 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-25T08:40:54.885 INFO:teuthology.orchestra.run.trial023.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1583b 0x10a2823 0xf47938 0x9e218ac 0x9e21ba9 0x9e21e1c 0x9e220d2 0x9e22694 0x9e2ab3b 0x9e2b21b 0x9f13d68 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:54.885 INFO:teuthology.orchestra.run.trial023.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xe40f5d 0x9d73524 0x9d73668 0x9d7374d 0x9d739a9 0x9d73a9c 0x9d73b78 0x9d73df4 0x9d7439b 0x9d7479f 0x9d74a5c 0x9d758a4 0x9eec474 0x9eec56d 0x7ba1825 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:54.886 INFO:teuthology.orchestra.run.trial023.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff88963497 0xffffffff87fd19d8 0xffffffff87fd4bab 0xffffffff87fd500b 0xffffffff87c910d2 0xffffffff88950451 0xffffffff88a00ba2 2026-02-25T08:40:54.886 INFO:teuthology.orchestra.run.trial023.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x318121e 0x9e21b47 0x9e21e1c 0x9e23b43 0x9f02d0e 0x9f03487 0x9f03972 0x9f03d05 0x9f1698c 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:55.039 INFO:teuthology.orchestra.run.trial023.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cc0c /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x9e21748 0x9e21ba9 0x9e21e1c 0x9e220d2 0x9e22694 0x9e2ab3b 0x9e2b21b 0x9f13d68 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:55.039 INFO:teuthology.orchestra.run.trial023.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x35fc02f 0x35fc215 0x3618092 0x9e33da7 0x9e33ed4 0x9ef3c57 0x9ef3dec 0x9ef417e 0x9ef5765 0x9ef6a80 0x9ef6ea0 0x9f1372b 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:55.039 INFO:teuthology.orchestra.run.trial023.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297e4 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x157c7 0xca7484 0xcb94be 0x9e1cc7e 0x9f169d8 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:57.198 INFO:teuthology.orchestra.run.trial023.stderr:----------------------------------------------------- 2026-02-25T08:40:57.198 INFO:teuthology.orchestra.run.trial023.stderr:Suppressions used: 2026-02-25T08:40:57.198 INFO:teuthology.orchestra.run.trial023.stderr: count bytes template 2026-02-25T08:40:57.198 INFO:teuthology.orchestra.run.trial023.stderr: 1 8 MallocExtension::Register 2026-02-25T08:40:57.198 INFO:teuthology.orchestra.run.trial023.stderr:----------------------------------------------------- 2026-02-25T08:40:57.198 INFO:teuthology.orchestra.run.trial023.stderr: 2026-02-25T08:40:57.229 DEBUG:teuthology.orchestra.run.trial023:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-25T08:40:57.254 DEBUG:teuthology.orchestra.run.trial030:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-25T08:40:57.280 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2026-02-25T08:40:57.281 INFO:tasks.ceph:role: osd.2 2026-02-25T08:40:57.281 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial030.front.sepia.ceph.com 2026-02-25T08:40:57.281 DEBUG:teuthology.orchestra.run.trial030:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-25T08:40:57.345 INFO:teuthology.orchestra.run.trial030.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-25T08:40:57.346 INFO:teuthology.orchestra.run.trial030.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-25T08:40:57.346 INFO:teuthology.orchestra.run.trial030.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-25T08:40:57.346 INFO:teuthology.orchestra.run.trial030.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-25T08:40:57.346 INFO:teuthology.orchestra.run.trial030.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-25T08:40:57.346 INFO:teuthology.orchestra.run.trial030.stdout: = sunit=0 swidth=0 blks 2026-02-25T08:40:57.346 INFO:teuthology.orchestra.run.trial030.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-25T08:40:57.346 INFO:teuthology.orchestra.run.trial030.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-25T08:40:57.346 INFO:teuthology.orchestra.run.trial030.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-25T08:40:57.346 INFO:teuthology.orchestra.run.trial030.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-25T08:40:57.924 INFO:teuthology.orchestra.run.trial030.stdout:Discarding blocks...Done. 2026-02-25T08:40:57.926 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial030.front.sepia.ceph.com -o noatime 2026-02-25T08:40:57.926 DEBUG:teuthology.orchestra.run.trial030:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2026-02-25T08:40:57.995 DEBUG:teuthology.orchestra.run.trial030:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-25T08:40:58.061 DEBUG:teuthology.orchestra.run.trial030:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2026-02-25T08:40:58.124 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2026-02-25T08:40:58.124 INFO:tasks.ceph:role: osd.3 2026-02-25T08:40:58.124 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial030.front.sepia.ceph.com 2026-02-25T08:40:58.124 DEBUG:teuthology.orchestra.run.trial030:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-25T08:40:58.189 INFO:teuthology.orchestra.run.trial030.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-25T08:40:58.190 INFO:teuthology.orchestra.run.trial030.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-25T08:40:58.190 INFO:teuthology.orchestra.run.trial030.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-25T08:40:58.190 INFO:teuthology.orchestra.run.trial030.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-25T08:40:58.190 INFO:teuthology.orchestra.run.trial030.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-25T08:40:58.190 INFO:teuthology.orchestra.run.trial030.stdout: = sunit=0 swidth=0 blks 2026-02-25T08:40:58.190 INFO:teuthology.orchestra.run.trial030.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-25T08:40:58.190 INFO:teuthology.orchestra.run.trial030.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-25T08:40:58.190 INFO:teuthology.orchestra.run.trial030.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-25T08:40:58.190 INFO:teuthology.orchestra.run.trial030.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-25T08:40:58.750 INFO:teuthology.orchestra.run.trial030.stdout:Discarding blocks...Done. 2026-02-25T08:40:58.751 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial030.front.sepia.ceph.com -o noatime 2026-02-25T08:40:58.751 DEBUG:teuthology.orchestra.run.trial030:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-3 2026-02-25T08:40:58.820 DEBUG:teuthology.orchestra.run.trial030:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2026-02-25T08:40:58.886 DEBUG:teuthology.orchestra.run.trial030:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:40:59.039 INFO:teuthology.orchestra.run.trial030.stderr:INFO  osd - OSD::main: parsing early config 2026-02-25T08:40:59.102 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-25 08:40:59,102 seastar - Reactor backend: linux-aio 2026-02-25T08:40:59.113 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-25 08:40:59,113 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:59.113 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-25 08:40:59,113 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:59.115 INFO:teuthology.orchestra.run.trial030.stderr:==39399==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:40:59.171 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-25 08:40:59,171 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-25T08:40:59.172 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-25 08:40:59,171 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-25T08:40:59.172 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-25 08:40:59,171 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-25T08:40:59.172 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-25 08:40:59,171 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-25T08:40:59.172 INFO:teuthology.orchestra.run.trial030.stderr:DEBUG 2026-02-25 08:40:59,171 [shard 0:main] seastar - reactor::drain 2026-02-25T08:41:01.227 INFO:teuthology.orchestra.run.trial030.stderr:----------------------------------------------------- 2026-02-25T08:41:01.227 INFO:teuthology.orchestra.run.trial030.stderr:Suppressions used: 2026-02-25T08:41:01.228 INFO:teuthology.orchestra.run.trial030.stderr: count bytes template 2026-02-25T08:41:01.228 INFO:teuthology.orchestra.run.trial030.stderr: 1 8 MallocExtension::Register 2026-02-25T08:41:01.228 INFO:teuthology.orchestra.run.trial030.stderr:----------------------------------------------------- 2026-02-25T08:41:01.228 INFO:teuthology.orchestra.run.trial030.stderr: 2026-02-25T08:41:01.246 INFO:teuthology.orchestra.run.trial030.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-25T08:41:01.246 INFO:teuthology.orchestra.run.trial030.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-25T08:41:01.294 INFO:teuthology.orchestra.run.trial030.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-25T08:41:01.294 INFO:teuthology.orchestra.run.trial030.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-25T08:41:01.301 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-25 08:41:01,301 seastar - Reactor backend: linux-aio 2026-02-25T08:41:01.309 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-25 08:41:01,309 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:41:01.310 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-25 08:41:01,309 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:41:01.313 INFO:teuthology.orchestra.run.trial030.stderr:==39397==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:41:01.313 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-25 08:41:01,313 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-25T08:41:01.373 INFO:teuthology.orchestra.run.trial030.stderr:DEBUG 2026-02-25 08:41:01,373 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-25T08:41:01.453 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x15197 0x11380c8 0xf4792d 0x9e218ac 0x9e21ba9 0x9e21e1c 0x9e220d2 0x9e22b81 0x9e2307f 0x9e236af 0x9e23819 0x9f03b89 0x9f1698c 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:41:01.454 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xe40f5d 0x9f1a2ce 0x78b69db 0x7ba17e8 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:41:01.454 INFO:teuthology.orchestra.run.trial030.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff9d81ff3b 0xffffffff9d8203b0 0xffffffff9d69525e 0xffffffff9d9e7269 0xffffffff9d9ca7d5 0xffffffff9d9d4bab 0xffffffff9d9d500b 0xffffffff9d6910d2 0xffffffff9e350451 0xffffffff9e400ba2 2026-02-25T08:41:01.454 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cc0c /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x157c7 0x10a2823 0xf47938 0x9e218ac 0x9e21ba9 0x9e21e1c 0x9e220d2 0x9e22b81 0x9e2307f 0x9e236af 0x9e23819 0x9f03b89 0x9f1698c 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:41:01.605 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xee75f 0x9cf61eb 0x9d73986 0x9d73a9c 0x9d73b78 0x9d74234 0x9d73ead 0x9d7439b 0x9d7479f 0x9d74a5c 0x9d758a4 0x9eec474 0x9eec56d 0x7ba1220 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:41:01.605 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0xe29e5c 0x1138071 0xf4792d 0x9e26106 0x9e26a45 0x9f12a15 0x9f13935 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:41:01.605 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65c6 /lib64/libasan.so.8+0x15197 0x9e6cfb9 0x9ed8b72 0x9f1a352 0x78b69db 0x7ba17e8 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:41:03.772 INFO:teuthology.orchestra.run.trial030.stderr:----------------------------------------------------- 2026-02-25T08:41:03.773 INFO:teuthology.orchestra.run.trial030.stderr:Suppressions used: 2026-02-25T08:41:03.773 INFO:teuthology.orchestra.run.trial030.stderr: count bytes template 2026-02-25T08:41:03.773 INFO:teuthology.orchestra.run.trial030.stderr: 1 8 MallocExtension::Register 2026-02-25T08:41:03.773 INFO:teuthology.orchestra.run.trial030.stderr:----------------------------------------------------- 2026-02-25T08:41:03.773 INFO:teuthology.orchestra.run.trial030.stderr: 2026-02-25T08:41:03.805 DEBUG:teuthology.orchestra.run.trial030:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-25T08:41:03.830 DEBUG:teuthology.orchestra.run.trial030:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:41:03.982 INFO:teuthology.orchestra.run.trial030.stderr:INFO  osd - OSD::main: parsing early config 2026-02-25T08:41:04.038 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-25 08:41:04,038 seastar - Reactor backend: linux-aio 2026-02-25T08:41:04.048 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-25 08:41:04,048 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:41:04.048 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-25 08:41:04,048 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:41:04.050 INFO:teuthology.orchestra.run.trial030.stderr:==39466==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:41:04.106 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-25 08:41:04,106 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-25T08:41:04.106 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-25 08:41:04,106 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-25T08:41:04.106 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-25 08:41:04,106 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-25T08:41:04.106 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-25 08:41:04,106 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-25T08:41:04.106 INFO:teuthology.orchestra.run.trial030.stderr:DEBUG 2026-02-25 08:41:04,106 [shard 0:main] seastar - reactor::drain 2026-02-25T08:41:06.174 INFO:teuthology.orchestra.run.trial030.stderr:----------------------------------------------------- 2026-02-25T08:41:06.174 INFO:teuthology.orchestra.run.trial030.stderr:Suppressions used: 2026-02-25T08:41:06.174 INFO:teuthology.orchestra.run.trial030.stderr: count bytes template 2026-02-25T08:41:06.174 INFO:teuthology.orchestra.run.trial030.stderr: 1 8 MallocExtension::Register 2026-02-25T08:41:06.175 INFO:teuthology.orchestra.run.trial030.stderr:----------------------------------------------------- 2026-02-25T08:41:06.175 INFO:teuthology.orchestra.run.trial030.stderr: 2026-02-25T08:41:06.194 INFO:teuthology.orchestra.run.trial030.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-25T08:41:06.194 INFO:teuthology.orchestra.run.trial030.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-25T08:41:06.236 INFO:teuthology.orchestra.run.trial030.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-25T08:41:06.236 INFO:teuthology.orchestra.run.trial030.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-25T08:41:06.243 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-25 08:41:06,243 seastar - Reactor backend: linux-aio 2026-02-25T08:41:06.257 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-25 08:41:06,257 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:41:06.257 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-25 08:41:06,257 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:41:06.261 INFO:teuthology.orchestra.run.trial030.stderr:==39464==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:41:06.261 INFO:teuthology.orchestra.run.trial030.stderr:INFO 2026-02-25 08:41:06,261 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-25T08:41:06.320 INFO:teuthology.orchestra.run.trial030.stderr:DEBUG 2026-02-25 08:41:06,320 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-25T08:41:06.400 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc4a3 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x755a777 0x725c0b0 0x711b842 0x7ba0f3a 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:41:06.400 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x151f5 0x9e21748 0x9e21ba9 0x9e21e1c 0x9e220d2 0x9e22694 0x9e2ab3b 0x9e2b21b 0x9f13d68 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:41:06.400 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x10a2826 0xf47938 0x9e216df 0x9e21ba9 0x9e21e1c 0x9e220d2 0x9e22694 0x9e2ab3b 0x9e2b21b 0x9f13d68 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:41:06.550 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc57c /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xe40f5d 0xe412b4 0x7ba14ec 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:41:06.551 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297e4 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x11380c8 0xf4792d 0x9e216df 0x9e21ba9 0x9e21e1c 0x9e220d2 0x9e22b81 0x9e2307f 0x9e236af 0x9e23819 0x9f03b89 0x9f1698c 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:41:06.551 INFO:teuthology.orchestra.run.trial030.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1583b 0x10a2823 0xf47938 0x9e218ac 0x9e21ba9 0x9e21e1c 0x9e22e1c 0x9e2338a 0x9e236af 0x9e23819 0x9f03b89 0x9f1698c 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:41:08.715 INFO:teuthology.orchestra.run.trial030.stderr:----------------------------------------------------- 2026-02-25T08:41:08.715 INFO:teuthology.orchestra.run.trial030.stderr:Suppressions used: 2026-02-25T08:41:08.715 INFO:teuthology.orchestra.run.trial030.stderr: count bytes template 2026-02-25T08:41:08.715 INFO:teuthology.orchestra.run.trial030.stderr: 1 8 MallocExtension::Register 2026-02-25T08:41:08.715 INFO:teuthology.orchestra.run.trial030.stderr:----------------------------------------------------- 2026-02-25T08:41:08.715 INFO:teuthology.orchestra.run.trial030.stderr: 2026-02-25T08:41:08.746 DEBUG:teuthology.orchestra.run.trial030:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2026-02-25T08:41:08.770 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-25T08:41:08.770 DEBUG:teuthology.orchestra.run.trial023:> set -ex 2026-02-25T08:41:08.770 DEBUG:teuthology.orchestra.run.trial023:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-25T08:41:08.795 DEBUG:teuthology.orchestra.run.trial023:> set -ex 2026-02-25T08:41:08.795 DEBUG:teuthology.orchestra.run.trial023:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-25T08:41:08.857 DEBUG:teuthology.orchestra.run.trial030:> set -ex 2026-02-25T08:41:08.857 DEBUG:teuthology.orchestra.run.trial030:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-25T08:41:08.882 DEBUG:teuthology.orchestra.run.trial030:> set -ex 2026-02-25T08:41:08.882 DEBUG:teuthology.orchestra.run.trial030:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-25T08:41:08.945 DEBUG:teuthology.orchestra.run.trial030:> set -ex 2026-02-25T08:41:08.945 DEBUG:teuthology.orchestra.run.trial030:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2026-02-25T08:41:09.009 DEBUG:teuthology.orchestra.run.trial023:> set -ex 2026-02-25T08:41:09.009 DEBUG:teuthology.orchestra.run.trial023:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-25T08:41:09.026 DEBUG:teuthology.orchestra.run.trial030:> set -ex 2026-02-25T08:41:09.026 DEBUG:teuthology.orchestra.run.trial030:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2026-02-25T08:41:09.065 INFO:tasks.ceph:Adding keys to all mons... 2026-02-25T08:41:09.065 DEBUG:teuthology.orchestra.run.trial023:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-25T08:41:09.088 INFO:teuthology.orchestra.run.trial023.stdout:[osd.0] 2026-02-25T08:41:09.089 INFO:teuthology.orchestra.run.trial023.stdout: key = AQCRtZ5pCETUNRAAGnQXpw65vG7y7z4CYY4AMg== 2026-02-25T08:41:09.089 INFO:teuthology.orchestra.run.trial023.stdout:[osd.1] 2026-02-25T08:41:09.089 INFO:teuthology.orchestra.run.trial023.stdout: key = AQCWtZ5p46MvMBAAgI9Md8d0jjVPLlH1LLsoWg== 2026-02-25T08:41:09.089 INFO:teuthology.orchestra.run.trial023.stdout:[mgr.x] 2026-02-25T08:41:09.089 INFO:teuthology.orchestra.run.trial023.stdout: key = AQCNtZ5pKyZdJhAAcB2igkfN8DJ3yoLwCwF5tg== 2026-02-25T08:41:09.089 INFO:teuthology.orchestra.run.trial023.stdout:[osd.2] 2026-02-25T08:41:09.089 INFO:teuthology.orchestra.run.trial023.stdout: key = AQCdtZ5pEb50FhAA5rhaH2ZsYEpm25TCcsiv/Q== 2026-02-25T08:41:09.089 INFO:teuthology.orchestra.run.trial023.stdout:[osd.3] 2026-02-25T08:41:09.089 INFO:teuthology.orchestra.run.trial023.stdout: key = AQCitZ5pPAtUExAAOeJSPBxmfjjqdfuThBsSYg== 2026-02-25T08:41:09.090 INFO:teuthology.orchestra.run.trial023.stdout:[client.0] 2026-02-25T08:41:09.090 INFO:teuthology.orchestra.run.trial023.stdout: key = AQCNtZ5pCr4SKRAAc5kaQfEnHh7WrE5gYWjeeA== 2026-02-25T08:41:09.090 INFO:teuthology.orchestra.run.trial023.stdout:[client.1] 2026-02-25T08:41:09.090 INFO:teuthology.orchestra.run.trial023.stdout: key = AQCNtZ5pLPLNLBAALrJOt/Z28zzmTCwFRzakqg== 2026-02-25T08:41:09.091 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-25T08:41:09.179 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-25T08:41:09.228 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-25T08:41:09.276 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-25T08:41:09.324 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-25T08:41:09.372 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-25T08:41:09.422 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-25T08:41:09.472 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-25T08:41:09.473 DEBUG:teuthology.orchestra.run.trial023:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-25T08:41:09.497 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-25T08:41:09.584 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:09.582+0000 7fde54d00c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:09.584 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:09.582+0000 7fde54d00c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:09.596 DEBUG:teuthology.orchestra.run.trial023:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-25T08:41:09.621 DEBUG:teuthology.orchestra.run.trial023:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:41:09.677 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-25T08:41:09.678 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-25T08:41:09.678 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-25T08:41:09.719 INFO:tasks.ceph.mon.a:Started 2026-02-25T08:41:09.719 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-25T08:41:09.719 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-25T08:41:09.719 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-25T08:41:09.721 INFO:tasks.ceph.mgr.x:Started 2026-02-25T08:41:09.721 DEBUG:tasks.ceph:set 0 configs 2026-02-25T08:41:09.721 DEBUG:teuthology.orchestra.run.trial023:> sudo ceph --cluster ceph config dump 2026-02-25T08:41:09.779 INFO:tasks.ceph.mon.a.trial023.stderr:2026-02-25T08:41:09.777+0000 7f586c033c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:09.779 INFO:tasks.ceph.mon.a.trial023.stderr:2026-02-25T08:41:09.777+0000 7f586c033c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:09.779 INFO:tasks.ceph.mon.a.trial023.stderr:2026-02-25T08:41:09.777+0000 7f586c033c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:09.782 INFO:tasks.ceph.mgr.x.trial030.stderr:2026-02-25T08:41:09.780+0000 7f5faec87fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:09.795 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:09.791+0000 7fe6f403d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:09.799 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:09.797+0000 7fe6f403d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:09.806 INFO:tasks.ceph.mgr.x.trial030.stderr:2026-02-25T08:41:09.804+0000 7f5faec87fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:09.806 INFO:tasks.ceph.mgr.x.trial030.stderr:2026-02-25T08:41:09.805+0000 7f5faec87fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:09.833 INFO:teuthology.orchestra.run.trial023.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-25T08:41:09.841 INFO:tasks.ceph:Setting crush tunables to default 2026-02-25T08:41:09.841 DEBUG:teuthology.orchestra.run.trial023:> sudo ceph --cluster ceph osd crush tunables default 2026-02-25T08:41:09.915 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:09.910+0000 7fdd505b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:09.917 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:09.915+0000 7fdd505b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:09.952 INFO:teuthology.orchestra.run.trial023.stderr:adjusted tunables profile to default 2026-02-25T08:41:09.960 INFO:tasks.ceph:check_enable_crimson: True 2026-02-25T08:41:09.961 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2026-02-25T08:41:09.961 DEBUG:teuthology.orchestra.run.trial023:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2026-02-25T08:41:10.034 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:10.028+0000 7fe41550b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:10.037 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:10.034+0000 7fe41550b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:10.076 INFO:tasks.ceph.mon.a.trial023.stderr:2026-02-25T08:41:10.074+0000 7f58633f1640 -1 WARNING: experimental feature 'crimson' is enabled 2026-02-25T08:41:10.076 INFO:tasks.ceph.mon.a.trial023.stderr:Please be aware that this feature is experimental, untested, 2026-02-25T08:41:10.076 INFO:tasks.ceph.mon.a.trial023.stderr:unsupported, and may result in data corruption, data loss, 2026-02-25T08:41:10.076 INFO:tasks.ceph.mon.a.trial023.stderr:and/or irreparable damage to your cluster. Do not use 2026-02-25T08:41:10.077 INFO:tasks.ceph.mon.a.trial023.stderr:feature with important data. 2026-02-25T08:41:10.077 INFO:tasks.ceph.mon.a.trial023.stderr: 2026-02-25T08:41:10.644 INFO:tasks.ceph.mgr.x.trial030.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-25T08:41:10.644 INFO:tasks.ceph.mgr.x.trial030.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-25T08:41:10.644 INFO:tasks.ceph.mgr.x.trial030.stderr: from numpy import show_config as show_numpy_config 2026-02-25T08:41:10.963 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-25T08:41:10.964 DEBUG:teuthology.orchestra.run.trial023:> set -ex 2026-02-25T08:41:10.964 DEBUG:teuthology.orchestra.run.trial023:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-25T08:41:10.989 DEBUG:teuthology.orchestra.run.trial023:> set -ex 2026-02-25T08:41:10.989 DEBUG:teuthology.orchestra.run.trial023:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-25T08:41:11.052 DEBUG:teuthology.orchestra.run.trial030:> set -ex 2026-02-25T08:41:11.053 DEBUG:teuthology.orchestra.run.trial030:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-25T08:41:11.077 DEBUG:teuthology.orchestra.run.trial030:> set -ex 2026-02-25T08:41:11.078 DEBUG:teuthology.orchestra.run.trial030:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2026-02-25T08:41:11.143 DEBUG:teuthology.orchestra.run.trial030:> sudo ceph --cluster ceph osd new 55b6cf18-122b-4c40-9723-22d660c2ef92 0 2026-02-25T08:41:11.256 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-25T08:41:11.252+0000 7fe65a0cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:11.258 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-25T08:41:11.257+0000 7fe65a0cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:11.295 INFO:teuthology.orchestra.run.trial030.stdout:0 2026-02-25T08:41:11.303 DEBUG:teuthology.orchestra.run.trial030:> sudo ceph --cluster ceph osd new cdd52fd3-67fb-4f05-a93e-3956d3a24248 1 2026-02-25T08:41:11.376 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-25T08:41:11.371+0000 7fd0ed6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:11.378 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-25T08:41:11.377+0000 7fd0ed6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:11.415 INFO:teuthology.orchestra.run.trial030.stdout:1 2026-02-25T08:41:11.423 DEBUG:teuthology.orchestra.run.trial030:> sudo ceph --cluster ceph osd new d4f198cc-0208-4e4c-b8f1-b7a028c520ca 2 2026-02-25T08:41:11.496 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-25T08:41:11.491+0000 7f33eb0c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:11.499 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-25T08:41:11.497+0000 7f33eb0c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:11.536 INFO:teuthology.orchestra.run.trial030.stdout:2 2026-02-25T08:41:11.544 DEBUG:teuthology.orchestra.run.trial030:> sudo ceph --cluster ceph osd new edabd0ff-8d49-4327-9a44-5a140c74d128 3 2026-02-25T08:41:11.618 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-25T08:41:11.614+0000 7f6b8625a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:11.621 INFO:teuthology.orchestra.run.trial030.stderr:2026-02-25T08:41:11.620+0000 7f6b8625a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:11.658 INFO:teuthology.orchestra.run.trial030.stdout:3 2026-02-25T08:41:11.666 DEBUG:teuthology.orchestra.run.trial023:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2026-02-25T08:41:11.694 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-25T08:41:11.694 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2026-02-25T08:41:11.736 INFO:tasks.ceph.osd.0:Started 2026-02-25T08:41:11.736 DEBUG:teuthology.orchestra.run.trial023:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2026-02-25T08:41:11.764 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-25T08:41:11.764 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2026-02-25T08:41:11.806 INFO:tasks.ceph.osd.1:Started 2026-02-25T08:41:11.806 DEBUG:teuthology.orchestra.run.trial030:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2026-02-25T08:41:11.833 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-25T08:41:11.834 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2026-02-25T08:41:11.875 INFO:tasks.ceph.osd.2:Started 2026-02-25T08:41:11.876 DEBUG:teuthology.orchestra.run.trial030:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.3.log 2026-02-25T08:41:11.903 INFO:tasks.ceph.osd.3:Restarting daemon 2026-02-25T08:41:11.904 DEBUG:teuthology.orchestra.run.trial030:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2026-02-25T08:41:11.946 INFO:tasks.ceph.osd.3:Started 2026-02-25T08:41:11.947 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-25T08:41:12.023 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:12.019+0000 7fa58925a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:12.026 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:12.024+0000 7fa58925a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:12.060 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:41:12.060 INFO:teuthology.orchestra.run.trial023.stdout:{"epoch":7,"fsid":"542a2d4e-0dbc-4a03-9358-454614077093","created":"2026-02-25T08:41:09.790160+0000","modified":"2026-02-25T08:41:11.657137+0000","last_up_change":"0.000000","last_in_change":"2026-02-25T08:41:11.657137+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"55b6cf18-122b-4c40-9723-22d660c2ef92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cdd52fd3-67fb-4f05-a93e-3956d3a24248","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d4f198cc-0208-4e4c-b8f1-b7a028c520ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"edabd0ff-8d49-4327-9a44-5a140c74d128","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T08:41:12.069 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-25T08:41:12.070 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-25T08:41:12.473 DEBUG:teuthology.orchestra.run.trial023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-25T08:41:12.539 INFO:teuthology.misc.health.trial023.stderr:2026-02-25T08:41:12.535+0000 7fdc10bd3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:12.542 INFO:teuthology.misc.health.trial023.stderr:2026-02-25T08:41:12.540+0000 7fdc10bd3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:12.575 INFO:teuthology.misc.health.trial023.stdout: 2026-02-25T08:41:12.575 INFO:teuthology.misc.health.trial023.stdout:{"epoch":7,"fsid":"542a2d4e-0dbc-4a03-9358-454614077093","created":"2026-02-25T08:41:09.790160+0000","modified":"2026-02-25T08:41:11.657137+0000","last_up_change":"0.000000","last_in_change":"2026-02-25T08:41:11.657137+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"55b6cf18-122b-4c40-9723-22d660c2ef92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cdd52fd3-67fb-4f05-a93e-3956d3a24248","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d4f198cc-0208-4e4c-b8f1-b7a028c520ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"edabd0ff-8d49-4327-9a44-5a140c74d128","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T08:41:12.582 DEBUG:teuthology.misc:0 of 4 OSDs are up 2026-02-25T08:41:13.018 INFO:tasks.ceph.mgr.x.trial030.stderr:2026-02-25T08:41:13.017+0000 7f5eca16a640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-25T08:41:18.985 DEBUG:teuthology.orchestra.run.trial023:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-25T08:41:19.056 INFO:teuthology.misc.health.trial023.stderr:2026-02-25T08:41:19.051+0000 7f3b10ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:19.058 INFO:teuthology.misc.health.trial023.stderr:2026-02-25T08:41:19.056+0000 7f3b10ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:19.168 INFO:teuthology.misc.health.trial023.stdout: 2026-02-25T08:41:19.168 INFO:teuthology.misc.health.trial023.stdout:{"epoch":11,"fsid":"542a2d4e-0dbc-4a03-9358-454614077093","created":"2026-02-25T08:41:09.790160+0000","modified":"2026-02-25T08:41:19.157809+0000","last_up_change":"2026-02-25T08:41:19.157809+0000","last_in_change":"2026-02-25T08:41:11.657137+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"55b6cf18-122b-4c40-9723-22d660c2ef92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6802","nonce":1434923082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6803","nonce":1434923082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6807","nonce":1434923082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6806","nonce":1434923082}]},"public_addr":"10.20.193.23:6802/1434923082","cluster_addr":"10.20.193.23:6803/1434923082","heartbeat_back_addr":"10.20.193.23:6807/1434923082","heartbeat_front_addr":"10.20.193.23:6806/1434923082","state":["exists","up"]},{"osd":1,"uuid":"cdd52fd3-67fb-4f05-a93e-3956d3a24248","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6801","nonce":2628765320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6800","nonce":2628765320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6808","nonce":2628765320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6809","nonce":2628765320}]},"public_addr":"10.20.193.23:6801/2628765320","cluster_addr":"10.20.193.23:6800/2628765320","heartbeat_back_addr":"10.20.193.23:6808/2628765320","heartbeat_front_addr":"10.20.193.23:6809/2628765320","state":["exists","up"]},{"osd":2,"uuid":"d4f198cc-0208-4e4c-b8f1-b7a028c520ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6802","nonce":185076637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6803","nonce":185076637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6807","nonce":185076637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6806","nonce":185076637}]},"public_addr":"10.20.193.30:6802/185076637","cluster_addr":"10.20.193.30:6803/185076637","heartbeat_back_addr":"10.20.193.30:6807/185076637","heartbeat_front_addr":"10.20.193.30:6806/185076637","state":["exists","up"]},{"osd":3,"uuid":"edabd0ff-8d49-4327-9a44-5a140c74d128","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6804","nonce":2656603590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6805","nonce":2656603590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6808","nonce":2656603590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6809","nonce":2656603590}]},"public_addr":"10.20.193.30:6804/2656603590","cluster_addr":"10.20.193.30:6805/2656603590","heartbeat_back_addr":"10.20.193.30:6808/2656603590","heartbeat_front_addr":"10.20.193.30:6809/2656603590","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T08:41:19.178 DEBUG:teuthology.misc:4 of 4 OSDs are up 2026-02-25T08:41:19.178 INFO:tasks.ceph:Creating RBD pool 2026-02-25T08:41:19.178 DEBUG:teuthology.orchestra.run.trial023:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-25T08:41:19.250 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:19.244+0000 7f41adccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:19.253 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:19.251+0000 7f41adccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:20.168 INFO:teuthology.orchestra.run.trial023.stderr:pool 'rbd' created 2026-02-25T08:41:20.182 DEBUG:teuthology.orchestra.run.trial023:> rbd --cluster ceph pool init rbd 2026-02-25T08:41:20.225 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:20.223+0000 7fa8e3233240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:20.226 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:20.223+0000 7fa8e3233240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:20.229 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:20.227+0000 7fa8e3233240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:23.206 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-25T08:41:23.207 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2026-02-25T08:41:23.208 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-25T08:41:23.323 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:23.318+0000 7f706e21a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:23.325 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:23.323+0000 7f706e21a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:41:23.446 INFO:teuthology.orchestra.run.trial023.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2026-02-25T08:41:23.446 INFO:tasks.ceph_manager:config epoch is 1 2026-02-25T08:41:23.446 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-25T08:41:23.446 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-25T08:41:23.447 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-25T08:41:23.523 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:23.519+0000 7f5c5255c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:23.525 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:23.523+0000 7f5c5255c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:23.669 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:41:23.678 INFO:teuthology.orchestra.run.trial023.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6800","nonce":1538846400},{"type":"v1","addr":"10.20.193.30:6801","nonce":1538846400}]},"active_addr":"10.20.193.30:6801/1538846400","active_change":"2026-02-25T08:41:12.011031+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often the agent on each host will gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"The first TCP port the agent will try to bind to. Up to 1000 subsequent ports will be attempted if this port cannot be bound","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this option can facilitate debugging daemons that encounter runtime problems.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"Fraction of total system memory to divide among autotuned daemons. Converged systems hosting both Ceph and compute should set a lower value; systems with more than 10 GiB of memory per OSD may benefit from a slightly larger value so that the abundance of RAM can be fully utilized","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self-signed certificates generated and signed by the cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently Podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_alloy":{"name":"container_image_alloy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/alloy:latest","min":"","max":"","enum_allowed":[],"desc":"Alloy container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.6","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"Location of dashboards to include in Grafana","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether the hardware monitoring daemon be deployed on every host?","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at the debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of daemons per service per host: guard against mistakes and runaways","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of OSDs that will be drained simultaneously when OSDs are removed","long_desc":"","tags":[],"see_also":[]},"max_parallel_osd_upgrades":{"name":"max_parallel_osd_upgrades","type":"int","level":"advanced","flags":0,"default_value":"16","min":"","max":"","enum_allowed":[],"desc":"Maximum number of OSD daemons upgraded in parallel.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"Mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default IP address for RedFish API (OOB management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"Location of alerts to include in Prometheus Alertmanager","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry URL for login purposes. This is not the default registry.","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"Cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Customized SSH config file to distribute to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many sequential SSH connection liveness check failures before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often SSH connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"How frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use the cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest to ensure that all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":true,"error_string":"","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"healthcheck_history_max_entries":{"name":"healthcheck_history_max_entries","type":"int","level":"advanced","flags":1,"default_value":"1000","min":"","max":"","enum_allowed":[],"desc":"Maximum number of health check history entries to keep","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.30:0","nonce":509125485}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.30:0","nonce":2618555674}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.30:0","nonce":956259295}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.30:0","nonce":1030115679}]}]} 2026-02-25T08:41:23.683 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-25T08:41:23.683 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-25T08:41:23.683 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-25T08:41:23.759 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:23.755+0000 7f2034c06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:23.762 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:23.760+0000 7f2034c06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:23.873 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:41:23.874 INFO:teuthology.orchestra.run.trial023.stdout:{"epoch":15,"fsid":"542a2d4e-0dbc-4a03-9358-454614077093","created":"2026-02-25T08:41:09.790160+0000","modified":"2026-02-25T08:41:23.176106+0000","last_up_change":"2026-02-25T08:41:19.157809+0000","last_in_change":"2026-02-25T08:41:11.657137+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-25T08:41:19.368756+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2026-02-25T08:41:20.034606+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"55b6cf18-122b-4c40-9723-22d660c2ef92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6802","nonce":1434923082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6803","nonce":1434923082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6807","nonce":1434923082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6806","nonce":1434923082}]},"public_addr":"10.20.193.23:6802/1434923082","cluster_addr":"10.20.193.23:6803/1434923082","heartbeat_back_addr":"10.20.193.23:6807/1434923082","heartbeat_front_addr":"10.20.193.23:6806/1434923082","state":["exists","up"]},{"osd":1,"uuid":"cdd52fd3-67fb-4f05-a93e-3956d3a24248","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6801","nonce":2628765320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6800","nonce":2628765320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6808","nonce":2628765320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6809","nonce":2628765320}]},"public_addr":"10.20.193.23:6801/2628765320","cluster_addr":"10.20.193.23:6800/2628765320","heartbeat_back_addr":"10.20.193.23:6808/2628765320","heartbeat_front_addr":"10.20.193.23:6809/2628765320","state":["exists","up"]},{"osd":2,"uuid":"d4f198cc-0208-4e4c-b8f1-b7a028c520ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6802","nonce":185076637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6803","nonce":185076637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6807","nonce":185076637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6806","nonce":185076637}]},"public_addr":"10.20.193.30:6802/185076637","cluster_addr":"10.20.193.30:6803/185076637","heartbeat_back_addr":"10.20.193.30:6807/185076637","heartbeat_front_addr":"10.20.193.30:6806/185076637","state":["exists","up"]},{"osd":3,"uuid":"edabd0ff-8d49-4327-9a44-5a140c74d128","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6804","nonce":2656603590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6805","nonce":2656603590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6808","nonce":2656603590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6809","nonce":2656603590}]},"public_addr":"10.20.193.30:6804/2656603590","cluster_addr":"10.20.193.30:6805/2656603590","heartbeat_back_addr":"10.20.193.30:6808/2656603590","heartbeat_front_addr":"10.20.193.30:6809/2656603590","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T08:41:23.883 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-25T08:41:23.883 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-25T08:41:23.957 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:23.953+0000 7f2c975a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:23.959 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:23.957+0000 7f2c975a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:24.072 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:41:24.073 INFO:teuthology.orchestra.run.trial023.stdout:{"epoch":15,"fsid":"542a2d4e-0dbc-4a03-9358-454614077093","created":"2026-02-25T08:41:09.790160+0000","modified":"2026-02-25T08:41:23.176106+0000","last_up_change":"2026-02-25T08:41:19.157809+0000","last_in_change":"2026-02-25T08:41:11.657137+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-25T08:41:19.368756+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2026-02-25T08:41:20.034606+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"55b6cf18-122b-4c40-9723-22d660c2ef92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6802","nonce":1434923082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6803","nonce":1434923082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6807","nonce":1434923082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6806","nonce":1434923082}]},"public_addr":"10.20.193.23:6802/1434923082","cluster_addr":"10.20.193.23:6803/1434923082","heartbeat_back_addr":"10.20.193.23:6807/1434923082","heartbeat_front_addr":"10.20.193.23:6806/1434923082","state":["exists","up"]},{"osd":1,"uuid":"cdd52fd3-67fb-4f05-a93e-3956d3a24248","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6801","nonce":2628765320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6800","nonce":2628765320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6808","nonce":2628765320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.23:6809","nonce":2628765320}]},"public_addr":"10.20.193.23:6801/2628765320","cluster_addr":"10.20.193.23:6800/2628765320","heartbeat_back_addr":"10.20.193.23:6808/2628765320","heartbeat_front_addr":"10.20.193.23:6809/2628765320","state":["exists","up"]},{"osd":2,"uuid":"d4f198cc-0208-4e4c-b8f1-b7a028c520ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6802","nonce":185076637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6803","nonce":185076637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6807","nonce":185076637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6806","nonce":185076637}]},"public_addr":"10.20.193.30:6802/185076637","cluster_addr":"10.20.193.30:6803/185076637","heartbeat_back_addr":"10.20.193.30:6807/185076637","heartbeat_front_addr":"10.20.193.30:6806/185076637","state":["exists","up"]},{"osd":3,"uuid":"edabd0ff-8d49-4327-9a44-5a140c74d128","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6804","nonce":2656603590}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6805","nonce":2656603590}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6808","nonce":2656603590}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.30:6809","nonce":2656603590}]},"public_addr":"10.20.193.30:6804/2656603590","cluster_addr":"10.20.193.30:6805/2656603590","heartbeat_back_addr":"10.20.193.30:6808/2656603590","heartbeat_front_addr":"10.20.193.30:6809/2656603590","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T08:41:24.082 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-25T08:41:24.083 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-25T08:41:24.084 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-25T08:41:24.084 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-25T08:41:24.166 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:24.161+0000 7f0e1a6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:24.167 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:24.161+0000 7f5d25c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:24.168 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:24.163+0000 7fd10d7dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:24.169 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:24.167+0000 7f5d25c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:24.169 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:24.167+0000 7f0e1a6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:24.170 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:24.166+0000 7fce9de5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:24.171 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:24.169+0000 7fd10d7dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:24.173 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:24.171+0000 7fce9de5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:24.188 INFO:teuthology.orchestra.run.trial023.stdout:47244640261 2026-02-25T08:41:24.188 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-25T08:41:24.189 INFO:teuthology.orchestra.run.trial023.stdout:47244640261 2026-02-25T08:41:24.190 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-25T08:41:24.191 INFO:teuthology.orchestra.run.trial023.stdout:42949672966 2026-02-25T08:41:24.192 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-25T08:41:24.193 INFO:teuthology.orchestra.run.trial023.stdout:47244640261 2026-02-25T08:41:24.193 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-25T08:41:24.309 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:24.304+0000 7fcffe18c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:24.311 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:24.306+0000 7f87ea3a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:24.312 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:24.310+0000 7fcffe18c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:24.314 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:24.312+0000 7f87ea3a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:24.351 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:24.346+0000 7f4fc665a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:24.353 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:24.347+0000 7fc5620cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:24.354 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:24.352+0000 7f4fc665a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:24.355 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:24.353+0000 7fc5620cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:24.428 INFO:teuthology.orchestra.run.trial023.stdout:47244640261 2026-02-25T08:41:24.434 INFO:teuthology.orchestra.run.trial023.stdout:47244640261 2026-02-25T08:41:24.440 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.0 2026-02-25T08:41:24.440 DEBUG:teuthology.parallel:result is None 2026-02-25T08:41:24.445 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.1 2026-02-25T08:41:24.445 DEBUG:teuthology.parallel:result is None 2026-02-25T08:41:24.470 INFO:teuthology.orchestra.run.trial023.stdout:47244640261 2026-02-25T08:41:24.474 INFO:teuthology.orchestra.run.trial023.stdout:42949672966 2026-02-25T08:41:24.480 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.3 2026-02-25T08:41:24.481 DEBUG:teuthology.parallel:result is None 2026-02-25T08:41:24.484 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672966 for osd.2 2026-02-25T08:41:24.485 DEBUG:teuthology.parallel:result is None 2026-02-25T08:41:24.485 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-25T08:41:24.485 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:41:24.601 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:24.597+0000 7fe9c457e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:24.604 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:24.602+0000 7fe9c457e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:24.714 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:41:24.714 INFO:teuthology.orchestra.run.trial023.stderr:dumped all 2026-02-25T08:41:24.723 INFO:teuthology.orchestra.run.trial023.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2026-02-25T08:41:24.183133+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209715136,"kb_used":83890120,"kb_used_data":83890120,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":125825016,"statfs":{"total":214748299264,"available":128844816384,"internally_reserved":0,"allocated":85903482880,"data_stored":85903482880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.191682+0000","last_change":"2026-02-25T08:41:23.195394+0000","last_active":"2026-02-25T08:41:23.191682+0000","last_peered":"2026-02-25T08:41:23.191682+0000","last_clean":"2026-02-25T08:41:23.191682+0000","last_became_active":"2026-02-25T08:41:21.223518+0000","last_became_peered":"2026-02-25T08:41:21.223518+0000","last_unstale":"2026-02-25T08:41:23.191682+0000","last_undegraded":"2026-02-25T08:41:23.191682+0000","last_fullsized":"2026-02-25T08:41:23.191682+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.190363+0000","last_change":"2026-02-25T08:41:23.192509+0000","last_active":"2026-02-25T08:41:23.190363+0000","last_peered":"2026-02-25T08:41:23.190363+0000","last_clean":"2026-02-25T08:41:23.190363+0000","last_became_active":"2026-02-25T08:41:21.240067+0000","last_became_peered":"2026-02-25T08:41:21.240067+0000","last_unstale":"2026-02-25T08:41:23.190363+0000","last_undegraded":"2026-02-25T08:41:23.190363+0000","last_fullsized":"2026-02-25T08:41:23.190363+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.190372+0000","last_change":"2026-02-25T08:41:23.195131+0000","last_active":"2026-02-25T08:41:23.190372+0000","last_peered":"2026-02-25T08:41:23.190372+0000","last_clean":"2026-02-25T08:41:23.190372+0000","last_became_active":"2026-02-25T08:41:21.226683+0000","last_became_peered":"2026-02-25T08:41:21.226683+0000","last_unstale":"2026-02-25T08:41:23.190372+0000","last_undegraded":"2026-02-25T08:41:23.190372+0000","last_fullsized":"2026-02-25T08:41:23.190372+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.190142+0000","last_change":"2026-02-25T08:41:23.192156+0000","last_active":"2026-02-25T08:41:23.190142+0000","last_peered":"2026-02-25T08:41:23.190142+0000","last_clean":"2026-02-25T08:41:23.190142+0000","last_became_active":"2026-02-25T08:41:21.216122+0000","last_became_peered":"2026-02-25T08:41:21.216122+0000","last_unstale":"2026-02-25T08:41:23.190142+0000","last_undegraded":"2026-02-25T08:41:23.190142+0000","last_fullsized":"2026-02-25T08:41:23.190142+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"13'52","reported_seq":82,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.189947+0000","last_change":"2026-02-25T08:41:21.235650+0000","last_active":"2026-02-25T08:41:23.189947+0000","last_peered":"2026-02-25T08:41:23.189947+0000","last_clean":"2026-02-25T08:41:23.189947+0000","last_became_active":"2026-02-25T08:41:21.226864+0000","last_became_peered":"2026-02-25T08:41:21.226864+0000","last_unstale":"2026-02-25T08:41:23.189947+0000","last_undegraded":"2026-02-25T08:41:23.189947+0000","last_fullsized":"2026-02-25T08:41:23.189947+0000","mapping_epoch":12,"log_start":"13'48","ondisk_log_start":"13'48","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":48,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.188719+0000","last_change":"2026-02-25T08:41:23.191189+0000","last_active":"2026-02-25T08:41:23.188719+0000","last_peered":"2026-02-25T08:41:23.188719+0000","last_clean":"2026-02-25T08:41:23.188719+0000","last_became_active":"2026-02-25T08:41:21.216133+0000","last_became_peered":"2026-02-25T08:41:21.216133+0000","last_unstale":"2026-02-25T08:41:23.188719+0000","last_undegraded":"2026-02-25T08:41:23.188719+0000","last_fullsized":"2026-02-25T08:41:23.188719+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"14'1","reported_seq":18,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.192134+0000","last_change":"2026-02-25T08:41:23.192134+0000","last_active":"2026-02-25T08:41:23.192134+0000","last_peered":"2026-02-25T08:41:23.192134+0000","last_clean":"2026-02-25T08:41:23.192134+0000","last_became_active":"2026-02-25T08:41:21.215058+0000","last_became_peered":"2026-02-25T08:41:21.215058+0000","last_unstale":"2026-02-25T08:41:23.192134+0000","last_undegraded":"2026-02-25T08:41:23.192134+0000","last_fullsized":"2026-02-25T08:41:23.192134+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.188725+0000","last_change":"2026-02-25T08:41:23.191858+0000","last_active":"2026-02-25T08:41:23.188725+0000","last_peered":"2026-02-25T08:41:23.188725+0000","last_clean":"2026-02-25T08:41:23.188725+0000","last_became_active":"2026-02-25T08:41:21.215894+0000","last_became_peered":"2026-02-25T08:41:21.215894+0000","last_unstale":"2026-02-25T08:41:23.188725+0000","last_undegraded":"2026-02-25T08:41:23.188725+0000","last_fullsized":"2026-02-25T08:41:23.188725+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.188713+0000","last_change":"2026-02-25T08:41:23.191742+0000","last_active":"2026-02-25T08:41:23.188713+0000","last_peered":"2026-02-25T08:41:23.188713+0000","last_clean":"2026-02-25T08:41:23.188713+0000","last_became_active":"2026-02-25T08:41:21.218886+0000","last_became_peered":"2026-02-25T08:41:21.218886+0000","last_unstale":"2026-02-25T08:41:23.188713+0000","last_undegraded":"2026-02-25T08:41:23.188713+0000","last_fullsized":"2026-02-25T08:41:23.188713+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32210972672,"internally_reserved":0,"allocated":21476102144,"data_stored":21476102144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96633774080,"internally_reserved":0,"allocated":64427450368,"data_stored":64427450368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":24,"acting":24,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972724,"kb_used_data":20972724,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456060,"statfs":{"total":53687074816,"available":32211005440,"internally_reserved":0,"allocated":21476069376,"data_stored":21476069376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972688,"kb_used_data":20972688,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456096,"statfs":{"total":53687074816,"available":32211042304,"internally_reserved":0,"allocated":21476032512,"data_stored":21476032512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20971952,"kb_used_data":20971952,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456832,"statfs":{"total":53687074816,"available":32211795968,"internally_reserved":0,"allocated":21475278848,"data_stored":21475278848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972756,"kb_used_data":20972756,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456028,"statfs":{"total":53687074816,"available":32210972672,"internally_reserved":0,"allocated":21476102144,"data_stored":21476102144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53687074816,"available":32210972672,"internally_reserved":0,"allocated":21476102144,"data_stored":21476102144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53687074816,"available":32211795968,"internally_reserved":0,"allocated":21475278848,"data_stored":21475278848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":53687074816,"available":32211005440,"internally_reserved":0,"allocated":21476069376,"data_stored":21476069376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53687074816,"available":32210972672,"internally_reserved":0,"allocated":21476102144,"data_stored":21476102144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T08:41:24.724 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:41:24.802 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:24.798+0000 7fb295651640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:24.805 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:24.803+0000 7fb295651640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:24.915 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:41:24.916 INFO:teuthology.orchestra.run.trial023.stderr:dumped all 2026-02-25T08:41:24.924 INFO:teuthology.orchestra.run.trial023.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2026-02-25T08:41:24.183133+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209715136,"kb_used":83890120,"kb_used_data":83890120,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":125825016,"statfs":{"total":214748299264,"available":128844816384,"internally_reserved":0,"allocated":85903482880,"data_stored":85903482880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.191682+0000","last_change":"2026-02-25T08:41:23.195394+0000","last_active":"2026-02-25T08:41:23.191682+0000","last_peered":"2026-02-25T08:41:23.191682+0000","last_clean":"2026-02-25T08:41:23.191682+0000","last_became_active":"2026-02-25T08:41:21.223518+0000","last_became_peered":"2026-02-25T08:41:21.223518+0000","last_unstale":"2026-02-25T08:41:23.191682+0000","last_undegraded":"2026-02-25T08:41:23.191682+0000","last_fullsized":"2026-02-25T08:41:23.191682+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.190363+0000","last_change":"2026-02-25T08:41:23.192509+0000","last_active":"2026-02-25T08:41:23.190363+0000","last_peered":"2026-02-25T08:41:23.190363+0000","last_clean":"2026-02-25T08:41:23.190363+0000","last_became_active":"2026-02-25T08:41:21.240067+0000","last_became_peered":"2026-02-25T08:41:21.240067+0000","last_unstale":"2026-02-25T08:41:23.190363+0000","last_undegraded":"2026-02-25T08:41:23.190363+0000","last_fullsized":"2026-02-25T08:41:23.190363+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.190372+0000","last_change":"2026-02-25T08:41:23.195131+0000","last_active":"2026-02-25T08:41:23.190372+0000","last_peered":"2026-02-25T08:41:23.190372+0000","last_clean":"2026-02-25T08:41:23.190372+0000","last_became_active":"2026-02-25T08:41:21.226683+0000","last_became_peered":"2026-02-25T08:41:21.226683+0000","last_unstale":"2026-02-25T08:41:23.190372+0000","last_undegraded":"2026-02-25T08:41:23.190372+0000","last_fullsized":"2026-02-25T08:41:23.190372+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.190142+0000","last_change":"2026-02-25T08:41:23.192156+0000","last_active":"2026-02-25T08:41:23.190142+0000","last_peered":"2026-02-25T08:41:23.190142+0000","last_clean":"2026-02-25T08:41:23.190142+0000","last_became_active":"2026-02-25T08:41:21.216122+0000","last_became_peered":"2026-02-25T08:41:21.216122+0000","last_unstale":"2026-02-25T08:41:23.190142+0000","last_undegraded":"2026-02-25T08:41:23.190142+0000","last_fullsized":"2026-02-25T08:41:23.190142+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"13'52","reported_seq":82,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.189947+0000","last_change":"2026-02-25T08:41:21.235650+0000","last_active":"2026-02-25T08:41:23.189947+0000","last_peered":"2026-02-25T08:41:23.189947+0000","last_clean":"2026-02-25T08:41:23.189947+0000","last_became_active":"2026-02-25T08:41:21.226864+0000","last_became_peered":"2026-02-25T08:41:21.226864+0000","last_unstale":"2026-02-25T08:41:23.189947+0000","last_undegraded":"2026-02-25T08:41:23.189947+0000","last_fullsized":"2026-02-25T08:41:23.189947+0000","mapping_epoch":12,"log_start":"13'48","ondisk_log_start":"13'48","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":48,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.188719+0000","last_change":"2026-02-25T08:41:23.191189+0000","last_active":"2026-02-25T08:41:23.188719+0000","last_peered":"2026-02-25T08:41:23.188719+0000","last_clean":"2026-02-25T08:41:23.188719+0000","last_became_active":"2026-02-25T08:41:21.216133+0000","last_became_peered":"2026-02-25T08:41:21.216133+0000","last_unstale":"2026-02-25T08:41:23.188719+0000","last_undegraded":"2026-02-25T08:41:23.188719+0000","last_fullsized":"2026-02-25T08:41:23.188719+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"14'1","reported_seq":18,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.192134+0000","last_change":"2026-02-25T08:41:23.192134+0000","last_active":"2026-02-25T08:41:23.192134+0000","last_peered":"2026-02-25T08:41:23.192134+0000","last_clean":"2026-02-25T08:41:23.192134+0000","last_became_active":"2026-02-25T08:41:21.215058+0000","last_became_peered":"2026-02-25T08:41:21.215058+0000","last_unstale":"2026-02-25T08:41:23.192134+0000","last_undegraded":"2026-02-25T08:41:23.192134+0000","last_fullsized":"2026-02-25T08:41:23.192134+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.188725+0000","last_change":"2026-02-25T08:41:23.191858+0000","last_active":"2026-02-25T08:41:23.188725+0000","last_peered":"2026-02-25T08:41:23.188725+0000","last_clean":"2026-02-25T08:41:23.188725+0000","last_became_active":"2026-02-25T08:41:21.215894+0000","last_became_peered":"2026-02-25T08:41:21.215894+0000","last_unstale":"2026-02-25T08:41:23.188725+0000","last_undegraded":"2026-02-25T08:41:23.188725+0000","last_fullsized":"2026-02-25T08:41:23.188725+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.188713+0000","last_change":"2026-02-25T08:41:23.191742+0000","last_active":"2026-02-25T08:41:23.188713+0000","last_peered":"2026-02-25T08:41:23.188713+0000","last_clean":"2026-02-25T08:41:23.188713+0000","last_became_active":"2026-02-25T08:41:21.218886+0000","last_became_peered":"2026-02-25T08:41:21.218886+0000","last_unstale":"2026-02-25T08:41:23.188713+0000","last_undegraded":"2026-02-25T08:41:23.188713+0000","last_fullsized":"2026-02-25T08:41:23.188713+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32210972672,"internally_reserved":0,"allocated":21476102144,"data_stored":21476102144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96633774080,"internally_reserved":0,"allocated":64427450368,"data_stored":64427450368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":24,"acting":24,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972724,"kb_used_data":20972724,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456060,"statfs":{"total":53687074816,"available":32211005440,"internally_reserved":0,"allocated":21476069376,"data_stored":21476069376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972688,"kb_used_data":20972688,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456096,"statfs":{"total":53687074816,"available":32211042304,"internally_reserved":0,"allocated":21476032512,"data_stored":21476032512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20971952,"kb_used_data":20971952,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456832,"statfs":{"total":53687074816,"available":32211795968,"internally_reserved":0,"allocated":21475278848,"data_stored":21475278848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972756,"kb_used_data":20972756,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456028,"statfs":{"total":53687074816,"available":32210972672,"internally_reserved":0,"allocated":21476102144,"data_stored":21476102144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53687074816,"available":32210972672,"internally_reserved":0,"allocated":21476102144,"data_stored":21476102144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53687074816,"available":32211795968,"internally_reserved":0,"allocated":21475278848,"data_stored":21475278848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":53687074816,"available":32211005440,"internally_reserved":0,"allocated":21476069376,"data_stored":21476069376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53687074816,"available":32210972672,"internally_reserved":0,"allocated":21476102144,"data_stored":21476102144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T08:41:24.926 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-25T08:41:24.926 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-25T08:41:24.926 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-25T08:41:24.926 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-25T08:41:25.002 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:24.998+0000 7fd0cf3b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:25.004 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:25.002+0000 7fd0cf3b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:25.142 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:41:25.142 INFO:teuthology.orchestra.run.trial023.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-25T08:41:25.151 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-25T08:41:25.152 INFO:teuthology.run_tasks:Running task thrashosds... 2026-02-25T08:41:25.171 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_trim_stale_osdmaps': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2026-02-25T08:41:25.172 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-logstoremanager'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd target transaction size': 5, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Degraded data redundancy', 'is down', 'osds down', 'pg .*? is .*?degraded.*?, acting', 'pg .*? is stuck', 'pg degraded', 'PG_BACKFILL_FULL', 'Low space hindering backfill .*? backfill_toofull', 'OSD_ROOT_DOWN', 'pgs degraded', 'pgs undersized', 'is active.*backfill_toofull.*'], 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'c32469671c6b0222f786dd36c34eb702f780a11d'}} 2026-02-25T08:41:25.172 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_trim_stale_osdmaps': 0, 'dump_ops_enable': False, 'min_in': 4, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2026-02-25T08:41:25.172 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2026-02-25T08:41:25.250 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:25.244+0000 7fe66750b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:25.252 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:25.250+0000 7fe66750b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:25.378 INFO:tasks.thrashosds:Beginning thrashosds... 2026-02-25T08:41:25.379 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-25T08:41:25.379 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:41:25.455 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:25.450+0000 7ff22cd5b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:25.457 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:25.456+0000 7ff22cd5b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:25.566 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:41:25.566 INFO:teuthology.orchestra.run.trial023.stderr:dumped all 2026-02-25T08:41:25.575 INFO:teuthology.orchestra.run.trial023.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2026-02-25T08:41:24.183133+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209715136,"kb_used":83890120,"kb_used_data":83890120,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":125825016,"statfs":{"total":214748299264,"available":128844816384,"internally_reserved":0,"allocated":85903482880,"data_stored":85903482880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.191682+0000","last_change":"2026-02-25T08:41:23.195394+0000","last_active":"2026-02-25T08:41:23.191682+0000","last_peered":"2026-02-25T08:41:23.191682+0000","last_clean":"2026-02-25T08:41:23.191682+0000","last_became_active":"2026-02-25T08:41:21.223518+0000","last_became_peered":"2026-02-25T08:41:21.223518+0000","last_unstale":"2026-02-25T08:41:23.191682+0000","last_undegraded":"2026-02-25T08:41:23.191682+0000","last_fullsized":"2026-02-25T08:41:23.191682+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.190363+0000","last_change":"2026-02-25T08:41:23.192509+0000","last_active":"2026-02-25T08:41:23.190363+0000","last_peered":"2026-02-25T08:41:23.190363+0000","last_clean":"2026-02-25T08:41:23.190363+0000","last_became_active":"2026-02-25T08:41:21.240067+0000","last_became_peered":"2026-02-25T08:41:21.240067+0000","last_unstale":"2026-02-25T08:41:23.190363+0000","last_undegraded":"2026-02-25T08:41:23.190363+0000","last_fullsized":"2026-02-25T08:41:23.190363+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.190372+0000","last_change":"2026-02-25T08:41:23.195131+0000","last_active":"2026-02-25T08:41:23.190372+0000","last_peered":"2026-02-25T08:41:23.190372+0000","last_clean":"2026-02-25T08:41:23.190372+0000","last_became_active":"2026-02-25T08:41:21.226683+0000","last_became_peered":"2026-02-25T08:41:21.226683+0000","last_unstale":"2026-02-25T08:41:23.190372+0000","last_undegraded":"2026-02-25T08:41:23.190372+0000","last_fullsized":"2026-02-25T08:41:23.190372+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.190142+0000","last_change":"2026-02-25T08:41:23.192156+0000","last_active":"2026-02-25T08:41:23.190142+0000","last_peered":"2026-02-25T08:41:23.190142+0000","last_clean":"2026-02-25T08:41:23.190142+0000","last_became_active":"2026-02-25T08:41:21.216122+0000","last_became_peered":"2026-02-25T08:41:21.216122+0000","last_unstale":"2026-02-25T08:41:23.190142+0000","last_undegraded":"2026-02-25T08:41:23.190142+0000","last_fullsized":"2026-02-25T08:41:23.190142+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"13'52","reported_seq":82,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.189947+0000","last_change":"2026-02-25T08:41:21.235650+0000","last_active":"2026-02-25T08:41:23.189947+0000","last_peered":"2026-02-25T08:41:23.189947+0000","last_clean":"2026-02-25T08:41:23.189947+0000","last_became_active":"2026-02-25T08:41:21.226864+0000","last_became_peered":"2026-02-25T08:41:21.226864+0000","last_unstale":"2026-02-25T08:41:23.189947+0000","last_undegraded":"2026-02-25T08:41:23.189947+0000","last_fullsized":"2026-02-25T08:41:23.189947+0000","mapping_epoch":12,"log_start":"13'48","ondisk_log_start":"13'48","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":48,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.188719+0000","last_change":"2026-02-25T08:41:23.191189+0000","last_active":"2026-02-25T08:41:23.188719+0000","last_peered":"2026-02-25T08:41:23.188719+0000","last_clean":"2026-02-25T08:41:23.188719+0000","last_became_active":"2026-02-25T08:41:21.216133+0000","last_became_peered":"2026-02-25T08:41:21.216133+0000","last_unstale":"2026-02-25T08:41:23.188719+0000","last_undegraded":"2026-02-25T08:41:23.188719+0000","last_fullsized":"2026-02-25T08:41:23.188719+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"14'1","reported_seq":18,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.192134+0000","last_change":"2026-02-25T08:41:23.192134+0000","last_active":"2026-02-25T08:41:23.192134+0000","last_peered":"2026-02-25T08:41:23.192134+0000","last_clean":"2026-02-25T08:41:23.192134+0000","last_became_active":"2026-02-25T08:41:21.215058+0000","last_became_peered":"2026-02-25T08:41:21.215058+0000","last_unstale":"2026-02-25T08:41:23.192134+0000","last_undegraded":"2026-02-25T08:41:23.192134+0000","last_fullsized":"2026-02-25T08:41:23.192134+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.188725+0000","last_change":"2026-02-25T08:41:23.191858+0000","last_active":"2026-02-25T08:41:23.188725+0000","last_peered":"2026-02-25T08:41:23.188725+0000","last_clean":"2026-02-25T08:41:23.188725+0000","last_became_active":"2026-02-25T08:41:21.215894+0000","last_became_peered":"2026-02-25T08:41:21.215894+0000","last_unstale":"2026-02-25T08:41:23.188725+0000","last_undegraded":"2026-02-25T08:41:23.188725+0000","last_fullsized":"2026-02-25T08:41:23.188725+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.188713+0000","last_change":"2026-02-25T08:41:23.191742+0000","last_active":"2026-02-25T08:41:23.188713+0000","last_peered":"2026-02-25T08:41:23.188713+0000","last_clean":"2026-02-25T08:41:23.188713+0000","last_became_active":"2026-02-25T08:41:21.218886+0000","last_became_peered":"2026-02-25T08:41:21.218886+0000","last_unstale":"2026-02-25T08:41:23.188713+0000","last_undegraded":"2026-02-25T08:41:23.188713+0000","last_fullsized":"2026-02-25T08:41:23.188713+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32210972672,"internally_reserved":0,"allocated":21476102144,"data_stored":21476102144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96633774080,"internally_reserved":0,"allocated":64427450368,"data_stored":64427450368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":24,"acting":24,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972724,"kb_used_data":20972724,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456060,"statfs":{"total":53687074816,"available":32211005440,"internally_reserved":0,"allocated":21476069376,"data_stored":21476069376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972688,"kb_used_data":20972688,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456096,"statfs":{"total":53687074816,"available":32211042304,"internally_reserved":0,"allocated":21476032512,"data_stored":21476032512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20971952,"kb_used_data":20971952,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456832,"statfs":{"total":53687074816,"available":32211795968,"internally_reserved":0,"allocated":21475278848,"data_stored":21475278848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972756,"kb_used_data":20972756,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456028,"statfs":{"total":53687074816,"available":32210972672,"internally_reserved":0,"allocated":21476102144,"data_stored":21476102144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53687074816,"available":32210972672,"internally_reserved":0,"allocated":21476102144,"data_stored":21476102144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53687074816,"available":32211795968,"internally_reserved":0,"allocated":21475278848,"data_stored":21475278848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":53687074816,"available":32211005440,"internally_reserved":0,"allocated":21476069376,"data_stored":21476069376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53687074816,"available":32210972672,"internally_reserved":0,"allocated":21476102144,"data_stored":21476102144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T08:41:25.576 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:41:25.653 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:25.649+0000 7fd047dc1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:25.655 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:25.653+0000 7fd047dc1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:25.763 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:41:25.764 INFO:teuthology.orchestra.run.trial023.stderr:dumped all 2026-02-25T08:41:25.772 INFO:teuthology.orchestra.run.trial023.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2026-02-25T08:41:24.183133+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209715136,"kb_used":83890120,"kb_used_data":83890120,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":125825016,"statfs":{"total":214748299264,"available":128844816384,"internally_reserved":0,"allocated":85903482880,"data_stored":85903482880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.191682+0000","last_change":"2026-02-25T08:41:23.195394+0000","last_active":"2026-02-25T08:41:23.191682+0000","last_peered":"2026-02-25T08:41:23.191682+0000","last_clean":"2026-02-25T08:41:23.191682+0000","last_became_active":"2026-02-25T08:41:21.223518+0000","last_became_peered":"2026-02-25T08:41:21.223518+0000","last_unstale":"2026-02-25T08:41:23.191682+0000","last_undegraded":"2026-02-25T08:41:23.191682+0000","last_fullsized":"2026-02-25T08:41:23.191682+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.190363+0000","last_change":"2026-02-25T08:41:23.192509+0000","last_active":"2026-02-25T08:41:23.190363+0000","last_peered":"2026-02-25T08:41:23.190363+0000","last_clean":"2026-02-25T08:41:23.190363+0000","last_became_active":"2026-02-25T08:41:21.240067+0000","last_became_peered":"2026-02-25T08:41:21.240067+0000","last_unstale":"2026-02-25T08:41:23.190363+0000","last_undegraded":"2026-02-25T08:41:23.190363+0000","last_fullsized":"2026-02-25T08:41:23.190363+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.190372+0000","last_change":"2026-02-25T08:41:23.195131+0000","last_active":"2026-02-25T08:41:23.190372+0000","last_peered":"2026-02-25T08:41:23.190372+0000","last_clean":"2026-02-25T08:41:23.190372+0000","last_became_active":"2026-02-25T08:41:21.226683+0000","last_became_peered":"2026-02-25T08:41:21.226683+0000","last_unstale":"2026-02-25T08:41:23.190372+0000","last_undegraded":"2026-02-25T08:41:23.190372+0000","last_fullsized":"2026-02-25T08:41:23.190372+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.190142+0000","last_change":"2026-02-25T08:41:23.192156+0000","last_active":"2026-02-25T08:41:23.190142+0000","last_peered":"2026-02-25T08:41:23.190142+0000","last_clean":"2026-02-25T08:41:23.190142+0000","last_became_active":"2026-02-25T08:41:21.216122+0000","last_became_peered":"2026-02-25T08:41:21.216122+0000","last_unstale":"2026-02-25T08:41:23.190142+0000","last_undegraded":"2026-02-25T08:41:23.190142+0000","last_fullsized":"2026-02-25T08:41:23.190142+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"13'52","reported_seq":82,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.189947+0000","last_change":"2026-02-25T08:41:21.235650+0000","last_active":"2026-02-25T08:41:23.189947+0000","last_peered":"2026-02-25T08:41:23.189947+0000","last_clean":"2026-02-25T08:41:23.189947+0000","last_became_active":"2026-02-25T08:41:21.226864+0000","last_became_peered":"2026-02-25T08:41:21.226864+0000","last_unstale":"2026-02-25T08:41:23.189947+0000","last_undegraded":"2026-02-25T08:41:23.189947+0000","last_fullsized":"2026-02-25T08:41:23.189947+0000","mapping_epoch":12,"log_start":"13'48","ondisk_log_start":"13'48","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":48,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.188719+0000","last_change":"2026-02-25T08:41:23.191189+0000","last_active":"2026-02-25T08:41:23.188719+0000","last_peered":"2026-02-25T08:41:23.188719+0000","last_clean":"2026-02-25T08:41:23.188719+0000","last_became_active":"2026-02-25T08:41:21.216133+0000","last_became_peered":"2026-02-25T08:41:21.216133+0000","last_unstale":"2026-02-25T08:41:23.188719+0000","last_undegraded":"2026-02-25T08:41:23.188719+0000","last_fullsized":"2026-02-25T08:41:23.188719+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"14'1","reported_seq":18,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.192134+0000","last_change":"2026-02-25T08:41:23.192134+0000","last_active":"2026-02-25T08:41:23.192134+0000","last_peered":"2026-02-25T08:41:23.192134+0000","last_clean":"2026-02-25T08:41:23.192134+0000","last_became_active":"2026-02-25T08:41:21.215058+0000","last_became_peered":"2026-02-25T08:41:21.215058+0000","last_unstale":"2026-02-25T08:41:23.192134+0000","last_undegraded":"2026-02-25T08:41:23.192134+0000","last_fullsized":"2026-02-25T08:41:23.192134+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.188725+0000","last_change":"2026-02-25T08:41:23.191858+0000","last_active":"2026-02-25T08:41:23.188725+0000","last_peered":"2026-02-25T08:41:23.188725+0000","last_clean":"2026-02-25T08:41:23.188725+0000","last_became_active":"2026-02-25T08:41:21.215894+0000","last_became_peered":"2026-02-25T08:41:21.215894+0000","last_unstale":"2026-02-25T08:41:23.188725+0000","last_undegraded":"2026-02-25T08:41:23.188725+0000","last_fullsized":"2026-02-25T08:41:23.188725+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.188713+0000","last_change":"2026-02-25T08:41:23.191742+0000","last_active":"2026-02-25T08:41:23.188713+0000","last_peered":"2026-02-25T08:41:23.188713+0000","last_clean":"2026-02-25T08:41:23.188713+0000","last_became_active":"2026-02-25T08:41:21.218886+0000","last_became_peered":"2026-02-25T08:41:21.218886+0000","last_unstale":"2026-02-25T08:41:23.188713+0000","last_undegraded":"2026-02-25T08:41:23.188713+0000","last_fullsized":"2026-02-25T08:41:23.188713+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32210972672,"internally_reserved":0,"allocated":21476102144,"data_stored":21476102144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96633774080,"internally_reserved":0,"allocated":64427450368,"data_stored":64427450368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":24,"acting":24,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972724,"kb_used_data":20972724,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456060,"statfs":{"total":53687074816,"available":32211005440,"internally_reserved":0,"allocated":21476069376,"data_stored":21476069376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972688,"kb_used_data":20972688,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456096,"statfs":{"total":53687074816,"available":32211042304,"internally_reserved":0,"allocated":21476032512,"data_stored":21476032512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20971952,"kb_used_data":20971952,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456832,"statfs":{"total":53687074816,"available":32211795968,"internally_reserved":0,"allocated":21475278848,"data_stored":21475278848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972756,"kb_used_data":20972756,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456028,"statfs":{"total":53687074816,"available":32210972672,"internally_reserved":0,"allocated":21476102144,"data_stored":21476102144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53687074816,"available":32210972672,"internally_reserved":0,"allocated":21476102144,"data_stored":21476102144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53687074816,"available":32211795968,"internally_reserved":0,"allocated":21475278848,"data_stored":21475278848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":53687074816,"available":32211005440,"internally_reserved":0,"allocated":21476069376,"data_stored":21476069376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53687074816,"available":32210972672,"internally_reserved":0,"allocated":21476102144,"data_stored":21476102144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T08:41:25.774 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-25T08:41:25.774 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2026-02-25T08:41:25.849 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:25.844+0000 7fa34e1cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:25.851 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:25.850+0000 7fa34e1cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:25.961 INFO:teuthology.orchestra.run.trial023.stdout:epoch 15 2026-02-25T08:41:25.962 INFO:teuthology.orchestra.run.trial023.stdout:fsid 542a2d4e-0dbc-4a03-9358-454614077093 2026-02-25T08:41:25.962 INFO:teuthology.orchestra.run.trial023.stdout:created 2026-02-25T08:41:09.790160+0000 2026-02-25T08:41:25.962 INFO:teuthology.orchestra.run.trial023.stdout:modified 2026-02-25T08:41:23.176106+0000 2026-02-25T08:41:25.962 INFO:teuthology.orchestra.run.trial023.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2026-02-25T08:41:25.962 INFO:teuthology.orchestra.run.trial023.stdout:crush_version 5 2026-02-25T08:41:25.962 INFO:teuthology.orchestra.run.trial023.stdout:full_ratio 0.95 2026-02-25T08:41:25.962 INFO:teuthology.orchestra.run.trial023.stdout:backfillfull_ratio 0.9 2026-02-25T08:41:25.962 INFO:teuthology.orchestra.run.trial023.stdout:nearfull_ratio 0.85 2026-02-25T08:41:25.962 INFO:teuthology.orchestra.run.trial023.stdout:require_min_compat_client luminous 2026-02-25T08:41:25.963 INFO:teuthology.orchestra.run.trial023.stdout:min_compat_client jewel 2026-02-25T08:41:25.963 INFO:teuthology.orchestra.run.trial023.stdout:require_osd_release tentacle 2026-02-25T08:41:25.963 INFO:teuthology.orchestra.run.trial023.stdout:stretch_mode_enabled false 2026-02-25T08:41:25.963 INFO:teuthology.orchestra.run.trial023.stdout:allow_crimson=true 2026-02-25T08:41:25.963 INFO:teuthology.orchestra.run.trial023.stdout:pool 1 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 15 flags hashpspool,selfmanaged_snaps,crimson stripe_width 0 application rbd read_balance_score 2.00 2026-02-25T08:41:25.963 INFO:teuthology.orchestra.run.trial023.stdout: removed_snaps_queue [2~1] 2026-02-25T08:41:25.963 INFO:teuthology.orchestra.run.trial023.stdout:pool 2 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 14 flags hashpspool,crimson stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 4.00 2026-02-25T08:41:25.963 INFO:teuthology.orchestra.run.trial023.stdout:max_osd 4 2026-02-25T08:41:25.963 INFO:teuthology.orchestra.run.trial023.stdout:osd.0 up in weight 1 up_from 11 up_thru 12 down_at 0 last_clean_interval [0,0) v2:10.20.193.23:6802/1434923082 v2:10.20.193.23:6803/1434923082 exists,up 55b6cf18-122b-4c40-9723-22d660c2ef92 2026-02-25T08:41:25.963 INFO:teuthology.orchestra.run.trial023.stdout:osd.1 up in weight 1 up_from 11 up_thru 12 down_at 0 last_clean_interval [0,0) v2:10.20.193.23:6801/2628765320 v2:10.20.193.23:6800/2628765320 exists,up cdd52fd3-67fb-4f05-a93e-3956d3a24248 2026-02-25T08:41:25.963 INFO:teuthology.orchestra.run.trial023.stdout:osd.2 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) v2:10.20.193.30:6802/185076637 v2:10.20.193.30:6803/185076637 exists,up d4f198cc-0208-4e4c-b8f1-b7a028c520ca 2026-02-25T08:41:25.963 INFO:teuthology.orchestra.run.trial023.stdout:osd.3 up in weight 1 up_from 11 up_thru 12 down_at 0 last_clean_interval [0,0) v2:10.20.193.30:6804/2656603590 v2:10.20.193.30:6805/2656603590 exists,up edabd0ff-8d49-4327-9a44-5a140c74d128 2026-02-25T08:41:25.971 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 11 up_thru 12 down_at 0 last_clean_interval [0,0) v2:10.20.193.23:6802/1434923082 v2:10.20.193.23:6803/1434923082 exists,up 55b6cf18-122b-4c40-9723-22d660c2ef92', 'osd.1 up in weight 1 up_from 11 up_thru 12 down_at 0 last_clean_interval [0,0) v2:10.20.193.23:6801/2628765320 v2:10.20.193.23:6800/2628765320 exists,up cdd52fd3-67fb-4f05-a93e-3956d3a24248', 'osd.2 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) v2:10.20.193.30:6802/185076637 v2:10.20.193.30:6803/185076637 exists,up d4f198cc-0208-4e4c-b8f1-b7a028c520ca', 'osd.3 up in weight 1 up_from 11 up_thru 12 down_at 0 last_clean_interval [0,0) v2:10.20.193.30:6804/2656603590 v2:10.20.193.30:6805/2656603590 exists,up edabd0ff-8d49-4327-9a44-5a140c74d128'] 2026-02-25T08:41:25.972 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2026-02-25T08:41:26.046 INFO:teuthology.orchestra.run.trial023.stdout:{ 2026-02-25T08:41:26.046 INFO:teuthology.orchestra.run.trial023.stdout: "name": "mon.a", 2026-02-25T08:41:26.046 INFO:teuthology.orchestra.run.trial023.stdout: "cluster": "ceph", 2026-02-25T08:41:26.046 INFO:teuthology.orchestra.run.trial023.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2026-02-25T08:41:26.046 INFO:teuthology.orchestra.run.trial023.stdout: "admin_socket_mode": "", 2026-02-25T08:41:26.046 INFO:teuthology.orchestra.run.trial023.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2026-02-25T08:41:26.046 INFO:teuthology.orchestra.run.trial023.stdout: "auth_client_required": "cephx, none", 2026-02-25T08:41:26.047 INFO:teuthology.orchestra.run.trial023.stdout: "auth_cluster_required": "cephx", 2026-02-25T08:41:26.047 INFO:teuthology.orchestra.run.trial023.stdout: "auth_debug": "true", 2026-02-25T08:41:26.047 INFO:teuthology.orchestra.run.trial023.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2026-02-25T08:41:26.047 INFO:teuthology.orchestra.run.trial023.stdout: "auth_mon_ticket_ttl": "660.000000", 2026-02-25T08:41:26.047 INFO:teuthology.orchestra.run.trial023.stdout: "auth_service_required": "cephx", 2026-02-25T08:41:26.047 INFO:teuthology.orchestra.run.trial023.stdout: "auth_service_ticket_ttl": "240.000000", 2026-02-25T08:41:26.047 INFO:teuthology.orchestra.run.trial023.stdout: "auth_supported": "cephx", 2026-02-25T08:41:26.047 INFO:teuthology.orchestra.run.trial023.stdout: "bdev_aio": "true", 2026-02-25T08:41:26.047 INFO:teuthology.orchestra.run.trial023.stdout: "bdev_aio_max_queue_depth": "1024", 2026-02-25T08:41:26.047 INFO:teuthology.orchestra.run.trial023.stdout: "bdev_aio_poll_ms": "250", 2026-02-25T08:41:26.047 INFO:teuthology.orchestra.run.trial023.stdout: "bdev_aio_reap_max": "16", 2026-02-25T08:41:26.048 INFO:teuthology.orchestra.run.trial023.stdout: "bdev_aio_submit_retry_initial_delay_us": "125", 2026-02-25T08:41:26.048 INFO:teuthology.orchestra.run.trial023.stdout: "bdev_aio_submit_retry_max": "16", 2026-02-25T08:41:26.048 INFO:teuthology.orchestra.run.trial023.stdout: "bdev_async_discard": "false", 2026-02-25T08:41:26.048 INFO:teuthology.orchestra.run.trial023.stdout: "bdev_async_discard_max_pending": "1000000", 2026-02-25T08:41:26.048 INFO:teuthology.orchestra.run.trial023.stdout: "bdev_async_discard_threads": "0", 2026-02-25T08:41:26.048 INFO:teuthology.orchestra.run.trial023.stdout: "bdev_block_size": "4096", 2026-02-25T08:41:26.048 INFO:teuthology.orchestra.run.trial023.stdout: "bdev_debug_aio": "false", 2026-02-25T08:41:26.048 INFO:teuthology.orchestra.run.trial023.stdout: "bdev_debug_aio_log_age": "5.000000", 2026-02-25T08:41:26.048 INFO:teuthology.orchestra.run.trial023.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2026-02-25T08:41:26.048 INFO:teuthology.orchestra.run.trial023.stdout: "bdev_debug_discard_sleep": "0", 2026-02-25T08:41:26.048 INFO:teuthology.orchestra.run.trial023.stdout: "bdev_debug_inflight_ios": "false", 2026-02-25T08:41:26.048 INFO:teuthology.orchestra.run.trial023.stdout: "bdev_discard_max_bytes": "10737418240", 2026-02-25T08:41:26.049 INFO:teuthology.orchestra.run.trial023.stdout: "bdev_enable_discard": "false", 2026-02-25T08:41:26.049 INFO:teuthology.orchestra.run.trial023.stdout: "bdev_flock_retry": "3", 2026-02-25T08:41:26.049 INFO:teuthology.orchestra.run.trial023.stdout: "bdev_flock_retry_interval": "0.100000", 2026-02-25T08:41:26.049 INFO:teuthology.orchestra.run.trial023.stdout: "bdev_inject_crash": "0", 2026-02-25T08:41:26.049 INFO:teuthology.orchestra.run.trial023.stdout: "bdev_inject_crash_flush_delay": "2", 2026-02-25T08:41:26.049 INFO:teuthology.orchestra.run.trial023.stdout: "bdev_ioring": "false", 2026-02-25T08:41:26.049 INFO:teuthology.orchestra.run.trial023.stdout: "bdev_ioring_hipri": "false", 2026-02-25T08:41:26.049 INFO:teuthology.orchestra.run.trial023.stdout: "bdev_ioring_sqthread_poll": "false", 2026-02-25T08:41:26.049 INFO:teuthology.orchestra.run.trial023.stdout: "bdev_max_discard_length": "2147483648", 2026-02-25T08:41:26.049 INFO:teuthology.orchestra.run.trial023.stdout: "bdev_nvme_unbind_from_kernel": "false", 2026-02-25T08:41:26.049 INFO:teuthology.orchestra.run.trial023.stdout: "bdev_read_buffer_alignment": "4096", 2026-02-25T08:41:26.049 INFO:teuthology.orchestra.run.trial023.stdout: "bdev_read_preallocated_huge_buffers": "", 2026-02-25T08:41:26.050 INFO:teuthology.orchestra.run.trial023.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2026-02-25T08:41:26.050 INFO:teuthology.orchestra.run.trial023.stdout: "bdev_stalled_read_warn_threshold": "1", 2026-02-25T08:41:26.050 INFO:teuthology.orchestra.run.trial023.stdout: "bdev_type": "", 2026-02-25T08:41:26.050 INFO:teuthology.orchestra.run.trial023.stdout: "bluefs_alloc_size": "1048576", 2026-02-25T08:41:26.050 INFO:teuthology.orchestra.run.trial023.stdout: "bluefs_allocator": "hybrid", 2026-02-25T08:41:26.050 INFO:teuthology.orchestra.run.trial023.stdout: "bluefs_buffered_io": "true", 2026-02-25T08:41:26.050 INFO:teuthology.orchestra.run.trial023.stdout: "bluefs_check_for_zeros": "false", 2026-02-25T08:41:26.050 INFO:teuthology.orchestra.run.trial023.stdout: "bluefs_check_volume_selector_often": "false", 2026-02-25T08:41:26.050 INFO:teuthology.orchestra.run.trial023.stdout: "bluefs_check_volume_selector_on_umount": "false", 2026-02-25T08:41:26.050 INFO:teuthology.orchestra.run.trial023.stdout: "bluefs_compact_log_sync": "false", 2026-02-25T08:41:26.050 INFO:teuthology.orchestra.run.trial023.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2026-02-25T08:41:26.050 INFO:teuthology.orchestra.run.trial023.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2026-02-25T08:41:26.050 INFO:teuthology.orchestra.run.trial023.stdout: "bluefs_log_compact_min_size": "16777216", 2026-02-25T08:41:26.051 INFO:teuthology.orchestra.run.trial023.stdout: "bluefs_log_replay_check_allocations": "true", 2026-02-25T08:41:26.051 INFO:teuthology.orchestra.run.trial023.stdout: "bluefs_max_log_runway": "4194304", 2026-02-25T08:41:26.051 INFO:teuthology.orchestra.run.trial023.stdout: "bluefs_max_prefetch": "1048576", 2026-02-25T08:41:26.051 INFO:teuthology.orchestra.run.trial023.stdout: "bluefs_min_flush_size": "524288", 2026-02-25T08:41:26.051 INFO:teuthology.orchestra.run.trial023.stdout: "bluefs_min_log_runway": "1048576", 2026-02-25T08:41:26.051 INFO:teuthology.orchestra.run.trial023.stdout: "bluefs_replay_recovery": "false", 2026-02-25T08:41:26.051 INFO:teuthology.orchestra.run.trial023.stdout: "bluefs_replay_recovery_disable_compact": "false", 2026-02-25T08:41:26.051 INFO:teuthology.orchestra.run.trial023.stdout: "bluefs_shared_alloc_size": "65536", 2026-02-25T08:41:26.051 INFO:teuthology.orchestra.run.trial023.stdout: "bluefs_sync_write": "false", 2026-02-25T08:41:26.051 INFO:teuthology.orchestra.run.trial023.stdout: "bluefs_wal_envelope_mode": "true", 2026-02-25T08:41:26.051 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2026-02-25T08:41:26.051 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2026-02-25T08:41:26.052 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2026-02-25T08:41:26.052 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_allocation_from_file": "true", 2026-02-25T08:41:26.052 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_allocator": "hybrid", 2026-02-25T08:41:26.052 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_allocator_lookup_policy": "auto", 2026-02-25T08:41:26.052 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_async_db_compaction": "true", 2026-02-25T08:41:26.052 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2026-02-25T08:41:26.052 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2026-02-25T08:41:26.052 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2026-02-25T08:41:26.052 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2026-02-25T08:41:26.052 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_bdev_label_multi": "true", 2026-02-25T08:41:26.052 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2026-02-25T08:41:26.052 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_bdev_label_require_all": "true", 2026-02-25T08:41:26.053 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2026-02-25T08:41:26.053 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_bitmapallocator_span_size": "1024", 2026-02-25T08:41:26.053 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_blobid_prealloc": "10240", 2026-02-25T08:41:26.053 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_block_create": "true", 2026-02-25T08:41:26.053 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_block_db_create": "false", 2026-02-25T08:41:26.053 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_block_db_path": "", 2026-02-25T08:41:26.053 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_block_db_size": "0", 2026-02-25T08:41:26.053 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_block_path": "", 2026-02-25T08:41:26.053 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_block_preallocate_file": "false", 2026-02-25T08:41:26.053 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_block_size": "107374182400", 2026-02-25T08:41:26.053 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_block_wal_create": "false", 2026-02-25T08:41:26.053 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_block_wal_path": "", 2026-02-25T08:41:26.053 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_block_wal_size": "100663296", 2026-02-25T08:41:26.054 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_bluefs": "true", 2026-02-25T08:41:26.054 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2026-02-25T08:41:26.054 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_bluefs_env_mirror": "false", 2026-02-25T08:41:26.054 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_bluefs_max_free": "10737418240", 2026-02-25T08:41:26.054 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2026-02-25T08:41:26.054 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2026-02-25T08:41:26.054 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2026-02-25T08:41:26.054 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2026-02-25T08:41:26.054 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2026-02-25T08:41:26.054 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2026-02-25T08:41:26.054 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_cache_autotune": "true", 2026-02-25T08:41:26.054 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_cache_autotune_interval": "5.000000", 2026-02-25T08:41:26.055 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2026-02-25T08:41:26.055 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_cache_kv_ratio": "0.450000", 2026-02-25T08:41:26.055 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_cache_meta_ratio": "0.450000", 2026-02-25T08:41:26.055 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_cache_size": "0", 2026-02-25T08:41:26.055 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_cache_size_hdd": "1073741824", 2026-02-25T08:41:26.055 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_cache_size_ssd": "3221225472", 2026-02-25T08:41:26.055 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_cache_trim_interval": "0.050000", 2026-02-25T08:41:26.055 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2026-02-25T08:41:26.055 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_cache_type": "2q", 2026-02-25T08:41:26.055 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2026-02-25T08:41:26.055 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_clone_cow": "true", 2026-02-25T08:41:26.055 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_compression_algorithm": "snappy", 2026-02-25T08:41:26.055 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_compression_max_blob_size": "0", 2026-02-25T08:41:26.056 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2026-02-25T08:41:26.056 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2026-02-25T08:41:26.056 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_compression_min_blob_size": "0", 2026-02-25T08:41:26.056 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2026-02-25T08:41:26.056 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2026-02-25T08:41:26.056 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_compression_mode": "none", 2026-02-25T08:41:26.056 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_compression_required_ratio": "0.875000", 2026-02-25T08:41:26.056 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_csum_type": "crc32c", 2026-02-25T08:41:26.056 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2026-02-25T08:41:26.056 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_debug_enforce_settings": "default", 2026-02-25T08:41:26.056 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_debug_extent_map_encode_check": "false", 2026-02-25T08:41:26.056 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_debug_freelist": "false", 2026-02-25T08:41:26.056 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_debug_fsck_abort": "false", 2026-02-25T08:41:26.057 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2026-02-25T08:41:26.057 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2026-02-25T08:41:26.057 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_debug_inject_read_err": "false", 2026-02-25T08:41:26.057 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_debug_legacy_omap": "false", 2026-02-25T08:41:26.057 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_debug_no_reuse_blocks": "false", 2026-02-25T08:41:26.057 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_debug_omit_block_device_write": "false", 2026-02-25T08:41:26.057 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_debug_omit_kv_commit": "false", 2026-02-25T08:41:26.057 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_debug_onode_segmentation_random": "false", 2026-02-25T08:41:26.057 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2026-02-25T08:41:26.057 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_debug_prefragment_max": "1048576", 2026-02-25T08:41:26.057 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_debug_random_read_err": "0.000000", 2026-02-25T08:41:26.057 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2026-02-25T08:41:26.058 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_debug_small_allocations": "0", 2026-02-25T08:41:26.058 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2026-02-25T08:41:26.058 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_default_buffered_read": "true", 2026-02-25T08:41:26.058 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_default_buffered_write": "false", 2026-02-25T08:41:26.058 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_deferred_batch_ops": "0", 2026-02-25T08:41:26.058 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2026-02-25T08:41:26.058 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2026-02-25T08:41:26.058 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_discard_on_mkfs": "true", 2026-02-25T08:41:26.058 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_elastic_shared_blobs": "true", 2026-02-25T08:41:26.058 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2026-02-25T08:41:26.058 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_extent_map_shard_max_size": "1200", 2026-02-25T08:41:26.058 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_extent_map_shard_min_size": "150", 2026-02-25T08:41:26.058 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_extent_map_shard_target_size": "500", 2026-02-25T08:41:26.059 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2026-02-25T08:41:26.059 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_fail_eio": "false", 2026-02-25T08:41:26.059 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_fragmentation_check_period": "3600", 2026-02-25T08:41:26.059 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_freelist_blocks_per_key": "128", 2026-02-25T08:41:26.059 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2026-02-25T08:41:26.059 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2026-02-25T08:41:26.059 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2026-02-25T08:41:26.059 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_fsck_on_mkfs": "true", 2026-02-25T08:41:26.059 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2026-02-25T08:41:26.059 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_fsck_on_mount": "false", 2026-02-25T08:41:26.059 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_fsck_on_mount_deep": "false", 2026-02-25T08:41:26.059 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_fsck_on_umount": "false", 2026-02-25T08:41:26.059 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_fsck_on_umount_deep": "false", 2026-02-25T08:41:26.060 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2026-02-25T08:41:26.060 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_fsck_quick_fix_threads": "2", 2026-02-25T08:41:26.060 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2026-02-25T08:41:26.060 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2026-02-25T08:41:26.060 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_gc_enable_blob_threshold": "0", 2026-02-25T08:41:26.060 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_gc_enable_total_threshold": "0", 2026-02-25T08:41:26.060 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2026-02-25T08:41:26.060 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_ignore_data_csum": "false", 2026-02-25T08:41:26.060 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2026-02-25T08:41:26.060 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_kvbackend": "rocksdb", 2026-02-25T08:41:26.060 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_log_collection_list_age": "60.000000", 2026-02-25T08:41:26.060 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2026-02-25T08:41:26.061 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_log_op_age": "5.000000", 2026-02-25T08:41:26.061 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_max_alloc_size": "0", 2026-02-25T08:41:26.061 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_max_blob_size": "0", 2026-02-25T08:41:26.061 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_max_blob_size_hdd": "65536", 2026-02-25T08:41:26.061 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_max_blob_size_ssd": "65536", 2026-02-25T08:41:26.061 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_max_defer_interval": "3.000000", 2026-02-25T08:41:26.061 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_max_deferred_txc": "32", 2026-02-25T08:41:26.061 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_min_alloc_size": "0", 2026-02-25T08:41:26.061 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_min_alloc_size_hdd": "4096", 2026-02-25T08:41:26.061 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_min_alloc_size_ssd": "4096", 2026-02-25T08:41:26.061 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_nid_prealloc": "1024", 2026-02-25T08:41:26.061 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_onode_segment_size": "0", 2026-02-25T08:41:26.061 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_prefer_deferred_size": "0", 2026-02-25T08:41:26.062 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2026-02-25T08:41:26.062 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2026-02-25T08:41:26.062 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_qfsck_on_mount": "true", 2026-02-25T08:41:26.062 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_recompression_min_gain": "1.200000", 2026-02-25T08:41:26.062 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_retry_disk_reads": "3", 2026-02-25T08:41:26.062 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_rocksdb_cf": "true", 2026-02-25T08:41:26.062 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2026-02-25T08:41:26.062 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2026-02-25T08:41:26.062 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_rocksdb_options_annex": "", 2026-02-25T08:41:26.062 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2026-02-25T08:41:26.062 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_slow_ops_warn_threshold": "1", 2026-02-25T08:41:26.062 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_spdk_coremask": "0x1", 2026-02-25T08:41:26.063 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_spdk_io_sleep": "5", 2026-02-25T08:41:26.063 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_spdk_max_io_completion": "0", 2026-02-25T08:41:26.063 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_spdk_mem": "512", 2026-02-25T08:41:26.063 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_sync_submit_transaction": "false", 2026-02-25T08:41:26.063 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_throttle_bytes": "67108864", 2026-02-25T08:41:26.063 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_throttle_cost_per_io": "0", 2026-02-25T08:41:26.063 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2026-02-25T08:41:26.063 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2026-02-25T08:41:26.063 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2026-02-25T08:41:26.063 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_throttle_trace_rate": "0.000000", 2026-02-25T08:41:26.063 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_tracing": "false", 2026-02-25T08:41:26.063 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2026-02-25T08:41:26.063 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2026-02-25T08:41:26.064 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_volume_selection_reserved": "0", 2026-02-25T08:41:26.064 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2026-02-25T08:41:26.064 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2026-02-25T08:41:26.064 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_warn_on_free_fragmentation": "0.800000", 2026-02-25T08:41:26.064 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_warn_on_legacy_statfs": "true", 2026-02-25T08:41:26.064 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2026-02-25T08:41:26.064 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2026-02-25T08:41:26.064 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2026-02-25T08:41:26.064 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_write_v2": "false", 2026-02-25T08:41:26.064 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_write_v2_random": "false", 2026-02-25T08:41:26.064 INFO:teuthology.orchestra.run.trial023.stdout: "bluestore_zero_block_detection": "false", 2026-02-25T08:41:26.064 INFO:teuthology.orchestra.run.trial023.stdout: "breakpad": "false", 2026-02-25T08:41:26.064 INFO:teuthology.orchestra.run.trial023.stdout: "ceph_assert_supresssions": "", 2026-02-25T08:41:26.065 INFO:teuthology.orchestra.run.trial023.stdout: "cephadm_path": "/usr/sbin/cephadm", 2026-02-25T08:41:26.065 INFO:teuthology.orchestra.run.trial023.stdout: "cephfs_mirror_action_update_interval": "2", 2026-02-25T08:41:26.065 INFO:teuthology.orchestra.run.trial023.stdout: "cephfs_mirror_directory_scan_interval": "10", 2026-02-25T08:41:26.065 INFO:teuthology.orchestra.run.trial023.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2026-02-25T08:41:26.065 INFO:teuthology.orchestra.run.trial023.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2026-02-25T08:41:26.065 INFO:teuthology.orchestra.run.trial023.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2026-02-25T08:41:26.065 INFO:teuthology.orchestra.run.trial023.stdout: "cephfs_mirror_mount_timeout": "10", 2026-02-25T08:41:26.065 INFO:teuthology.orchestra.run.trial023.stdout: "cephfs_mirror_perf_stats_prio": "5", 2026-02-25T08:41:26.065 INFO:teuthology.orchestra.run.trial023.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2026-02-25T08:41:26.065 INFO:teuthology.orchestra.run.trial023.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2026-02-25T08:41:26.065 INFO:teuthology.orchestra.run.trial023.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2026-02-25T08:41:26.065 INFO:teuthology.orchestra.run.trial023.stdout: "cephsqlite_blocklist_dead_locker": "true", 2026-02-25T08:41:26.065 INFO:teuthology.orchestra.run.trial023.stdout: "cephsqlite_lock_renewal_interval": "2000", 2026-02-25T08:41:26.066 INFO:teuthology.orchestra.run.trial023.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2026-02-25T08:41:26.066 INFO:teuthology.orchestra.run.trial023.stdout: "cephx_cluster_require_signatures": "false", 2026-02-25T08:41:26.066 INFO:teuthology.orchestra.run.trial023.stdout: "cephx_cluster_require_version": "2", 2026-02-25T08:41:26.066 INFO:teuthology.orchestra.run.trial023.stdout: "cephx_require_signatures": "false", 2026-02-25T08:41:26.066 INFO:teuthology.orchestra.run.trial023.stdout: "cephx_require_version": "2", 2026-02-25T08:41:26.066 INFO:teuthology.orchestra.run.trial023.stdout: "cephx_service_require_signatures": "false", 2026-02-25T08:41:26.066 INFO:teuthology.orchestra.run.trial023.stdout: "cephx_service_require_version": "2", 2026-02-25T08:41:26.066 INFO:teuthology.orchestra.run.trial023.stdout: "cephx_sign_messages": "true", 2026-02-25T08:41:26.066 INFO:teuthology.orchestra.run.trial023.stdout: "chdir": "", 2026-02-25T08:41:26.066 INFO:teuthology.orchestra.run.trial023.stdout: "client_acl_type": "", 2026-02-25T08:41:26.066 INFO:teuthology.orchestra.run.trial023.stdout: "client_asio_thread_count": "2", 2026-02-25T08:41:26.066 INFO:teuthology.orchestra.run.trial023.stdout: "client_cache_mid": "0.750000", 2026-02-25T08:41:26.067 INFO:teuthology.orchestra.run.trial023.stdout: "client_cache_size": "16384", 2026-02-25T08:41:26.067 INFO:teuthology.orchestra.run.trial023.stdout: "client_caps_release_delay": "5", 2026-02-25T08:41:26.067 INFO:teuthology.orchestra.run.trial023.stdout: "client_check_pool_perm": "true", 2026-02-25T08:41:26.067 INFO:teuthology.orchestra.run.trial023.stdout: "client_collect_and_send_global_metrics": "false", 2026-02-25T08:41:26.067 INFO:teuthology.orchestra.run.trial023.stdout: "client_debug_force_sync_read": "false", 2026-02-25T08:41:26.067 INFO:teuthology.orchestra.run.trial023.stdout: "client_debug_getattr_caps": "false", 2026-02-25T08:41:26.067 INFO:teuthology.orchestra.run.trial023.stdout: "client_debug_inject_features": "", 2026-02-25T08:41:26.067 INFO:teuthology.orchestra.run.trial023.stdout: "client_debug_inject_tick_delay": "0", 2026-02-25T08:41:26.067 INFO:teuthology.orchestra.run.trial023.stdout: "client_die_on_failed_dentry_invalidate": "true", 2026-02-25T08:41:26.067 INFO:teuthology.orchestra.run.trial023.stdout: "client_die_on_failed_remount": "false", 2026-02-25T08:41:26.067 INFO:teuthology.orchestra.run.trial023.stdout: "client_dirsize_rbytes": "true", 2026-02-25T08:41:26.067 INFO:teuthology.orchestra.run.trial023.stdout: "client_file_blockdiff_max_concurrent_object_scans": "16", 2026-02-25T08:41:26.067 INFO:teuthology.orchestra.run.trial023.stdout: "client_force_lazyio": "false", 2026-02-25T08:41:26.068 INFO:teuthology.orchestra.run.trial023.stdout: "client_fs": "", 2026-02-25T08:41:26.068 INFO:teuthology.orchestra.run.trial023.stdout: "client_fscrypt_as": "true", 2026-02-25T08:41:26.068 INFO:teuthology.orchestra.run.trial023.stdout: "client_fscrypt_dummy_encryption": "false", 2026-02-25T08:41:26.068 INFO:teuthology.orchestra.run.trial023.stdout: "client_inject_fixed_oldest_tid": "false", 2026-02-25T08:41:26.068 INFO:teuthology.orchestra.run.trial023.stdout: "client_inject_release_failure": "false", 2026-02-25T08:41:26.068 INFO:teuthology.orchestra.run.trial023.stdout: "client_inject_write_delay_secs": "0", 2026-02-25T08:41:26.068 INFO:teuthology.orchestra.run.trial023.stdout: "client_max_inline_size": "4096", 2026-02-25T08:41:26.068 INFO:teuthology.orchestra.run.trial023.stdout: "client_max_retries_on_remount_failure": "5", 2026-02-25T08:41:26.068 INFO:teuthology.orchestra.run.trial023.stdout: "client_mds_namespace": "", 2026-02-25T08:41:26.068 INFO:teuthology.orchestra.run.trial023.stdout: "client_metadata": "", 2026-02-25T08:41:26.068 INFO:teuthology.orchestra.run.trial023.stdout: "client_mount_gid": "-1", 2026-02-25T08:41:26.068 INFO:teuthology.orchestra.run.trial023.stdout: "client_mount_timeout": "300", 2026-02-25T08:41:26.068 INFO:teuthology.orchestra.run.trial023.stdout: "client_mount_uid": "-1", 2026-02-25T08:41:26.069 INFO:teuthology.orchestra.run.trial023.stdout: "client_mountpoint": "/", 2026-02-25T08:41:26.069 INFO:teuthology.orchestra.run.trial023.stdout: "client_notify_timeout": "10", 2026-02-25T08:41:26.069 INFO:teuthology.orchestra.run.trial023.stdout: "client_oc": "true", 2026-02-25T08:41:26.069 INFO:teuthology.orchestra.run.trial023.stdout: "client_oc_max_dirty": "104857600", 2026-02-25T08:41:26.069 INFO:teuthology.orchestra.run.trial023.stdout: "client_oc_max_dirty_age": "5.000000", 2026-02-25T08:41:26.069 INFO:teuthology.orchestra.run.trial023.stdout: "client_oc_max_objects": "1000", 2026-02-25T08:41:26.069 INFO:teuthology.orchestra.run.trial023.stdout: "client_oc_size": "209715200", 2026-02-25T08:41:26.069 INFO:teuthology.orchestra.run.trial023.stdout: "client_oc_target_dirty": "8388608", 2026-02-25T08:41:26.069 INFO:teuthology.orchestra.run.trial023.stdout: "client_permissions": "true", 2026-02-25T08:41:26.069 INFO:teuthology.orchestra.run.trial023.stdout: "client_quota": "true", 2026-02-25T08:41:26.069 INFO:teuthology.orchestra.run.trial023.stdout: "client_quota_df": "true", 2026-02-25T08:41:26.069 INFO:teuthology.orchestra.run.trial023.stdout: "client_readahead_max_bytes": "0", 2026-02-25T08:41:26.069 INFO:teuthology.orchestra.run.trial023.stdout: "client_readahead_max_periods": "4", 2026-02-25T08:41:26.070 INFO:teuthology.orchestra.run.trial023.stdout: "client_readahead_min": "131072", 2026-02-25T08:41:26.070 INFO:teuthology.orchestra.run.trial023.stdout: "client_reconnect_stale": "false", 2026-02-25T08:41:26.070 INFO:teuthology.orchestra.run.trial023.stdout: "client_respect_subvolume_snapshot_visibility": "false", 2026-02-25T08:41:26.070 INFO:teuthology.orchestra.run.trial023.stdout: "client_shutdown_timeout": "30", 2026-02-25T08:41:26.070 INFO:teuthology.orchestra.run.trial023.stdout: "client_snapdir": ".snap", 2026-02-25T08:41:26.070 INFO:teuthology.orchestra.run.trial023.stdout: "client_tick_interval": "1", 2026-02-25T08:41:26.070 INFO:teuthology.orchestra.run.trial023.stdout: "client_trace": "", 2026-02-25T08:41:26.070 INFO:teuthology.orchestra.run.trial023.stdout: "client_try_dentry_invalidate": "false", 2026-02-25T08:41:26.070 INFO:teuthology.orchestra.run.trial023.stdout: "client_use_faked_inos": "false", 2026-02-25T08:41:26.070 INFO:teuthology.orchestra.run.trial023.stdout: "client_use_random_mds": "false", 2026-02-25T08:41:26.070 INFO:teuthology.orchestra.run.trial023.stdout: "clog_to_graylog": "false", 2026-02-25T08:41:26.070 INFO:teuthology.orchestra.run.trial023.stdout: "clog_to_graylog_host": "127.0.0.1", 2026-02-25T08:41:26.070 INFO:teuthology.orchestra.run.trial023.stdout: "clog_to_graylog_port": "12201", 2026-02-25T08:41:26.071 INFO:teuthology.orchestra.run.trial023.stdout: "clog_to_monitors": "default=true", 2026-02-25T08:41:26.071 INFO:teuthology.orchestra.run.trial023.stdout: "clog_to_syslog": "false", 2026-02-25T08:41:26.071 INFO:teuthology.orchestra.run.trial023.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2026-02-25T08:41:26.071 INFO:teuthology.orchestra.run.trial023.stdout: "clog_to_syslog_level": "info", 2026-02-25T08:41:26.071 INFO:teuthology.orchestra.run.trial023.stdout: "cluster_addr": "-", 2026-02-25T08:41:26.071 INFO:teuthology.orchestra.run.trial023.stdout: "cluster_network": "", 2026-02-25T08:41:26.071 INFO:teuthology.orchestra.run.trial023.stdout: "cluster_network_interface": "", 2026-02-25T08:41:26.071 INFO:teuthology.orchestra.run.trial023.stdout: "compressor_zlib_isal": "false", 2026-02-25T08:41:26.071 INFO:teuthology.orchestra.run.trial023.stdout: "compressor_zlib_level": "5", 2026-02-25T08:41:26.071 INFO:teuthology.orchestra.run.trial023.stdout: "compressor_zlib_winsize": "-15", 2026-02-25T08:41:26.071 INFO:teuthology.orchestra.run.trial023.stdout: "compressor_zstd_level": "1", 2026-02-25T08:41:26.071 INFO:teuthology.orchestra.run.trial023.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2026-02-25T08:41:26.071 INFO:teuthology.orchestra.run.trial023.stdout: "crash_dir": "/var/lib/ceph/crash", 2026-02-25T08:41:26.071 INFO:teuthology.orchestra.run.trial023.stdout: "crimson_allow_pg_split": "true", 2026-02-25T08:41:26.072 INFO:teuthology.orchestra.run.trial023.stdout: "crimson_bluestore_cpu_set": "", 2026-02-25T08:41:26.072 INFO:teuthology.orchestra.run.trial023.stdout: "crimson_bluestore_num_threads": "6", 2026-02-25T08:41:26.072 INFO:teuthology.orchestra.run.trial023.stdout: "crimson_cpu_num": "0", 2026-02-25T08:41:26.072 INFO:teuthology.orchestra.run.trial023.stdout: "crimson_cpu_set": "", 2026-02-25T08:41:26.072 INFO:teuthology.orchestra.run.trial023.stdout: "crimson_osd_obc_lru_size": "512", 2026-02-25T08:41:26.072 INFO:teuthology.orchestra.run.trial023.stdout: "crimson_osd_scheduler_concurrency": "0", 2026-02-25T08:41:26.072 INFO:teuthology.orchestra.run.trial023.stdout: "crimson_osd_stat_interval": "0", 2026-02-25T08:41:26.072 INFO:teuthology.orchestra.run.trial023.stdout: "crimson_poll_mode": "false", 2026-02-25T08:41:26.072 INFO:teuthology.orchestra.run.trial023.stdout: "crimson_reactor_idle_poll_time_us": "200", 2026-02-25T08:41:26.072 INFO:teuthology.orchestra.run.trial023.stdout: "crimson_reactor_io_latency_goal_ms": "0.000000", 2026-02-25T08:41:26.072 INFO:teuthology.orchestra.run.trial023.stdout: "crimson_reactor_task_quota_ms": "0.500000", 2026-02-25T08:41:26.072 INFO:teuthology.orchestra.run.trial023.stdout: "crush_location": "", 2026-02-25T08:41:26.073 INFO:teuthology.orchestra.run.trial023.stdout: "crush_location_hook": "", 2026-02-25T08:41:26.073 INFO:teuthology.orchestra.run.trial023.stdout: "crush_location_hook_timeout": "10", 2026-02-25T08:41:26.073 INFO:teuthology.orchestra.run.trial023.stdout: "d4n_writecache_enabled": "false", 2026-02-25T08:41:26.073 INFO:teuthology.orchestra.run.trial023.stdout: "daemonize": "false", 2026-02-25T08:41:26.073 INFO:teuthology.orchestra.run.trial023.stdout: "daos_pool": "tank", 2026-02-25T08:41:26.073 INFO:teuthology.orchestra.run.trial023.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2026-02-25T08:41:26.073 INFO:teuthology.orchestra.run.trial023.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2026-02-25T08:41:26.073 INFO:teuthology.orchestra.run.trial023.stdout: "dbstore_db_name_prefix": "dbstore", 2026-02-25T08:41:26.073 INFO:teuthology.orchestra.run.trial023.stdout: "debug_alienstore": "0/5", 2026-02-25T08:41:26.073 INFO:teuthology.orchestra.run.trial023.stdout: "debug_allow_any_pool_priority": "false", 2026-02-25T08:41:26.073 INFO:teuthology.orchestra.run.trial023.stdout: "debug_asok": "1/5", 2026-02-25T08:41:26.073 INFO:teuthology.orchestra.run.trial023.stdout: "debug_asok_assert_abort": "false", 2026-02-25T08:41:26.073 INFO:teuthology.orchestra.run.trial023.stdout: "debug_asserts_on_shutdown": "true", 2026-02-25T08:41:26.074 INFO:teuthology.orchestra.run.trial023.stdout: "debug_auth": "20/20", 2026-02-25T08:41:26.074 INFO:teuthology.orchestra.run.trial023.stdout: "debug_bdev": "1/3", 2026-02-25T08:41:26.074 INFO:teuthology.orchestra.run.trial023.stdout: "debug_bluefs": "1/5", 2026-02-25T08:41:26.074 INFO:teuthology.orchestra.run.trial023.stdout: "debug_bluestore": "1/5", 2026-02-25T08:41:26.074 INFO:teuthology.orchestra.run.trial023.stdout: "debug_bluestore_compression": "1/5", 2026-02-25T08:41:26.074 INFO:teuthology.orchestra.run.trial023.stdout: "debug_buffer": "0/1", 2026-02-25T08:41:26.074 INFO:teuthology.orchestra.run.trial023.stdout: "debug_ceph_dedup": "0/5", 2026-02-25T08:41:26.074 INFO:teuthology.orchestra.run.trial023.stdout: "debug_ceph_exporter": "1/5", 2026-02-25T08:41:26.074 INFO:teuthology.orchestra.run.trial023.stdout: "debug_cephfs_mirror": "0/5", 2026-02-25T08:41:26.074 INFO:teuthology.orchestra.run.trial023.stdout: "debug_cephsqlite": "0/5", 2026-02-25T08:41:26.074 INFO:teuthology.orchestra.run.trial023.stdout: "debug_client": "0/5", 2026-02-25T08:41:26.074 INFO:teuthology.orchestra.run.trial023.stdout: "debug_compressor": "1/5", 2026-02-25T08:41:26.074 INFO:teuthology.orchestra.run.trial023.stdout: "debug_context": "0/1", 2026-02-25T08:41:26.074 INFO:teuthology.orchestra.run.trial023.stdout: "debug_crimson_interrupt": "0/5", 2026-02-25T08:41:26.075 INFO:teuthology.orchestra.run.trial023.stdout: "debug_crush": "1/1", 2026-02-25T08:41:26.075 INFO:teuthology.orchestra.run.trial023.stdout: "debug_crypto": "1/5", 2026-02-25T08:41:26.075 INFO:teuthology.orchestra.run.trial023.stdout: "debug_cyanstore": "0/5", 2026-02-25T08:41:26.075 INFO:teuthology.orchestra.run.trial023.stdout: "debug_deliberately_leak_memory": "false", 2026-02-25T08:41:26.075 INFO:teuthology.orchestra.run.trial023.stdout: "debug_disable_randomized_ping": "false", 2026-02-25T08:41:26.075 INFO:teuthology.orchestra.run.trial023.stdout: "debug_dpdk": "1/5", 2026-02-25T08:41:26.075 INFO:teuthology.orchestra.run.trial023.stdout: "debug_eventtrace": "1/5", 2026-02-25T08:41:26.075 INFO:teuthology.orchestra.run.trial023.stdout: "debug_filer": "0/1", 2026-02-25T08:41:26.075 INFO:teuthology.orchestra.run.trial023.stdout: "debug_filestore": "1/3", 2026-02-25T08:41:26.075 INFO:teuthology.orchestra.run.trial023.stdout: "debug_finisher": "1/1", 2026-02-25T08:41:26.075 INFO:teuthology.orchestra.run.trial023.stdout: "debug_fuse": "1/5", 2026-02-25T08:41:26.075 INFO:teuthology.orchestra.run.trial023.stdout: "debug_heartbeat_testing_span": "0", 2026-02-25T08:41:26.076 INFO:teuthology.orchestra.run.trial023.stdout: "debug_heartbeatmap": "1/5", 2026-02-25T08:41:26.076 INFO:teuthology.orchestra.run.trial023.stdout: "debug_immutable_obj_cache": "0/5", 2026-02-25T08:41:26.076 INFO:teuthology.orchestra.run.trial023.stdout: "debug_javaclient": "1/5", 2026-02-25T08:41:26.076 INFO:teuthology.orchestra.run.trial023.stdout: "debug_journal": "1/3", 2026-02-25T08:41:26.076 INFO:teuthology.orchestra.run.trial023.stdout: "debug_journaler": "0/5", 2026-02-25T08:41:26.076 INFO:teuthology.orchestra.run.trial023.stdout: "debug_kstore": "1/5", 2026-02-25T08:41:26.076 INFO:teuthology.orchestra.run.trial023.stdout: "debug_lockdep": "0/1", 2026-02-25T08:41:26.076 INFO:teuthology.orchestra.run.trial023.stdout: "debug_mclock": "1/5", 2026-02-25T08:41:26.076 INFO:teuthology.orchestra.run.trial023.stdout: "debug_mds": "1/5", 2026-02-25T08:41:26.076 INFO:teuthology.orchestra.run.trial023.stdout: "debug_mds_balancer": "1/5", 2026-02-25T08:41:26.076 INFO:teuthology.orchestra.run.trial023.stdout: "debug_mds_locker": "1/5", 2026-02-25T08:41:26.076 INFO:teuthology.orchestra.run.trial023.stdout: "debug_mds_log": "1/5", 2026-02-25T08:41:26.076 INFO:teuthology.orchestra.run.trial023.stdout: "debug_mds_log_expire": "1/5", 2026-02-25T08:41:26.076 INFO:teuthology.orchestra.run.trial023.stdout: "debug_mds_migrator": "1/5", 2026-02-25T08:41:26.076 INFO:teuthology.orchestra.run.trial023.stdout: "debug_mds_quiesce": "3/5", 2026-02-25T08:41:26.077 INFO:teuthology.orchestra.run.trial023.stdout: "debug_memstore": "1/5", 2026-02-25T08:41:26.077 INFO:teuthology.orchestra.run.trial023.stdout: "debug_mgr": "2/5", 2026-02-25T08:41:26.077 INFO:teuthology.orchestra.run.trial023.stdout: "debug_mgrc": "1/5", 2026-02-25T08:41:26.077 INFO:teuthology.orchestra.run.trial023.stdout: "debug_mon": "20/20", 2026-02-25T08:41:26.077 INFO:teuthology.orchestra.run.trial023.stdout: "debug_monc": "0/10", 2026-02-25T08:41:26.077 INFO:teuthology.orchestra.run.trial023.stdout: "debug_ms": "1/1", 2026-02-25T08:41:26.077 INFO:teuthology.orchestra.run.trial023.stdout: "debug_none": "0/5", 2026-02-25T08:41:26.077 INFO:teuthology.orchestra.run.trial023.stdout: "debug_objclass": "0/5", 2026-02-25T08:41:26.077 INFO:teuthology.orchestra.run.trial023.stdout: "debug_objectcacher": "0/5", 2026-02-25T08:41:26.077 INFO:teuthology.orchestra.run.trial023.stdout: "debug_objecter": "0/1", 2026-02-25T08:41:26.077 INFO:teuthology.orchestra.run.trial023.stdout: "debug_optracker": "0/5", 2026-02-25T08:41:26.077 INFO:teuthology.orchestra.run.trial023.stdout: "debug_osd": "1/5", 2026-02-25T08:41:26.077 INFO:teuthology.orchestra.run.trial023.stdout: "debug_paxos": "20/20", 2026-02-25T08:41:26.077 INFO:teuthology.orchestra.run.trial023.stdout: "debug_perfcounter": "1/5", 2026-02-25T08:41:26.078 INFO:teuthology.orchestra.run.trial023.stdout: "debug_prioritycache": "1/5", 2026-02-25T08:41:26.078 INFO:teuthology.orchestra.run.trial023.stdout: "debug_rados": "0/5", 2026-02-25T08:41:26.078 INFO:teuthology.orchestra.run.trial023.stdout: "debug_rbd": "0/5", 2026-02-25T08:41:26.078 INFO:teuthology.orchestra.run.trial023.stdout: "debug_rbd_mirror": "0/5", 2026-02-25T08:41:26.078 INFO:teuthology.orchestra.run.trial023.stdout: "debug_rbd_pwl": "0/5", 2026-02-25T08:41:26.078 INFO:teuthology.orchestra.run.trial023.stdout: "debug_rbd_replay": "0/5", 2026-02-25T08:41:26.078 INFO:teuthology.orchestra.run.trial023.stdout: "debug_refs": "0/0", 2026-02-25T08:41:26.078 INFO:teuthology.orchestra.run.trial023.stdout: "debug_reserver": "1/1", 2026-02-25T08:41:26.078 INFO:teuthology.orchestra.run.trial023.stdout: "debug_rgw": "1/5", 2026-02-25T08:41:26.078 INFO:teuthology.orchestra.run.trial023.stdout: "debug_rgw_access": "1/5", 2026-02-25T08:41:26.078 INFO:teuthology.orchestra.run.trial023.stdout: "debug_rgw_bucket_logging": "1/5", 2026-02-25T08:41:26.078 INFO:teuthology.orchestra.run.trial023.stdout: "debug_rgw_datacache": "1/5", 2026-02-25T08:41:26.078 INFO:teuthology.orchestra.run.trial023.stdout: "debug_rgw_dbstore": "1/5", 2026-02-25T08:41:26.078 INFO:teuthology.orchestra.run.trial023.stdout: "debug_rgw_dedup": "1/5", 2026-02-25T08:41:26.078 INFO:teuthology.orchestra.run.trial023.stdout: "debug_rgw_flight": "1/5", 2026-02-25T08:41:26.079 INFO:teuthology.orchestra.run.trial023.stdout: "debug_rgw_lifecycle": "1/5", 2026-02-25T08:41:26.079 INFO:teuthology.orchestra.run.trial023.stdout: "debug_rgw_notification": "1/5", 2026-02-25T08:41:26.079 INFO:teuthology.orchestra.run.trial023.stdout: "debug_rgw_restore": "1/5", 2026-02-25T08:41:26.079 INFO:teuthology.orchestra.run.trial023.stdout: "debug_rgw_sync": "1/5", 2026-02-25T08:41:26.079 INFO:teuthology.orchestra.run.trial023.stdout: "debug_rocksdb": "4/5", 2026-02-25T08:41:26.079 INFO:teuthology.orchestra.run.trial023.stdout: "debug_seastore": "0/5", 2026-02-25T08:41:26.079 INFO:teuthology.orchestra.run.trial023.stdout: "debug_seastore_backref": "0/5", 2026-02-25T08:41:26.079 INFO:teuthology.orchestra.run.trial023.stdout: "debug_seastore_cache": "0/5", 2026-02-25T08:41:26.079 INFO:teuthology.orchestra.run.trial023.stdout: "debug_seastore_cleaner": "0/5", 2026-02-25T08:41:26.079 INFO:teuthology.orchestra.run.trial023.stdout: "debug_seastore_device": "0/5", 2026-02-25T08:41:26.079 INFO:teuthology.orchestra.run.trial023.stdout: "debug_seastore_epm": "0/5", 2026-02-25T08:41:26.079 INFO:teuthology.orchestra.run.trial023.stdout: "debug_seastore_fixedkv_tree": "0/5", 2026-02-25T08:41:26.079 INFO:teuthology.orchestra.run.trial023.stdout: "debug_seastore_journal": "0/5", 2026-02-25T08:41:26.079 INFO:teuthology.orchestra.run.trial023.stdout: "debug_seastore_lba": "0/5", 2026-02-25T08:41:26.080 INFO:teuthology.orchestra.run.trial023.stdout: "debug_seastore_odata": "0/5", 2026-02-25T08:41:26.080 INFO:teuthology.orchestra.run.trial023.stdout: "debug_seastore_omap": "0/5", 2026-02-25T08:41:26.080 INFO:teuthology.orchestra.run.trial023.stdout: "debug_seastore_onode": "0/5", 2026-02-25T08:41:26.080 INFO:teuthology.orchestra.run.trial023.stdout: "debug_seastore_t": "0/5", 2026-02-25T08:41:26.080 INFO:teuthology.orchestra.run.trial023.stdout: "debug_seastore_tm": "0/5", 2026-02-25T08:41:26.080 INFO:teuthology.orchestra.run.trial023.stdout: "debug_striper": "0/1", 2026-02-25T08:41:26.080 INFO:teuthology.orchestra.run.trial023.stdout: "debug_test": "0/5", 2026-02-25T08:41:26.080 INFO:teuthology.orchestra.run.trial023.stdout: "debug_throttle": "1/1", 2026-02-25T08:41:26.080 INFO:teuthology.orchestra.run.trial023.stdout: "debug_timer": "0/1", 2026-02-25T08:41:26.080 INFO:teuthology.orchestra.run.trial023.stdout: "debug_tp": "0/5", 2026-02-25T08:41:26.080 INFO:teuthology.orchestra.run.trial023.stdout: "debug_trace": "1/5", 2026-02-25T08:41:26.080 INFO:teuthology.orchestra.run.trial023.stdout: "defer_client_eviction_on_laggy_osds": "false", 2026-02-25T08:41:26.080 INFO:teuthology.orchestra.run.trial023.stdout: "device_failure_prediction_mode": "none", 2026-02-25T08:41:26.080 INFO:teuthology.orchestra.run.trial023.stdout: "ec_extent_cache_size": "10485760", 2026-02-25T08:41:26.080 INFO:teuthology.orchestra.run.trial023.stdout: "ec_pdw_write_mode": "0", 2026-02-25T08:41:26.081 INFO:teuthology.orchestra.run.trial023.stdout: "enable_availability_tracking": "false", 2026-02-25T08:41:26.081 INFO:teuthology.orchestra.run.trial023.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "crimson", 2026-02-25T08:41:26.081 INFO:teuthology.orchestra.run.trial023.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2026-02-25T08:41:26.081 INFO:teuthology.orchestra.run.trial023.stdout: "err_to_graylog": "false", 2026-02-25T08:41:26.081 INFO:teuthology.orchestra.run.trial023.stdout: "err_to_journald": "false", 2026-02-25T08:41:26.081 INFO:teuthology.orchestra.run.trial023.stdout: "err_to_stderr": "true", 2026-02-25T08:41:26.081 INFO:teuthology.orchestra.run.trial023.stdout: "err_to_syslog": "false", 2026-02-25T08:41:26.081 INFO:teuthology.orchestra.run.trial023.stdout: "event_tracing": "false", 2026-02-25T08:41:26.081 INFO:teuthology.orchestra.run.trial023.stdout: "exporter_addr": "0.0.0.0", 2026-02-25T08:41:26.081 INFO:teuthology.orchestra.run.trial023.stdout: "exporter_cert_file": "", 2026-02-25T08:41:26.081 INFO:teuthology.orchestra.run.trial023.stdout: "exporter_http_port": "9926", 2026-02-25T08:41:26.081 INFO:teuthology.orchestra.run.trial023.stdout: "exporter_key_file": "", 2026-02-25T08:41:26.081 INFO:teuthology.orchestra.run.trial023.stdout: "exporter_prio_limit": "5", 2026-02-25T08:41:26.081 INFO:teuthology.orchestra.run.trial023.stdout: "exporter_sock_dir": "/var/run/ceph/", 2026-02-25T08:41:26.081 INFO:teuthology.orchestra.run.trial023.stdout: "exporter_sort_metrics": "true", 2026-02-25T08:41:26.081 INFO:teuthology.orchestra.run.trial023.stdout: "exporter_stats_period": "5", 2026-02-25T08:41:26.082 INFO:teuthology.orchestra.run.trial023.stdout: "fake_statfs_for_testing": "0", 2026-02-25T08:41:26.082 INFO:teuthology.orchestra.run.trial023.stdout: "fatal_signal_handlers": "true", 2026-02-25T08:41:26.082 INFO:teuthology.orchestra.run.trial023.stdout: "filer_max_purge_ops": "10", 2026-02-25T08:41:26.082 INFO:teuthology.orchestra.run.trial023.stdout: "filer_max_truncate_ops": "128", 2026-02-25T08:41:26.082 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_apply_finisher_threads": "1", 2026-02-25T08:41:26.082 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_blackhole": "false", 2026-02-25T08:41:26.082 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_btrfs_clone_range": "true", 2026-02-25T08:41:26.082 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_btrfs_snap": "true", 2026-02-25T08:41:26.082 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_caller_concurrency": "10", 2026-02-25T08:41:26.082 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_collect_device_partition_information": "true", 2026-02-25T08:41:26.082 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_commit_timeout": "600.000000", 2026-02-25T08:41:26.082 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_debug_inject_read_err": "false", 2026-02-25T08:41:26.082 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_debug_omap_check": "false", 2026-02-25T08:41:26.082 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_debug_verify_split": "false", 2026-02-25T08:41:26.082 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_dump_file": "", 2026-02-25T08:41:26.082 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2026-02-25T08:41:26.083 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_expected_throughput_ops": "200.000000", 2026-02-25T08:41:26.083 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_fadvise": "true", 2026-02-25T08:41:26.083 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_fail_eio": "true", 2026-02-25T08:41:26.083 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_fd_cache_shards": "16", 2026-02-25T08:41:26.083 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_fd_cache_size": "128", 2026-02-25T08:41:26.083 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_fiemap": "false", 2026-02-25T08:41:26.083 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_fiemap_threshold": "4096", 2026-02-25T08:41:26.083 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_fsync_flushes_journal_data": "false", 2026-02-25T08:41:26.083 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_index_retry_probability": "0.000000", 2026-02-25T08:41:26.083 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_inject_stall": "0", 2026-02-25T08:41:26.083 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_journal_parallel": "false", 2026-02-25T08:41:26.083 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_journal_trailing": "false", 2026-02-25T08:41:26.083 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_journal_writeahead": "false", 2026-02-25T08:41:26.083 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_kill_at": "0", 2026-02-25T08:41:26.083 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_max_alloc_hint_size": "1048576", 2026-02-25T08:41:26.083 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_max_inline_xattr_size": "0", 2026-02-25T08:41:26.083 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2026-02-25T08:41:26.084 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_max_inline_xattr_size_other": "512", 2026-02-25T08:41:26.084 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2026-02-25T08:41:26.084 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_max_inline_xattrs": "0", 2026-02-25T08:41:26.084 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2026-02-25T08:41:26.084 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_max_inline_xattrs_other": "2", 2026-02-25T08:41:26.084 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_max_inline_xattrs_xfs": "10", 2026-02-25T08:41:26.084 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_max_sync_interval": "5.000000", 2026-02-25T08:41:26.084 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_max_xattr_value_size": "0", 2026-02-25T08:41:26.084 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2026-02-25T08:41:26.084 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_max_xattr_value_size_other": "1024", 2026-02-25T08:41:26.084 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2026-02-25T08:41:26.084 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_merge_threshold": "-10", 2026-02-25T08:41:26.084 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_min_sync_interval": "0.010000", 2026-02-25T08:41:26.084 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_odsync_write": "false", 2026-02-25T08:41:26.084 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_omap_backend": "rocksdb", 2026-02-25T08:41:26.084 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_omap_backend_path": "", 2026-02-25T08:41:26.084 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_omap_header_cache_size": "1024", 2026-02-25T08:41:26.084 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_ondisk_finisher_threads": "1", 2026-02-25T08:41:26.085 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_op_thread_suicide_timeout": "180", 2026-02-25T08:41:26.085 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_op_thread_timeout": "60", 2026-02-25T08:41:26.085 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_op_threads": "2", 2026-02-25T08:41:26.085 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_punch_hole": "false", 2026-02-25T08:41:26.085 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2026-02-25T08:41:26.085 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2026-02-25T08:41:26.085 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2026-02-25T08:41:26.085 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_queue_high_threshhold": "0.900000", 2026-02-25T08:41:26.085 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_queue_low_threshhold": "0.300000", 2026-02-25T08:41:26.085 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_queue_max_bytes": "104857600", 2026-02-25T08:41:26.085 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2026-02-25T08:41:26.085 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2026-02-25T08:41:26.085 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2026-02-25T08:41:26.085 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_queue_max_ops": "50", 2026-02-25T08:41:26.085 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2026-02-25T08:41:26.085 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_seek_data_hole": "false", 2026-02-25T08:41:26.085 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_sloppy_crc": "false", 2026-02-25T08:41:26.085 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_sloppy_crc_block_size": "65536", 2026-02-25T08:41:26.085 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_splice": "false", 2026-02-25T08:41:26.086 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_split_multiple": "2", 2026-02-25T08:41:26.086 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_split_rand_factor": "20", 2026-02-25T08:41:26.086 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_update_to": "1000", 2026-02-25T08:41:26.086 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2026-02-25T08:41:26.086 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2026-02-25T08:41:26.086 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2026-02-25T08:41:26.086 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2026-02-25T08:41:26.086 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2026-02-25T08:41:26.086 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2026-02-25T08:41:26.086 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_wbthrottle_enable": "true", 2026-02-25T08:41:26.086 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2026-02-25T08:41:26.086 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2026-02-25T08:41:26.086 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2026-02-25T08:41:26.086 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2026-02-25T08:41:26.086 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2026-02-25T08:41:26.086 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2026-02-25T08:41:26.086 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_xfs_extsize": "false", 2026-02-25T08:41:26.086 INFO:teuthology.orchestra.run.trial023.stdout: "filestore_zfs_snap": "false", 2026-02-25T08:41:26.087 INFO:teuthology.orchestra.run.trial023.stdout: "fio_dir": "/tmp/fio", 2026-02-25T08:41:26.087 INFO:teuthology.orchestra.run.trial023.stdout: "fsid": "542a2d4e-0dbc-4a03-9358-454614077093", 2026-02-25T08:41:26.087 INFO:teuthology.orchestra.run.trial023.stdout: "fuse_allow_other": "true", 2026-02-25T08:41:26.087 INFO:teuthology.orchestra.run.trial023.stdout: "fuse_atomic_o_trunc": "true", 2026-02-25T08:41:26.087 INFO:teuthology.orchestra.run.trial023.stdout: "fuse_big_writes": "true", 2026-02-25T08:41:26.087 INFO:teuthology.orchestra.run.trial023.stdout: "fuse_debug": "false", 2026-02-25T08:41:26.087 INFO:teuthology.orchestra.run.trial023.stdout: "fuse_default_permissions": "false", 2026-02-25T08:41:26.087 INFO:teuthology.orchestra.run.trial023.stdout: "fuse_disable_pagecache": "false", 2026-02-25T08:41:26.087 INFO:teuthology.orchestra.run.trial023.stdout: "fuse_max_write": "0", 2026-02-25T08:41:26.087 INFO:teuthology.orchestra.run.trial023.stdout: "fuse_multithreaded": "true", 2026-02-25T08:41:26.087 INFO:teuthology.orchestra.run.trial023.stdout: "fuse_require_active_mds": "true", 2026-02-25T08:41:26.087 INFO:teuthology.orchestra.run.trial023.stdout: "fuse_set_user_groups": "true", 2026-02-25T08:41:26.087 INFO:teuthology.orchestra.run.trial023.stdout: "fuse_splice_move": "true", 2026-02-25T08:41:26.087 INFO:teuthology.orchestra.run.trial023.stdout: "fuse_splice_read": "true", 2026-02-25T08:41:26.087 INFO:teuthology.orchestra.run.trial023.stdout: "fuse_splice_write": "true", 2026-02-25T08:41:26.087 INFO:teuthology.orchestra.run.trial023.stdout: "fuse_syncfs_on_mksnap": "true", 2026-02-25T08:41:26.087 INFO:teuthology.orchestra.run.trial023.stdout: "fuse_use_invalidate_cb": "true", 2026-02-25T08:41:26.087 INFO:teuthology.orchestra.run.trial023.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2026-02-25T08:41:26.087 INFO:teuthology.orchestra.run.trial023.stdout: "gss_target_name": "ceph", 2026-02-25T08:41:26.087 INFO:teuthology.orchestra.run.trial023.stdout: "heartbeat_file": "", 2026-02-25T08:41:26.088 INFO:teuthology.orchestra.run.trial023.stdout: "heartbeat_inject_failure": "0", 2026-02-25T08:41:26.088 INFO:teuthology.orchestra.run.trial023.stdout: "heartbeat_interval": "5", 2026-02-25T08:41:26.088 INFO:teuthology.orchestra.run.trial023.stdout: "host": "trial023", 2026-02-25T08:41:26.088 INFO:teuthology.orchestra.run.trial023.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2026-02-25T08:41:26.088 INFO:teuthology.orchestra.run.trial023.stdout: "immutable_object_cache_max_inflight_ops": "128", 2026-02-25T08:41:26.088 INFO:teuthology.orchestra.run.trial023.stdout: "immutable_object_cache_max_size": "1073741824", 2026-02-25T08:41:26.088 INFO:teuthology.orchestra.run.trial023.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2026-02-25T08:41:26.088 INFO:teuthology.orchestra.run.trial023.stdout: "immutable_object_cache_qos_bps_burst": "0", 2026-02-25T08:41:26.088 INFO:teuthology.orchestra.run.trial023.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2026-02-25T08:41:26.088 INFO:teuthology.orchestra.run.trial023.stdout: "immutable_object_cache_qos_bps_limit": "0", 2026-02-25T08:41:26.088 INFO:teuthology.orchestra.run.trial023.stdout: "immutable_object_cache_qos_iops_burst": "0", 2026-02-25T08:41:26.088 INFO:teuthology.orchestra.run.trial023.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2026-02-25T08:41:26.088 INFO:teuthology.orchestra.run.trial023.stdout: "immutable_object_cache_qos_iops_limit": "0", 2026-02-25T08:41:26.088 INFO:teuthology.orchestra.run.trial023.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2026-02-25T08:41:26.088 INFO:teuthology.orchestra.run.trial023.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2026-02-25T08:41:26.088 INFO:teuthology.orchestra.run.trial023.stdout: "immutable_object_cache_watermark": "0.900000", 2026-02-25T08:41:26.088 INFO:teuthology.orchestra.run.trial023.stdout: "inject_early_sigterm": "false", 2026-02-25T08:41:26.088 INFO:teuthology.orchestra.run.trial023.stdout: "jaeger_agent_port": "6799", 2026-02-25T08:41:26.089 INFO:teuthology.orchestra.run.trial023.stdout: "jaeger_tracing_enable": "false", 2026-02-25T08:41:26.089 INFO:teuthology.orchestra.run.trial023.stdout: "journal_aio": "true", 2026-02-25T08:41:26.089 INFO:teuthology.orchestra.run.trial023.stdout: "journal_align_min_size": "65536", 2026-02-25T08:41:26.089 INFO:teuthology.orchestra.run.trial023.stdout: "journal_block_align": "true", 2026-02-25T08:41:26.089 INFO:teuthology.orchestra.run.trial023.stdout: "journal_block_size": "4096", 2026-02-25T08:41:26.089 INFO:teuthology.orchestra.run.trial023.stdout: "journal_dio": "true", 2026-02-25T08:41:26.089 INFO:teuthology.orchestra.run.trial023.stdout: "journal_discard": "false", 2026-02-25T08:41:26.089 INFO:teuthology.orchestra.run.trial023.stdout: "journal_force_aio": "false", 2026-02-25T08:41:26.089 INFO:teuthology.orchestra.run.trial023.stdout: "journal_ignore_corruption": "false", 2026-02-25T08:41:26.089 INFO:teuthology.orchestra.run.trial023.stdout: "journal_max_write_bytes": "10485760", 2026-02-25T08:41:26.089 INFO:teuthology.orchestra.run.trial023.stdout: "journal_max_write_entries": "100", 2026-02-25T08:41:26.089 INFO:teuthology.orchestra.run.trial023.stdout: "journal_replay_from": "0", 2026-02-25T08:41:26.089 INFO:teuthology.orchestra.run.trial023.stdout: "journal_throttle_high_multiple": "0.000000", 2026-02-25T08:41:26.089 INFO:teuthology.orchestra.run.trial023.stdout: "journal_throttle_high_threshhold": "0.900000", 2026-02-25T08:41:26.089 INFO:teuthology.orchestra.run.trial023.stdout: "journal_throttle_low_threshhold": "0.600000", 2026-02-25T08:41:26.089 INFO:teuthology.orchestra.run.trial023.stdout: "journal_throttle_max_multiple": "0.000000", 2026-02-25T08:41:26.089 INFO:teuthology.orchestra.run.trial023.stdout: "journal_write_header_frequency": "0", 2026-02-25T08:41:26.089 INFO:teuthology.orchestra.run.trial023.stdout: "journal_zero_on_create": "false", 2026-02-25T08:41:26.089 INFO:teuthology.orchestra.run.trial023.stdout: "journaler_prefetch_periods": "10", 2026-02-25T08:41:26.089 INFO:teuthology.orchestra.run.trial023.stdout: "journaler_prezero_periods": "5", 2026-02-25T08:41:26.089 INFO:teuthology.orchestra.run.trial023.stdout: "journaler_write_head_interval": "15", 2026-02-25T08:41:26.090 INFO:teuthology.orchestra.run.trial023.stdout: "key": "", 2026-02-25T08:41:26.090 INFO:teuthology.orchestra.run.trial023.stdout: "keyfile": "", 2026-02-25T08:41:26.090 INFO:teuthology.orchestra.run.trial023.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2026-02-25T08:41:26.090 INFO:teuthology.orchestra.run.trial023.stdout: "librados_thread_count": "2", 2026-02-25T08:41:26.090 INFO:teuthology.orchestra.run.trial023.stdout: "lockdep": "false", 2026-02-25T08:41:26.090 INFO:teuthology.orchestra.run.trial023.stdout: "lockdep_force_backtrace": "false", 2026-02-25T08:41:26.090 INFO:teuthology.orchestra.run.trial023.stdout: "log_coarse_timestamps": "true", 2026-02-25T08:41:26.090 INFO:teuthology.orchestra.run.trial023.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2026-02-25T08:41:26.090 INFO:teuthology.orchestra.run.trial023.stdout: "log_flush_on_exit": "false", 2026-02-25T08:41:26.090 INFO:teuthology.orchestra.run.trial023.stdout: "log_graylog_host": "127.0.0.1", 2026-02-25T08:41:26.090 INFO:teuthology.orchestra.run.trial023.stdout: "log_graylog_port": "12201", 2026-02-25T08:41:26.090 INFO:teuthology.orchestra.run.trial023.stdout: "log_max_new": "1000", 2026-02-25T08:41:26.090 INFO:teuthology.orchestra.run.trial023.stdout: "log_max_recent": "10000", 2026-02-25T08:41:26.090 INFO:teuthology.orchestra.run.trial023.stdout: "log_stderr_prefix": "", 2026-02-25T08:41:26.090 INFO:teuthology.orchestra.run.trial023.stdout: "log_stop_at_utilization": "0.970000", 2026-02-25T08:41:26.090 INFO:teuthology.orchestra.run.trial023.stdout: "log_to_file": "true", 2026-02-25T08:41:26.090 INFO:teuthology.orchestra.run.trial023.stdout: "log_to_graylog": "false", 2026-02-25T08:41:26.090 INFO:teuthology.orchestra.run.trial023.stdout: "log_to_journald": "false", 2026-02-25T08:41:26.090 INFO:teuthology.orchestra.run.trial023.stdout: "log_to_stderr": "false", 2026-02-25T08:41:26.090 INFO:teuthology.orchestra.run.trial023.stdout: "log_to_syslog": "false", 2026-02-25T08:41:26.090 INFO:teuthology.orchestra.run.trial023.stdout: "max_rotating_auth_attempts": "10", 2026-02-25T08:41:26.090 INFO:teuthology.orchestra.run.trial023.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2026-02-25T08:41:26.091 INFO:teuthology.orchestra.run.trial023.stdout: "mds_action_on_write_error": "1", 2026-02-25T08:41:26.091 INFO:teuthology.orchestra.run.trial023.stdout: "mds_allow_async_dirops": "true", 2026-02-25T08:41:26.091 INFO:teuthology.orchestra.run.trial023.stdout: "mds_allow_batched_ops": "true", 2026-02-25T08:41:26.091 INFO:teuthology.orchestra.run.trial023.stdout: "mds_alternate_name_max": "8192", 2026-02-25T08:41:26.091 INFO:teuthology.orchestra.run.trial023.stdout: "mds_asio_thread_count": "2", 2026-02-25T08:41:26.091 INFO:teuthology.orchestra.run.trial023.stdout: "mds_bal_export_pin": "true", 2026-02-25T08:41:26.091 INFO:teuthology.orchestra.run.trial023.stdout: "mds_bal_fragment_dirs": "true", 2026-02-25T08:41:26.091 INFO:teuthology.orchestra.run.trial023.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2026-02-25T08:41:26.091 INFO:teuthology.orchestra.run.trial023.stdout: "mds_bal_fragment_interval": "5", 2026-02-25T08:41:26.091 INFO:teuthology.orchestra.run.trial023.stdout: "mds_bal_fragment_size_max": "100000", 2026-02-25T08:41:26.091 INFO:teuthology.orchestra.run.trial023.stdout: "mds_bal_idle_threshold": "0.000000", 2026-02-25T08:41:26.091 INFO:teuthology.orchestra.run.trial023.stdout: "mds_bal_interval": "10", 2026-02-25T08:41:26.091 INFO:teuthology.orchestra.run.trial023.stdout: "mds_bal_max": "-1", 2026-02-25T08:41:26.091 INFO:teuthology.orchestra.run.trial023.stdout: "mds_bal_max_until": "-1", 2026-02-25T08:41:26.091 INFO:teuthology.orchestra.run.trial023.stdout: "mds_bal_merge_size": "50", 2026-02-25T08:41:26.091 INFO:teuthology.orchestra.run.trial023.stdout: "mds_bal_midchunk": "0.300000", 2026-02-25T08:41:26.091 INFO:teuthology.orchestra.run.trial023.stdout: "mds_bal_min_rebalance": "0.100000", 2026-02-25T08:41:26.091 INFO:teuthology.orchestra.run.trial023.stdout: "mds_bal_min_start": "0.200000", 2026-02-25T08:41:26.091 INFO:teuthology.orchestra.run.trial023.stdout: "mds_bal_minchunk": "0.001000", 2026-02-25T08:41:26.091 INFO:teuthology.orchestra.run.trial023.stdout: "mds_bal_mode": "0", 2026-02-25T08:41:26.091 INFO:teuthology.orchestra.run.trial023.stdout: "mds_bal_need_max": "1.200000", 2026-02-25T08:41:26.092 INFO:teuthology.orchestra.run.trial023.stdout: "mds_bal_need_min": "0.800000", 2026-02-25T08:41:26.092 INFO:teuthology.orchestra.run.trial023.stdout: "mds_bal_overload_epochs": "2", 2026-02-25T08:41:26.092 INFO:teuthology.orchestra.run.trial023.stdout: "mds_bal_replicate_threshold": "8000.000000", 2026-02-25T08:41:26.092 INFO:teuthology.orchestra.run.trial023.stdout: "mds_bal_sample_interval": "3.000000", 2026-02-25T08:41:26.092 INFO:teuthology.orchestra.run.trial023.stdout: "mds_bal_split_bits": "3", 2026-02-25T08:41:26.092 INFO:teuthology.orchestra.run.trial023.stdout: "mds_bal_split_rd": "25000.000000", 2026-02-25T08:41:26.092 INFO:teuthology.orchestra.run.trial023.stdout: "mds_bal_split_size": "10000", 2026-02-25T08:41:26.092 INFO:teuthology.orchestra.run.trial023.stdout: "mds_bal_split_wr": "10000.000000", 2026-02-25T08:41:26.092 INFO:teuthology.orchestra.run.trial023.stdout: "mds_bal_target_decay": "10.000000", 2026-02-25T08:41:26.092 INFO:teuthology.orchestra.run.trial023.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2026-02-25T08:41:26.092 INFO:teuthology.orchestra.run.trial023.stdout: "mds_beacon_grace": "15.000000", 2026-02-25T08:41:26.092 INFO:teuthology.orchestra.run.trial023.stdout: "mds_beacon_interval": "4.000000", 2026-02-25T08:41:26.092 INFO:teuthology.orchestra.run.trial023.stdout: "mds_beacon_mon_down_grace": "60", 2026-02-25T08:41:26.092 INFO:teuthology.orchestra.run.trial023.stdout: "mds_cache_memory_limit": "4294967296", 2026-02-25T08:41:26.092 INFO:teuthology.orchestra.run.trial023.stdout: "mds_cache_mid": "0.700000", 2026-02-25T08:41:26.092 INFO:teuthology.orchestra.run.trial023.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2026-02-25T08:41:26.092 INFO:teuthology.orchestra.run.trial023.stdout: "mds_cache_quiesce_delay": "0", 2026-02-25T08:41:26.092 INFO:teuthology.orchestra.run.trial023.stdout: "mds_cache_quiesce_sleep": "200", 2026-02-25T08:41:26.092 INFO:teuthology.orchestra.run.trial023.stdout: "mds_cache_quiesce_splitauth": "true", 2026-02-25T08:41:26.092 INFO:teuthology.orchestra.run.trial023.stdout: "mds_cache_quiesce_threshold": "524288", 2026-02-25T08:41:26.092 INFO:teuthology.orchestra.run.trial023.stdout: "mds_cache_release_free_interval": "10", 2026-02-25T08:41:26.092 INFO:teuthology.orchestra.run.trial023.stdout: "mds_cache_reservation": "0.050000", 2026-02-25T08:41:26.092 INFO:teuthology.orchestra.run.trial023.stdout: "mds_cache_trim_decay_rate": "1.000000", 2026-02-25T08:41:26.093 INFO:teuthology.orchestra.run.trial023.stdout: "mds_cache_trim_interval": "1", 2026-02-25T08:41:26.093 INFO:teuthology.orchestra.run.trial023.stdout: "mds_cache_trim_threshold": "262144", 2026-02-25T08:41:26.093 INFO:teuthology.orchestra.run.trial023.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2026-02-25T08:41:26.093 INFO:teuthology.orchestra.run.trial023.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2026-02-25T08:41:26.093 INFO:teuthology.orchestra.run.trial023.stdout: "mds_client_delegate_inos_pct": "50", 2026-02-25T08:41:26.093 INFO:teuthology.orchestra.run.trial023.stdout: "mds_client_prealloc_inos": "1000", 2026-02-25T08:41:26.093 INFO:teuthology.orchestra.run.trial023.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2026-02-25T08:41:26.093 INFO:teuthology.orchestra.run.trial023.stdout: "mds_connect_bootstrapping": "false", 2026-02-25T08:41:26.093 INFO:teuthology.orchestra.run.trial023.stdout: "mds_damage_table_max_entries": "10000", 2026-02-25T08:41:26.093 INFO:teuthology.orchestra.run.trial023.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2026-02-25T08:41:26.093 INFO:teuthology.orchestra.run.trial023.stdout: "mds_debug_auth_pins": "false", 2026-02-25T08:41:26.093 INFO:teuthology.orchestra.run.trial023.stdout: "mds_debug_frag": "false", 2026-02-25T08:41:26.093 INFO:teuthology.orchestra.run.trial023.stdout: "mds_debug_scatterstat": "false", 2026-02-25T08:41:26.093 INFO:teuthology.orchestra.run.trial023.stdout: "mds_debug_subtrees": "false", 2026-02-25T08:41:26.093 INFO:teuthology.orchestra.run.trial023.stdout: "mds_decay_halflife": "5.000000", 2026-02-25T08:41:26.093 INFO:teuthology.orchestra.run.trial023.stdout: "mds_default_dir_hash": "2", 2026-02-25T08:41:26.093 INFO:teuthology.orchestra.run.trial023.stdout: "mds_defer_session_stale": "true", 2026-02-25T08:41:26.093 INFO:teuthology.orchestra.run.trial023.stdout: "mds_delay_journal_replay_for_testing": "0", 2026-02-25T08:41:26.093 INFO:teuthology.orchestra.run.trial023.stdout: "mds_deny_all_reconnect": "false", 2026-02-25T08:41:26.093 INFO:teuthology.orchestra.run.trial023.stdout: "mds_dir_keys_per_op": "16384", 2026-02-25T08:41:26.093 INFO:teuthology.orchestra.run.trial023.stdout: "mds_dir_max_commit_size": "10", 2026-02-25T08:41:26.093 INFO:teuthology.orchestra.run.trial023.stdout: "mds_dir_max_entries": "0", 2026-02-25T08:41:26.093 INFO:teuthology.orchestra.run.trial023.stdout: "mds_dir_prefetch": "true", 2026-02-25T08:41:26.094 INFO:teuthology.orchestra.run.trial023.stdout: "mds_dirstat_min_interval": "1.000000", 2026-02-25T08:41:26.094 INFO:teuthology.orchestra.run.trial023.stdout: "mds_dump_cache_after_rejoin": "false", 2026-02-25T08:41:26.094 INFO:teuthology.orchestra.run.trial023.stdout: "mds_dump_cache_on_map": "false", 2026-02-25T08:41:26.094 INFO:teuthology.orchestra.run.trial023.stdout: "mds_dump_cache_threshold_file": "0", 2026-02-25T08:41:26.094 INFO:teuthology.orchestra.run.trial023.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2026-02-25T08:41:26.094 INFO:teuthology.orchestra.run.trial023.stdout: "mds_early_reply": "true", 2026-02-25T08:41:26.094 INFO:teuthology.orchestra.run.trial023.stdout: "mds_enable_op_tracker": "true", 2026-02-25T08:41:26.094 INFO:teuthology.orchestra.run.trial023.stdout: "mds_enforce_unique_name": "true", 2026-02-25T08:41:26.094 INFO:teuthology.orchestra.run.trial023.stdout: "mds_export_ephemeral_distributed": "true", 2026-02-25T08:41:26.094 INFO:teuthology.orchestra.run.trial023.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2026-02-25T08:41:26.094 INFO:teuthology.orchestra.run.trial023.stdout: "mds_export_ephemeral_random": "true", 2026-02-25T08:41:26.094 INFO:teuthology.orchestra.run.trial023.stdout: "mds_export_ephemeral_random_max": "0.010000", 2026-02-25T08:41:26.094 INFO:teuthology.orchestra.run.trial023.stdout: "mds_extraordinary_events_dump_interval": "0", 2026-02-25T08:41:26.094 INFO:teuthology.orchestra.run.trial023.stdout: "mds_file_blockdiff_max_concurrent_object_scans": "16", 2026-02-25T08:41:26.094 INFO:teuthology.orchestra.run.trial023.stdout: "mds_forward_all_requests_to_auth": "false", 2026-02-25T08:41:26.094 INFO:teuthology.orchestra.run.trial023.stdout: "mds_freeze_tree_timeout": "30.000000", 2026-02-25T08:41:26.094 INFO:teuthology.orchestra.run.trial023.stdout: "mds_fscrypt_last_block_max_size": "4096", 2026-02-25T08:41:26.094 INFO:teuthology.orchestra.run.trial023.stdout: "mds_go_bad_corrupt_dentry": "true", 2026-02-25T08:41:26.094 INFO:teuthology.orchestra.run.trial023.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2026-02-25T08:41:26.094 INFO:teuthology.orchestra.run.trial023.stdout: "mds_health_cache_threshold": "1.500000", 2026-02-25T08:41:26.094 INFO:teuthology.orchestra.run.trial023.stdout: "mds_health_summarize_threshold": "10", 2026-02-25T08:41:26.094 INFO:teuthology.orchestra.run.trial023.stdout: "mds_heartbeat_grace": "15.000000", 2026-02-25T08:41:26.094 INFO:teuthology.orchestra.run.trial023.stdout: "mds_heartbeat_reset_grace": "1000", 2026-02-25T08:41:26.094 INFO:teuthology.orchestra.run.trial023.stdout: "mds_inject_health_dummy": "false", 2026-02-25T08:41:26.095 INFO:teuthology.orchestra.run.trial023.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2026-02-25T08:41:26.095 INFO:teuthology.orchestra.run.trial023.stdout: "mds_inject_migrator_session_race": "false", 2026-02-25T08:41:26.095 INFO:teuthology.orchestra.run.trial023.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2026-02-25T08:41:26.095 INFO:teuthology.orchestra.run.trial023.stdout: "mds_inject_skip_replaying_inotable": "false", 2026-02-25T08:41:26.095 INFO:teuthology.orchestra.run.trial023.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2026-02-25T08:41:26.095 INFO:teuthology.orchestra.run.trial023.stdout: "mds_join_fs": "", 2026-02-25T08:41:26.095 INFO:teuthology.orchestra.run.trial023.stdout: "mds_journal_format": "1", 2026-02-25T08:41:26.095 INFO:teuthology.orchestra.run.trial023.stdout: "mds_kill_after_journal_logs_flushed": "false", 2026-02-25T08:41:26.095 INFO:teuthology.orchestra.run.trial023.stdout: "mds_kill_create_at": "0", 2026-02-25T08:41:26.095 INFO:teuthology.orchestra.run.trial023.stdout: "mds_kill_dirfrag_at": "0", 2026-02-25T08:41:26.095 INFO:teuthology.orchestra.run.trial023.stdout: "mds_kill_export_at": "0", 2026-02-25T08:41:26.095 INFO:teuthology.orchestra.run.trial023.stdout: "mds_kill_import_at": "0", 2026-02-25T08:41:26.095 INFO:teuthology.orchestra.run.trial023.stdout: "mds_kill_journal_at": "0", 2026-02-25T08:41:26.095 INFO:teuthology.orchestra.run.trial023.stdout: "mds_kill_journal_expire_at": "0", 2026-02-25T08:41:26.095 INFO:teuthology.orchestra.run.trial023.stdout: "mds_kill_journal_replay_at": "0", 2026-02-25T08:41:26.095 INFO:teuthology.orchestra.run.trial023.stdout: "mds_kill_link_at": "0", 2026-02-25T08:41:26.095 INFO:teuthology.orchestra.run.trial023.stdout: "mds_kill_mdstable_at": "0", 2026-02-25T08:41:26.095 INFO:teuthology.orchestra.run.trial023.stdout: "mds_kill_openc_at": "0", 2026-02-25T08:41:26.095 INFO:teuthology.orchestra.run.trial023.stdout: "mds_kill_rename_at": "0", 2026-02-25T08:41:26.095 INFO:teuthology.orchestra.run.trial023.stdout: "mds_kill_shutdown_at": "0", 2026-02-25T08:41:26.095 INFO:teuthology.orchestra.run.trial023.stdout: "mds_log_event_large_threshold": "524288", 2026-02-25T08:41:26.095 INFO:teuthology.orchestra.run.trial023.stdout: "mds_log_events_per_segment": "1024", 2026-02-25T08:41:26.096 INFO:teuthology.orchestra.run.trial023.stdout: "mds_log_max_events": "-1", 2026-02-25T08:41:26.096 INFO:teuthology.orchestra.run.trial023.stdout: "mds_log_max_segments": "128", 2026-02-25T08:41:26.096 INFO:teuthology.orchestra.run.trial023.stdout: "mds_log_minor_segments_per_major_segment": "16", 2026-02-25T08:41:26.096 INFO:teuthology.orchestra.run.trial023.stdout: "mds_log_pause": "false", 2026-02-25T08:41:26.096 INFO:teuthology.orchestra.run.trial023.stdout: "mds_log_segment_size": "0", 2026-02-25T08:41:26.096 INFO:teuthology.orchestra.run.trial023.stdout: "mds_log_skip_corrupt_events": "false", 2026-02-25T08:41:26.096 INFO:teuthology.orchestra.run.trial023.stdout: "mds_log_skip_unbounded_events": "false", 2026-02-25T08:41:26.096 INFO:teuthology.orchestra.run.trial023.stdout: "mds_log_trim_decay_rate": "1.000000", 2026-02-25T08:41:26.096 INFO:teuthology.orchestra.run.trial023.stdout: "mds_log_trim_threshold": "128", 2026-02-25T08:41:26.096 INFO:teuthology.orchestra.run.trial023.stdout: "mds_log_trim_upkeep_interval": "1000", 2026-02-25T08:41:26.096 INFO:teuthology.orchestra.run.trial023.stdout: "mds_log_warn_factor": "2.000000", 2026-02-25T08:41:26.096 INFO:teuthology.orchestra.run.trial023.stdout: "mds_max_caps_per_client": "1048576", 2026-02-25T08:41:26.096 INFO:teuthology.orchestra.run.trial023.stdout: "mds_max_completed_flushes": "100000", 2026-02-25T08:41:26.096 INFO:teuthology.orchestra.run.trial023.stdout: "mds_max_completed_requests": "100000", 2026-02-25T08:41:26.096 INFO:teuthology.orchestra.run.trial023.stdout: "mds_max_export_size": "20971520", 2026-02-25T08:41:26.096 INFO:teuthology.orchestra.run.trial023.stdout: "mds_max_file_recover": "32", 2026-02-25T08:41:26.096 INFO:teuthology.orchestra.run.trial023.stdout: "mds_max_purge_files": "64", 2026-02-25T08:41:26.096 INFO:teuthology.orchestra.run.trial023.stdout: "mds_max_purge_ops": "8192", 2026-02-25T08:41:26.096 INFO:teuthology.orchestra.run.trial023.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2026-02-25T08:41:26.096 INFO:teuthology.orchestra.run.trial023.stdout: "mds_max_scrub_ops_in_progress": "5", 2026-02-25T08:41:26.096 INFO:teuthology.orchestra.run.trial023.stdout: "mds_max_snaps_per_dir": "100", 2026-02-25T08:41:26.096 INFO:teuthology.orchestra.run.trial023.stdout: "mds_metrics_update_interval": "2", 2026-02-25T08:41:26.096 INFO:teuthology.orchestra.run.trial023.stdout: "mds_min_caps_per_client": "100", 2026-02-25T08:41:26.096 INFO:teuthology.orchestra.run.trial023.stdout: "mds_min_caps_working_set": "10000", 2026-02-25T08:41:26.096 INFO:teuthology.orchestra.run.trial023.stdout: "mds_mon_shutdown_timeout": "5.000000", 2026-02-25T08:41:26.097 INFO:teuthology.orchestra.run.trial023.stdout: "mds_numa_node": "-1", 2026-02-25T08:41:26.097 INFO:teuthology.orchestra.run.trial023.stdout: "mds_oft_prefetch_dirfrags": "false", 2026-02-25T08:41:26.097 INFO:teuthology.orchestra.run.trial023.stdout: "mds_op_complaint_time": "30.000000", 2026-02-25T08:41:26.097 INFO:teuthology.orchestra.run.trial023.stdout: "mds_op_history_duration": "600", 2026-02-25T08:41:26.097 INFO:teuthology.orchestra.run.trial023.stdout: "mds_op_history_size": "20", 2026-02-25T08:41:26.097 INFO:teuthology.orchestra.run.trial023.stdout: "mds_op_history_slow_op_size": "20", 2026-02-25T08:41:26.097 INFO:teuthology.orchestra.run.trial023.stdout: "mds_op_history_slow_op_threshold": "10", 2026-02-25T08:41:26.097 INFO:teuthology.orchestra.run.trial023.stdout: "mds_op_log_threshold": "5", 2026-02-25T08:41:26.097 INFO:teuthology.orchestra.run.trial023.stdout: "mds_ping_grace": "15", 2026-02-25T08:41:26.097 INFO:teuthology.orchestra.run.trial023.stdout: "mds_ping_interval": "5", 2026-02-25T08:41:26.097 INFO:teuthology.orchestra.run.trial023.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2026-02-25T08:41:26.097 INFO:teuthology.orchestra.run.trial023.stdout: "mds_recall_global_max_decay_threshold": "131072", 2026-02-25T08:41:26.097 INFO:teuthology.orchestra.run.trial023.stdout: "mds_recall_max_caps": "30000", 2026-02-25T08:41:26.097 INFO:teuthology.orchestra.run.trial023.stdout: "mds_recall_max_decay_rate": "1.500000", 2026-02-25T08:41:26.097 INFO:teuthology.orchestra.run.trial023.stdout: "mds_recall_max_decay_threshold": "131072", 2026-02-25T08:41:26.097 INFO:teuthology.orchestra.run.trial023.stdout: "mds_recall_warning_decay_rate": "60.000000", 2026-02-25T08:41:26.097 INFO:teuthology.orchestra.run.trial023.stdout: "mds_recall_warning_threshold": "262144", 2026-02-25T08:41:26.097 INFO:teuthology.orchestra.run.trial023.stdout: "mds_reconnect_timeout": "45.000000", 2026-02-25T08:41:26.097 INFO:teuthology.orchestra.run.trial023.stdout: "mds_replay_interval": "1.000000", 2026-02-25T08:41:26.097 INFO:teuthology.orchestra.run.trial023.stdout: "mds_replay_unsafe_with_closed_session": "false", 2026-02-25T08:41:26.097 INFO:teuthology.orchestra.run.trial023.stdout: "mds_request_load_average_decay_rate": "60.000000", 2026-02-25T08:41:26.097 INFO:teuthology.orchestra.run.trial023.stdout: "mds_root_ino_gid": "0", 2026-02-25T08:41:26.097 INFO:teuthology.orchestra.run.trial023.stdout: "mds_root_ino_uid": "0", 2026-02-25T08:41:26.097 INFO:teuthology.orchestra.run.trial023.stdout: "mds_scatter_nudge_interval": "5.000000", 2026-02-25T08:41:26.098 INFO:teuthology.orchestra.run.trial023.stdout: "mds_scrub_stats_review_period": "1", 2026-02-25T08:41:26.098 INFO:teuthology.orchestra.run.trial023.stdout: "mds_server_dispatch_client_request_delay": "0", 2026-02-25T08:41:26.098 INFO:teuthology.orchestra.run.trial023.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2026-02-25T08:41:26.098 INFO:teuthology.orchestra.run.trial023.stdout: "mds_session_blocklist_on_evict": "true", 2026-02-25T08:41:26.098 INFO:teuthology.orchestra.run.trial023.stdout: "mds_session_blocklist_on_timeout": "true", 2026-02-25T08:41:26.098 INFO:teuthology.orchestra.run.trial023.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2026-02-25T08:41:26.098 INFO:teuthology.orchestra.run.trial023.stdout: "mds_session_cache_liveness_magnitude": "10", 2026-02-25T08:41:26.098 INFO:teuthology.orchestra.run.trial023.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2026-02-25T08:41:26.098 INFO:teuthology.orchestra.run.trial023.stdout: "mds_session_cap_acquisition_throttle": "100000", 2026-02-25T08:41:26.098 INFO:teuthology.orchestra.run.trial023.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2026-02-25T08:41:26.098 INFO:teuthology.orchestra.run.trial023.stdout: "mds_session_metadata_threshold": "16777216", 2026-02-25T08:41:26.098 INFO:teuthology.orchestra.run.trial023.stdout: "mds_sessionmap_keys_per_op": "1024", 2026-02-25T08:41:26.098 INFO:teuthology.orchestra.run.trial023.stdout: "mds_shutdown_check": "0", 2026-02-25T08:41:26.098 INFO:teuthology.orchestra.run.trial023.stdout: "mds_skip_ino": "0", 2026-02-25T08:41:26.098 INFO:teuthology.orchestra.run.trial023.stdout: "mds_sleep_rank_change": "0.000000", 2026-02-25T08:41:26.098 INFO:teuthology.orchestra.run.trial023.stdout: "mds_snap_max_uid": "4294967294", 2026-02-25T08:41:26.098 INFO:teuthology.orchestra.run.trial023.stdout: "mds_snap_min_uid": "0", 2026-02-25T08:41:26.098 INFO:teuthology.orchestra.run.trial023.stdout: "mds_snap_rstat": "false", 2026-02-25T08:41:26.098 INFO:teuthology.orchestra.run.trial023.stdout: "mds_standby_replay_damaged": "false", 2026-02-25T08:41:26.098 INFO:teuthology.orchestra.run.trial023.stdout: "mds_symlink_recovery": "true", 2026-02-25T08:41:26.098 INFO:teuthology.orchestra.run.trial023.stdout: "mds_task_status_update_interval": "2.000000", 2026-02-25T08:41:26.098 INFO:teuthology.orchestra.run.trial023.stdout: "mds_thrash_exports": "0", 2026-02-25T08:41:26.098 INFO:teuthology.orchestra.run.trial023.stdout: "mds_thrash_fragments": "0", 2026-02-25T08:41:26.098 INFO:teuthology.orchestra.run.trial023.stdout: "mds_tick_interval": "5.000000", 2026-02-25T08:41:26.099 INFO:teuthology.orchestra.run.trial023.stdout: "mds_use_global_snaprealm_seq_for_subvol": "true", 2026-02-25T08:41:26.099 INFO:teuthology.orchestra.run.trial023.stdout: "mds_valgrind_exit": "false", 2026-02-25T08:41:26.099 INFO:teuthology.orchestra.run.trial023.stdout: "mds_verify_backtrace": "1", 2026-02-25T08:41:26.099 INFO:teuthology.orchestra.run.trial023.stdout: "mds_verify_scatter": "false", 2026-02-25T08:41:26.099 INFO:teuthology.orchestra.run.trial023.stdout: "mds_wipe_ino_prealloc": "false", 2026-02-25T08:41:26.099 INFO:teuthology.orchestra.run.trial023.stdout: "mds_wipe_sessions": "false", 2026-02-25T08:41:26.099 INFO:teuthology.orchestra.run.trial023.stdout: "mempool_debug": "false", 2026-02-25T08:41:26.099 INFO:teuthology.orchestra.run.trial023.stdout: "memstore_debug_omit_block_device_write": "false", 2026-02-25T08:41:26.099 INFO:teuthology.orchestra.run.trial023.stdout: "memstore_device_bytes": "1073741824", 2026-02-25T08:41:26.099 INFO:teuthology.orchestra.run.trial023.stdout: "memstore_page_set": "false", 2026-02-25T08:41:26.099 INFO:teuthology.orchestra.run.trial023.stdout: "memstore_page_size": "65536", 2026-02-25T08:41:26.099 INFO:teuthology.orchestra.run.trial023.stdout: "mgr_client_bytes": "134217728", 2026-02-25T08:41:26.099 INFO:teuthology.orchestra.run.trial023.stdout: "mgr_client_messages": "512", 2026-02-25T08:41:26.099 INFO:teuthology.orchestra.run.trial023.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2026-02-25T08:41:26.099 INFO:teuthology.orchestra.run.trial023.stdout: "mgr_connect_retry_interval": "1.000000", 2026-02-25T08:41:26.099 INFO:teuthology.orchestra.run.trial023.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2026-02-25T08:41:26.099 INFO:teuthology.orchestra.run.trial023.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2026-02-25T08:41:26.099 INFO:teuthology.orchestra.run.trial023.stdout: "mgr_disabled_modules": "", 2026-02-25T08:41:26.099 INFO:teuthology.orchestra.run.trial023.stdout: "mgr_enable_op_tracker": "true", 2026-02-25T08:41:26.099 INFO:teuthology.orchestra.run.trial023.stdout: "mgr_initial_modules": "iostat nfs", 2026-02-25T08:41:26.099 INFO:teuthology.orchestra.run.trial023.stdout: "mgr_max_pg_creating": "1024", 2026-02-25T08:41:26.099 INFO:teuthology.orchestra.run.trial023.stdout: "mgr_max_pg_num_change": "128", 2026-02-25T08:41:26.099 INFO:teuthology.orchestra.run.trial023.stdout: "mgr_mds_bytes": "134217728", 2026-02-25T08:41:26.099 INFO:teuthology.orchestra.run.trial023.stdout: "mgr_mds_messages": "128", 2026-02-25T08:41:26.099 INFO:teuthology.orchestra.run.trial023.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2026-02-25T08:41:26.099 INFO:teuthology.orchestra.run.trial023.stdout: "mgr_mon_bytes": "134217728", 2026-02-25T08:41:26.099 INFO:teuthology.orchestra.run.trial023.stdout: "mgr_mon_messages": "128", 2026-02-25T08:41:26.100 INFO:teuthology.orchestra.run.trial023.stdout: "mgr_num_op_tracker_shard": "32", 2026-02-25T08:41:26.100 INFO:teuthology.orchestra.run.trial023.stdout: "mgr_op_complaint_time": "30.000000", 2026-02-25T08:41:26.100 INFO:teuthology.orchestra.run.trial023.stdout: "mgr_op_history_duration": "600", 2026-02-25T08:41:26.100 INFO:teuthology.orchestra.run.trial023.stdout: "mgr_op_history_size": "20", 2026-02-25T08:41:26.100 INFO:teuthology.orchestra.run.trial023.stdout: "mgr_op_history_slow_op_size": "20", 2026-02-25T08:41:26.100 INFO:teuthology.orchestra.run.trial023.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2026-02-25T08:41:26.100 INFO:teuthology.orchestra.run.trial023.stdout: "mgr_op_log_threshold": "5", 2026-02-25T08:41:26.100 INFO:teuthology.orchestra.run.trial023.stdout: "mgr_osd_bytes": "536870912", 2026-02-25T08:41:26.100 INFO:teuthology.orchestra.run.trial023.stdout: "mgr_osd_messages": "8192", 2026-02-25T08:41:26.100 INFO:teuthology.orchestra.run.trial023.stdout: "mgr_pool": "true", 2026-02-25T08:41:26.100 INFO:teuthology.orchestra.run.trial023.stdout: "mgr_service_beacon_grace": "60.000000", 2026-02-25T08:41:26.100 INFO:teuthology.orchestra.run.trial023.stdout: "mgr_standby_modules": "true", 2026-02-25T08:41:26.100 INFO:teuthology.orchestra.run.trial023.stdout: "mgr_stats_period": "5", 2026-02-25T08:41:26.100 INFO:teuthology.orchestra.run.trial023.stdout: "mgr_stats_threshold": "5", 2026-02-25T08:41:26.100 INFO:teuthology.orchestra.run.trial023.stdout: "mgr_tick_period": "2", 2026-02-25T08:41:26.100 INFO:teuthology.orchestra.run.trial023.stdout: "mgr_ttl_cache_expire_seconds": "0", 2026-02-25T08:41:26.100 INFO:teuthology.orchestra.run.trial023.stdout: "mon_accept_timeout_factor": "2.000000", 2026-02-25T08:41:26.100 INFO:teuthology.orchestra.run.trial023.stdout: "mon_allow_pool_delete": "true", 2026-02-25T08:41:26.100 INFO:teuthology.orchestra.run.trial023.stdout: "mon_allow_pool_size_one": "true", 2026-02-25T08:41:26.100 INFO:teuthology.orchestra.run.trial023.stdout: "mon_auth_validate_all_caps": "true", 2026-02-25T08:41:26.100 INFO:teuthology.orchestra.run.trial023.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2026-02-25T08:41:26.100 INFO:teuthology.orchestra.run.trial023.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2026-02-25T08:41:26.100 INFO:teuthology.orchestra.run.trial023.stdout: "mon_client_bytes": "104857600", 2026-02-25T08:41:26.100 INFO:teuthology.orchestra.run.trial023.stdout: "mon_client_directed_command_retry": "2", 2026-02-25T08:41:26.100 INFO:teuthology.orchestra.run.trial023.stdout: "mon_client_hunt_interval": "3.000000", 2026-02-25T08:41:26.100 INFO:teuthology.orchestra.run.trial023.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2026-02-25T08:41:26.101 INFO:teuthology.orchestra.run.trial023.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2026-02-25T08:41:26.101 INFO:teuthology.orchestra.run.trial023.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2026-02-25T08:41:26.101 INFO:teuthology.orchestra.run.trial023.stdout: "mon_client_hunt_on_resend": "true", 2026-02-25T08:41:26.101 INFO:teuthology.orchestra.run.trial023.stdout: "mon_client_hunt_parallel": "3", 2026-02-25T08:41:26.101 INFO:teuthology.orchestra.run.trial023.stdout: "mon_client_log_interval": "1.000000", 2026-02-25T08:41:26.101 INFO:teuthology.orchestra.run.trial023.stdout: "mon_client_max_log_entries_per_message": "1000", 2026-02-25T08:41:26.101 INFO:teuthology.orchestra.run.trial023.stdout: "mon_client_ping_interval": "10.000000", 2026-02-25T08:41:26.101 INFO:teuthology.orchestra.run.trial023.stdout: "mon_client_ping_timeout": "30.000000", 2026-02-25T08:41:26.101 INFO:teuthology.orchestra.run.trial023.stdout: "mon_client_target_rank": "-1", 2026-02-25T08:41:26.101 INFO:teuthology.orchestra.run.trial023.stdout: "mon_clock_drift_allowed": "1.000000", 2026-02-25T08:41:26.101 INFO:teuthology.orchestra.run.trial023.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2026-02-25T08:41:26.101 INFO:teuthology.orchestra.run.trial023.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2026-02-25T08:41:26.101 INFO:teuthology.orchestra.run.trial023.stdout: "mon_cluster_log_level": "debug", 2026-02-25T08:41:26.101 INFO:teuthology.orchestra.run.trial023.stdout: "mon_cluster_log_to_file": "true", 2026-02-25T08:41:26.101 INFO:teuthology.orchestra.run.trial023.stdout: "mon_cluster_log_to_graylog": "false", 2026-02-25T08:41:26.101 INFO:teuthology.orchestra.run.trial023.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2026-02-25T08:41:26.101 INFO:teuthology.orchestra.run.trial023.stdout: "mon_cluster_log_to_graylog_port": "12201", 2026-02-25T08:41:26.101 INFO:teuthology.orchestra.run.trial023.stdout: "mon_cluster_log_to_journald": "false", 2026-02-25T08:41:26.101 INFO:teuthology.orchestra.run.trial023.stdout: "mon_cluster_log_to_stderr": "false", 2026-02-25T08:41:26.101 INFO:teuthology.orchestra.run.trial023.stdout: "mon_cluster_log_to_syslog": "default=false", 2026-02-25T08:41:26.101 INFO:teuthology.orchestra.run.trial023.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2026-02-25T08:41:26.101 INFO:teuthology.orchestra.run.trial023.stdout: "mon_compact_on_bootstrap": "false", 2026-02-25T08:41:26.101 INFO:teuthology.orchestra.run.trial023.stdout: "mon_compact_on_start": "false", 2026-02-25T08:41:26.101 INFO:teuthology.orchestra.run.trial023.stdout: "mon_compact_on_trim": "true", 2026-02-25T08:41:26.101 INFO:teuthology.orchestra.run.trial023.stdout: "mon_con_tracker_persist_interval": "10", 2026-02-25T08:41:26.101 INFO:teuthology.orchestra.run.trial023.stdout: "mon_con_tracker_score_halflife": "43200", 2026-02-25T08:41:26.101 INFO:teuthology.orchestra.run.trial023.stdout: "mon_config_key_max_entry_size": "65536", 2026-02-25T08:41:26.102 INFO:teuthology.orchestra.run.trial023.stdout: "mon_cpu_threads": "4", 2026-02-25T08:41:26.102 INFO:teuthology.orchestra.run.trial023.stdout: "mon_crush_min_required_version": "hammer", 2026-02-25T08:41:26.102 INFO:teuthology.orchestra.run.trial023.stdout: "mon_daemon_bytes": "419430400", 2026-02-25T08:41:26.102 INFO:teuthology.orchestra.run.trial023.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2026-02-25T08:41:26.102 INFO:teuthology.orchestra.run.trial023.stdout: "mon_data_avail_crit": "5", 2026-02-25T08:41:26.102 INFO:teuthology.orchestra.run.trial023.stdout: "mon_data_avail_warn": "5", 2026-02-25T08:41:26.102 INFO:teuthology.orchestra.run.trial023.stdout: "mon_data_size_warn": "16106127360", 2026-02-25T08:41:26.102 INFO:teuthology.orchestra.run.trial023.stdout: "mon_debug_block_osdmap_trim": "false", 2026-02-25T08:41:26.102 INFO:teuthology.orchestra.run.trial023.stdout: "mon_debug_deprecated_as_obsolete": "false", 2026-02-25T08:41:26.102 INFO:teuthology.orchestra.run.trial023.stdout: "mon_debug_dump_json": "false", 2026-02-25T08:41:26.102 INFO:teuthology.orchestra.run.trial023.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2026-02-25T08:41:26.102 INFO:teuthology.orchestra.run.trial023.stdout: "mon_debug_dump_transactions": "false", 2026-02-25T08:41:26.102 INFO:teuthology.orchestra.run.trial023.stdout: "mon_debug_extra_checks": "false", 2026-02-25T08:41:26.102 INFO:teuthology.orchestra.run.trial023.stdout: "mon_debug_no_initial_persistent_features": "false", 2026-02-25T08:41:26.102 INFO:teuthology.orchestra.run.trial023.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2026-02-25T08:41:26.102 INFO:teuthology.orchestra.run.trial023.stdout: "mon_debug_no_require_squid": "false", 2026-02-25T08:41:26.102 INFO:teuthology.orchestra.run.trial023.stdout: "mon_debug_no_require_tentacle": "false", 2026-02-25T08:41:26.102 INFO:teuthology.orchestra.run.trial023.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2026-02-25T08:41:26.102 INFO:teuthology.orchestra.run.trial023.stdout: "mon_delta_reset_interval": "10.000000", 2026-02-25T08:41:26.102 INFO:teuthology.orchestra.run.trial023.stdout: "mon_dns_srv_name": "ceph-mon", 2026-02-25T08:41:26.102 INFO:teuthology.orchestra.run.trial023.stdout: "mon_down_added_grace": "180", 2026-02-25T08:41:26.102 INFO:teuthology.orchestra.run.trial023.stdout: "mon_down_mkfs_grace": "120", 2026-02-25T08:41:26.102 INFO:teuthology.orchestra.run.trial023.stdout: "mon_down_uptime_grace": "60", 2026-02-25T08:41:26.102 INFO:teuthology.orchestra.run.trial023.stdout: "mon_election_default_strategy": "1", 2026-02-25T08:41:26.102 INFO:teuthology.orchestra.run.trial023.stdout: "mon_election_timeout": "5.000000", 2026-02-25T08:41:26.102 INFO:teuthology.orchestra.run.trial023.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2026-02-25T08:41:26.102 INFO:teuthology.orchestra.run.trial023.stdout: "mon_elector_ping_divisor": "2", 2026-02-25T08:41:26.102 INFO:teuthology.orchestra.run.trial023.stdout: "mon_elector_ping_timeout": "2.000000", 2026-02-25T08:41:26.103 INFO:teuthology.orchestra.run.trial023.stdout: "mon_enable_op_tracker": "true", 2026-02-25T08:41:26.103 INFO:teuthology.orchestra.run.trial023.stdout: "mon_fake_pool_delete": "false", 2026-02-25T08:41:26.103 INFO:teuthology.orchestra.run.trial023.stdout: "mon_force_quorum_join": "false", 2026-02-25T08:41:26.103 INFO:teuthology.orchestra.run.trial023.stdout: "mon_fsmap_prune_threshold": "300", 2026-02-25T08:41:26.103 INFO:teuthology.orchestra.run.trial023.stdout: "mon_globalid_prealloc": "10000", 2026-02-25T08:41:26.103 INFO:teuthology.orchestra.run.trial023.stdout: "mon_health_detail_to_clog": "false", 2026-02-25T08:41:26.103 INFO:teuthology.orchestra.run.trial023.stdout: "mon_health_log_update_period": "5", 2026-02-25T08:41:26.103 INFO:teuthology.orchestra.run.trial023.stdout: "mon_health_max_detail": "50", 2026-02-25T08:41:26.103 INFO:teuthology.orchestra.run.trial023.stdout: "mon_health_to_clog": "true", 2026-02-25T08:41:26.103 INFO:teuthology.orchestra.run.trial023.stdout: "mon_health_to_clog_interval": "600", 2026-02-25T08:41:26.103 INFO:teuthology.orchestra.run.trial023.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2026-02-25T08:41:26.103 INFO:teuthology.orchestra.run.trial023.stdout: "mon_host": "10.20.193.23", 2026-02-25T08:41:26.103 INFO:teuthology.orchestra.run.trial023.stdout: "mon_host_override": "", 2026-02-25T08:41:26.103 INFO:teuthology.orchestra.run.trial023.stdout: "mon_initial_members": "", 2026-02-25T08:41:26.103 INFO:teuthology.orchestra.run.trial023.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2026-02-25T08:41:26.103 INFO:teuthology.orchestra.run.trial023.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2026-02-25T08:41:26.103 INFO:teuthology.orchestra.run.trial023.stdout: "mon_inject_transaction_delay_max": "10.000000", 2026-02-25T08:41:26.103 INFO:teuthology.orchestra.run.trial023.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2026-02-25T08:41:26.103 INFO:teuthology.orchestra.run.trial023.stdout: "mon_keyvaluedb": "rocksdb", 2026-02-25T08:41:26.103 INFO:teuthology.orchestra.run.trial023.stdout: "mon_lease": "5.000000", 2026-02-25T08:41:26.103 INFO:teuthology.orchestra.run.trial023.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2026-02-25T08:41:26.103 INFO:teuthology.orchestra.run.trial023.stdout: "mon_lease_renew_interval_factor": "0.600000", 2026-02-25T08:41:26.103 INFO:teuthology.orchestra.run.trial023.stdout: "mon_log_full_interval": "50", 2026-02-25T08:41:26.103 INFO:teuthology.orchestra.run.trial023.stdout: "mon_log_max": "10000", 2026-02-25T08:41:26.103 INFO:teuthology.orchestra.run.trial023.stdout: "mon_log_max_summary": "50", 2026-02-25T08:41:26.103 INFO:teuthology.orchestra.run.trial023.stdout: "mon_max_log_entries_per_event": "4096", 2026-02-25T08:41:26.103 INFO:teuthology.orchestra.run.trial023.stdout: "mon_max_log_epochs": "500", 2026-02-25T08:41:26.103 INFO:teuthology.orchestra.run.trial023.stdout: "mon_max_mdsmap_epochs": "500", 2026-02-25T08:41:26.103 INFO:teuthology.orchestra.run.trial023.stdout: "mon_max_mgrmap_epochs": "500", 2026-02-25T08:41:26.104 INFO:teuthology.orchestra.run.trial023.stdout: "mon_max_nvmeof_epochs": "500", 2026-02-25T08:41:26.104 INFO:teuthology.orchestra.run.trial023.stdout: "mon_max_osd": "10000", 2026-02-25T08:41:26.104 INFO:teuthology.orchestra.run.trial023.stdout: "mon_max_pg_per_osd": "10000", 2026-02-25T08:41:26.104 INFO:teuthology.orchestra.run.trial023.stdout: "mon_max_pool_pg_num": "65536", 2026-02-25T08:41:26.104 INFO:teuthology.orchestra.run.trial023.stdout: "mon_max_snap_prune_per_epoch": "100", 2026-02-25T08:41:26.104 INFO:teuthology.orchestra.run.trial023.stdout: "mon_mds_blocklist_interval": "86400.000000", 2026-02-25T08:41:26.104 INFO:teuthology.orchestra.run.trial023.stdout: "mon_mds_force_trim_to": "0", 2026-02-25T08:41:26.104 INFO:teuthology.orchestra.run.trial023.stdout: "mon_mds_skip_sanity": "false", 2026-02-25T08:41:26.104 INFO:teuthology.orchestra.run.trial023.stdout: "mon_memory_autotune": "true", 2026-02-25T08:41:26.104 INFO:teuthology.orchestra.run.trial023.stdout: "mon_memory_target": "2147483648", 2026-02-25T08:41:26.104 INFO:teuthology.orchestra.run.trial023.stdout: "mon_mgr_beacon_grace": "30", 2026-02-25T08:41:26.104 INFO:teuthology.orchestra.run.trial023.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2026-02-25T08:41:26.104 INFO:teuthology.orchestra.run.trial023.stdout: "mon_mgr_digest_period": "5", 2026-02-25T08:41:26.104 INFO:teuthology.orchestra.run.trial023.stdout: "mon_mgr_inactive_grace": "60", 2026-02-25T08:41:26.104 INFO:teuthology.orchestra.run.trial023.stdout: "mon_mgr_mkfs_grace": "240", 2026-02-25T08:41:26.104 INFO:teuthology.orchestra.run.trial023.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2026-02-25T08:41:26.104 INFO:teuthology.orchestra.run.trial023.stdout: "mon_min_osdmap_epochs": "50", 2026-02-25T08:41:26.104 INFO:teuthology.orchestra.run.trial023.stdout: "mon_netsplit_grace_period": "9", 2026-02-25T08:41:26.104 INFO:teuthology.orchestra.run.trial023.stdout: "mon_nvmeofgw_beacon_grace": "7", 2026-02-25T08:41:26.104 INFO:teuthology.orchestra.run.trial023.stdout: "mon_nvmeofgw_beacons_till_ack": "15", 2026-02-25T08:41:26.104 INFO:teuthology.orchestra.run.trial023.stdout: "mon_nvmeofgw_delete_grace": "900", 2026-02-25T08:41:26.104 INFO:teuthology.orchestra.run.trial023.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2026-02-25T08:41:26.104 INFO:teuthology.orchestra.run.trial023.stdout: "mon_nvmeofgw_skip_failovers_interval": "16", 2026-02-25T08:41:26.104 INFO:teuthology.orchestra.run.trial023.stdout: "mon_nvmeofgw_wrong_map_ignore_sec": "15", 2026-02-25T08:41:26.104 INFO:teuthology.orchestra.run.trial023.stdout: "mon_op_complaint_time": "30", 2026-02-25T08:41:26.104 INFO:teuthology.orchestra.run.trial023.stdout: "mon_op_history_duration": "600", 2026-02-25T08:41:26.104 INFO:teuthology.orchestra.run.trial023.stdout: "mon_op_history_size": "20", 2026-02-25T08:41:26.104 INFO:teuthology.orchestra.run.trial023.stdout: "mon_op_history_slow_op_size": "20", 2026-02-25T08:41:26.105 INFO:teuthology.orchestra.run.trial023.stdout: "mon_op_history_slow_op_threshold": "10", 2026-02-25T08:41:26.105 INFO:teuthology.orchestra.run.trial023.stdout: "mon_op_log_threshold": "5", 2026-02-25T08:41:26.105 INFO:teuthology.orchestra.run.trial023.stdout: "mon_osd_adjust_down_out_interval": "true", 2026-02-25T08:41:26.105 INFO:teuthology.orchestra.run.trial023.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2026-02-25T08:41:26.105 INFO:teuthology.orchestra.run.trial023.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2026-02-25T08:41:26.105 INFO:teuthology.orchestra.run.trial023.stdout: "mon_osd_auto_mark_in": "false", 2026-02-25T08:41:26.105 INFO:teuthology.orchestra.run.trial023.stdout: "mon_osd_auto_mark_new_in": "true", 2026-02-25T08:41:26.105 INFO:teuthology.orchestra.run.trial023.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2026-02-25T08:41:26.105 INFO:teuthology.orchestra.run.trial023.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2026-02-25T08:41:26.105 INFO:teuthology.orchestra.run.trial023.stdout: "mon_osd_cache_size": "500", 2026-02-25T08:41:26.105 INFO:teuthology.orchestra.run.trial023.stdout: "mon_osd_cache_size_min": "134217728", 2026-02-25T08:41:26.105 INFO:teuthology.orchestra.run.trial023.stdout: "mon_osd_crush_smoke_test": "true", 2026-02-25T08:41:26.105 INFO:teuthology.orchestra.run.trial023.stdout: "mon_osd_destroyed_out_interval": "600", 2026-02-25T08:41:26.105 INFO:teuthology.orchestra.run.trial023.stdout: "mon_osd_down_out_interval": "600", 2026-02-25T08:41:26.105 INFO:teuthology.orchestra.run.trial023.stdout: "mon_osd_down_out_subtree_limit": "rack", 2026-02-25T08:41:26.105 INFO:teuthology.orchestra.run.trial023.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2026-02-25T08:41:26.105 INFO:teuthology.orchestra.run.trial023.stdout: "mon_osd_force_trim_to": "0", 2026-02-25T08:41:26.105 INFO:teuthology.orchestra.run.trial023.stdout: "mon_osd_full_ratio": "0.950000", 2026-02-25T08:41:26.105 INFO:teuthology.orchestra.run.trial023.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2026-02-25T08:41:26.105 INFO:teuthology.orchestra.run.trial023.stdout: "mon_osd_laggy_halflife": "3600", 2026-02-25T08:41:26.105 INFO:teuthology.orchestra.run.trial023.stdout: "mon_osd_laggy_max_interval": "300", 2026-02-25T08:41:26.105 INFO:teuthology.orchestra.run.trial023.stdout: "mon_osd_laggy_weight": "0.300000", 2026-02-25T08:41:26.105 INFO:teuthology.orchestra.run.trial023.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2026-02-25T08:41:26.105 INFO:teuthology.orchestra.run.trial023.stdout: "mon_osd_max_initial_pgs": "1024", 2026-02-25T08:41:26.105 INFO:teuthology.orchestra.run.trial023.stdout: "mon_osd_min_down_reporters": "2", 2026-02-25T08:41:26.105 INFO:teuthology.orchestra.run.trial023.stdout: "mon_osd_min_in_ratio": "0.750000", 2026-02-25T08:41:26.105 INFO:teuthology.orchestra.run.trial023.stdout: "mon_osd_min_up_ratio": "0.300000", 2026-02-25T08:41:26.105 INFO:teuthology.orchestra.run.trial023.stdout: "mon_osd_nearfull_ratio": "0.850000", 2026-02-25T08:41:26.106 INFO:teuthology.orchestra.run.trial023.stdout: "mon_osd_prime_pg_temp": "true", 2026-02-25T08:41:26.106 INFO:teuthology.orchestra.run.trial023.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2026-02-25T08:41:26.106 INFO:teuthology.orchestra.run.trial023.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2026-02-25T08:41:26.106 INFO:teuthology.orchestra.run.trial023.stdout: "mon_osd_report_timeout": "900", 2026-02-25T08:41:26.106 INFO:teuthology.orchestra.run.trial023.stdout: "mon_osd_reporter_subtree_level": "osd", 2026-02-25T08:41:26.106 INFO:teuthology.orchestra.run.trial023.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2026-02-25T08:41:26.106 INFO:teuthology.orchestra.run.trial023.stdout: "mon_osd_warn_num_repaired": "10", 2026-02-25T08:41:26.106 INFO:teuthology.orchestra.run.trial023.stdout: "mon_osd_warn_op_age": "32.000000", 2026-02-25T08:41:26.106 INFO:teuthology.orchestra.run.trial023.stdout: "mon_osdmap_full_prune_enabled": "true", 2026-02-25T08:41:26.106 INFO:teuthology.orchestra.run.trial023.stdout: "mon_osdmap_full_prune_interval": "2", 2026-02-25T08:41:26.106 INFO:teuthology.orchestra.run.trial023.stdout: "mon_osdmap_full_prune_min": "15", 2026-02-25T08:41:26.106 INFO:teuthology.orchestra.run.trial023.stdout: "mon_osdmap_full_prune_txsize": "2", 2026-02-25T08:41:26.106 INFO:teuthology.orchestra.run.trial023.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2026-02-25T08:41:26.106 INFO:teuthology.orchestra.run.trial023.stdout: "mon_pg_stuck_threshold": "60", 2026-02-25T08:41:26.106 INFO:teuthology.orchestra.run.trial023.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2026-02-25T08:41:26.106 INFO:teuthology.orchestra.run.trial023.stdout: "mon_pg_warn_min_objects": "10000", 2026-02-25T08:41:26.106 INFO:teuthology.orchestra.run.trial023.stdout: "mon_pg_warn_min_per_osd": "0", 2026-02-25T08:41:26.106 INFO:teuthology.orchestra.run.trial023.stdout: "mon_pg_warn_min_pool_objects": "1000", 2026-02-25T08:41:26.106 INFO:teuthology.orchestra.run.trial023.stdout: "mon_pool_quota_crit_threshold": "0", 2026-02-25T08:41:26.106 INFO:teuthology.orchestra.run.trial023.stdout: "mon_pool_quota_warn_threshold": "0", 2026-02-25T08:41:26.106 INFO:teuthology.orchestra.run.trial023.stdout: "mon_probe_timeout": "2.000000", 2026-02-25T08:41:26.106 INFO:teuthology.orchestra.run.trial023.stdout: "mon_reweight_max_change": "0.050000", 2026-02-25T08:41:26.106 INFO:teuthology.orchestra.run.trial023.stdout: "mon_reweight_max_osds": "4", 2026-02-25T08:41:26.106 INFO:teuthology.orchestra.run.trial023.stdout: "mon_reweight_min_bytes_per_osd": "10", 2026-02-25T08:41:26.106 INFO:teuthology.orchestra.run.trial023.stdout: "mon_reweight_min_pgs_per_osd": "4", 2026-02-25T08:41:26.106 INFO:teuthology.orchestra.run.trial023.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2026-02-25T08:41:26.106 INFO:teuthology.orchestra.run.trial023.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2026-02-25T08:41:26.106 INFO:teuthology.orchestra.run.trial023.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2026-02-25T08:41:26.107 INFO:teuthology.orchestra.run.trial023.stdout: "mon_scrub_interval": "86400", 2026-02-25T08:41:26.107 INFO:teuthology.orchestra.run.trial023.stdout: "mon_scrub_max_keys": "100", 2026-02-25T08:41:26.107 INFO:teuthology.orchestra.run.trial023.stdout: "mon_scrub_timeout": "300", 2026-02-25T08:41:26.107 INFO:teuthology.orchestra.run.trial023.stdout: "mon_session_timeout": "300", 2026-02-25T08:41:26.107 INFO:teuthology.orchestra.run.trial023.stdout: "mon_smart_report_timeout": "5", 2026-02-25T08:41:26.107 INFO:teuthology.orchestra.run.trial023.stdout: "mon_stat_smooth_intervals": "6", 2026-02-25T08:41:26.107 INFO:teuthology.orchestra.run.trial023.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2026-02-25T08:41:26.107 INFO:teuthology.orchestra.run.trial023.stdout: "mon_stretch_pool_min_size": "2", 2026-02-25T08:41:26.107 INFO:teuthology.orchestra.run.trial023.stdout: "mon_stretch_pool_size": "4", 2026-02-25T08:41:26.107 INFO:teuthology.orchestra.run.trial023.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2026-02-25T08:41:26.107 INFO:teuthology.orchestra.run.trial023.stdout: "mon_subscribe_interval": "86400.000000", 2026-02-25T08:41:26.107 INFO:teuthology.orchestra.run.trial023.stdout: "mon_sync_debug": "false", 2026-02-25T08:41:26.107 INFO:teuthology.orchestra.run.trial023.stdout: "mon_sync_max_payload_keys": "2000", 2026-02-25T08:41:26.107 INFO:teuthology.orchestra.run.trial023.stdout: "mon_sync_max_payload_size": "1048576", 2026-02-25T08:41:26.107 INFO:teuthology.orchestra.run.trial023.stdout: "mon_sync_provider_kill_at": "0", 2026-02-25T08:41:26.107 INFO:teuthology.orchestra.run.trial023.stdout: "mon_sync_requester_kill_at": "0", 2026-02-25T08:41:26.107 INFO:teuthology.orchestra.run.trial023.stdout: "mon_sync_timeout": "60.000000", 2026-02-25T08:41:26.107 INFO:teuthology.orchestra.run.trial023.stdout: "mon_target_pg_per_osd": "100", 2026-02-25T08:41:26.107 INFO:teuthology.orchestra.run.trial023.stdout: "mon_tick_interval": "5", 2026-02-25T08:41:26.107 INFO:teuthology.orchestra.run.trial023.stdout: "mon_timecheck_interval": "300.000000", 2026-02-25T08:41:26.107 INFO:teuthology.orchestra.run.trial023.stdout: "mon_timecheck_skew_interval": "30.000000", 2026-02-25T08:41:26.107 INFO:teuthology.orchestra.run.trial023.stdout: "mon_use_min_delay_socket": "false", 2026-02-25T08:41:26.107 INFO:teuthology.orchestra.run.trial023.stdout: "mon_warn_older_version_delay": "604800", 2026-02-25T08:41:26.107 INFO:teuthology.orchestra.run.trial023.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2026-02-25T08:41:26.107 INFO:teuthology.orchestra.run.trial023.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2026-02-25T08:41:26.107 INFO:teuthology.orchestra.run.trial023.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2026-02-25T08:41:26.107 INFO:teuthology.orchestra.run.trial023.stdout: "mon_warn_on_filestore_osds": "false", 2026-02-25T08:41:26.107 INFO:teuthology.orchestra.run.trial023.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2026-02-25T08:41:26.108 INFO:teuthology.orchestra.run.trial023.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2026-02-25T08:41:26.108 INFO:teuthology.orchestra.run.trial023.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2026-02-25T08:41:26.108 INFO:teuthology.orchestra.run.trial023.stdout: "mon_warn_on_misplaced": "false", 2026-02-25T08:41:26.108 INFO:teuthology.orchestra.run.trial023.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2026-02-25T08:41:26.108 INFO:teuthology.orchestra.run.trial023.stdout: "mon_warn_on_older_version": "true", 2026-02-25T08:41:26.108 INFO:teuthology.orchestra.run.trial023.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2026-02-25T08:41:26.108 INFO:teuthology.orchestra.run.trial023.stdout: "mon_warn_on_pool_no_app": "true", 2026-02-25T08:41:26.108 INFO:teuthology.orchestra.run.trial023.stdout: "mon_warn_on_pool_no_app_grace": "300", 2026-02-25T08:41:26.108 INFO:teuthology.orchestra.run.trial023.stdout: "mon_warn_on_pool_no_redundancy": "false", 2026-02-25T08:41:26.108 INFO:teuthology.orchestra.run.trial023.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2026-02-25T08:41:26.108 INFO:teuthology.orchestra.run.trial023.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2026-02-25T08:41:26.108 INFO:teuthology.orchestra.run.trial023.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2026-02-25T08:41:26.108 INFO:teuthology.orchestra.run.trial023.stdout: "mon_warn_on_too_few_osds": "false", 2026-02-25T08:41:26.108 INFO:teuthology.orchestra.run.trial023.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2026-02-25T08:41:26.108 INFO:teuthology.orchestra.run.trial023.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2026-02-25T08:41:26.108 INFO:teuthology.orchestra.run.trial023.stdout: "monmap": "", 2026-02-25T08:41:26.108 INFO:teuthology.orchestra.run.trial023.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2026-02-25T08:41:26.108 INFO:teuthology.orchestra.run.trial023.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2026-02-25T08:41:26.108 INFO:teuthology.orchestra.run.trial023.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2026-02-25T08:41:26.108 INFO:teuthology.orchestra.run.trial023.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2026-02-25T08:41:26.108 INFO:teuthology.orchestra.run.trial023.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2026-02-25T08:41:26.108 INFO:teuthology.orchestra.run.trial023.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2026-02-25T08:41:26.108 INFO:teuthology.orchestra.run.trial023.stdout: "motr_tracing_enabled": "false", 2026-02-25T08:41:26.108 INFO:teuthology.orchestra.run.trial023.stdout: "ms_async_op_threads": "3", 2026-02-25T08:41:26.108 INFO:teuthology.orchestra.run.trial023.stdout: "ms_async_rdma_buffer_size": "131072", 2026-02-25T08:41:26.108 INFO:teuthology.orchestra.run.trial023.stdout: "ms_async_rdma_cm": "false", 2026-02-25T08:41:26.108 INFO:teuthology.orchestra.run.trial023.stdout: "ms_async_rdma_device_name": "", 2026-02-25T08:41:26.108 INFO:teuthology.orchestra.run.trial023.stdout: "ms_async_rdma_dscp": "96", 2026-02-25T08:41:26.108 INFO:teuthology.orchestra.run.trial023.stdout: "ms_async_rdma_enable_hugepage": "false", 2026-02-25T08:41:26.109 INFO:teuthology.orchestra.run.trial023.stdout: "ms_async_rdma_gid_idx": "0", 2026-02-25T08:41:26.109 INFO:teuthology.orchestra.run.trial023.stdout: "ms_async_rdma_local_gid": "", 2026-02-25T08:41:26.109 INFO:teuthology.orchestra.run.trial023.stdout: "ms_async_rdma_polling_us": "1000", 2026-02-25T08:41:26.109 INFO:teuthology.orchestra.run.trial023.stdout: "ms_async_rdma_port_num": "1", 2026-02-25T08:41:26.109 INFO:teuthology.orchestra.run.trial023.stdout: "ms_async_rdma_receive_buffers": "32768", 2026-02-25T08:41:26.109 INFO:teuthology.orchestra.run.trial023.stdout: "ms_async_rdma_receive_queue_len": "4096", 2026-02-25T08:41:26.109 INFO:teuthology.orchestra.run.trial023.stdout: "ms_async_rdma_roce_ver": "1", 2026-02-25T08:41:26.109 INFO:teuthology.orchestra.run.trial023.stdout: "ms_async_rdma_send_buffers": "1024", 2026-02-25T08:41:26.109 INFO:teuthology.orchestra.run.trial023.stdout: "ms_async_rdma_sl": "3", 2026-02-25T08:41:26.109 INFO:teuthology.orchestra.run.trial023.stdout: "ms_async_rdma_support_srq": "true", 2026-02-25T08:41:26.109 INFO:teuthology.orchestra.run.trial023.stdout: "ms_async_rdma_type": "ib", 2026-02-25T08:41:26.109 INFO:teuthology.orchestra.run.trial023.stdout: "ms_async_reap_threshold": "5", 2026-02-25T08:41:26.109 INFO:teuthology.orchestra.run.trial023.stdout: "ms_bind_before_connect": "false", 2026-02-25T08:41:26.109 INFO:teuthology.orchestra.run.trial023.stdout: "ms_bind_ipv4": "true", 2026-02-25T08:41:26.109 INFO:teuthology.orchestra.run.trial023.stdout: "ms_bind_ipv6": "false", 2026-02-25T08:41:26.109 INFO:teuthology.orchestra.run.trial023.stdout: "ms_bind_msgr1": "true", 2026-02-25T08:41:26.109 INFO:teuthology.orchestra.run.trial023.stdout: "ms_bind_msgr2": "true", 2026-02-25T08:41:26.109 INFO:teuthology.orchestra.run.trial023.stdout: "ms_bind_port_max": "7568", 2026-02-25T08:41:26.109 INFO:teuthology.orchestra.run.trial023.stdout: "ms_bind_port_min": "6800", 2026-02-25T08:41:26.109 INFO:teuthology.orchestra.run.trial023.stdout: "ms_bind_prefer_ipv4": "false", 2026-02-25T08:41:26.109 INFO:teuthology.orchestra.run.trial023.stdout: "ms_bind_retry_count": "3", 2026-02-25T08:41:26.109 INFO:teuthology.orchestra.run.trial023.stdout: "ms_bind_retry_delay": "5", 2026-02-25T08:41:26.109 INFO:teuthology.orchestra.run.trial023.stdout: "ms_blackhole_client": "false", 2026-02-25T08:41:26.109 INFO:teuthology.orchestra.run.trial023.stdout: "ms_blackhole_mds": "false", 2026-02-25T08:41:26.109 INFO:teuthology.orchestra.run.trial023.stdout: "ms_blackhole_mgr": "false", 2026-02-25T08:41:26.109 INFO:teuthology.orchestra.run.trial023.stdout: "ms_blackhole_mon": "false", 2026-02-25T08:41:26.109 INFO:teuthology.orchestra.run.trial023.stdout: "ms_blackhole_osd": "false", 2026-02-25T08:41:26.109 INFO:teuthology.orchestra.run.trial023.stdout: "ms_client_mode": "crc", 2026-02-25T08:41:26.109 INFO:teuthology.orchestra.run.trial023.stdout: "ms_client_throttle_retry_time_interval": "5000", 2026-02-25T08:41:26.109 INFO:teuthology.orchestra.run.trial023.stdout: "ms_cluster_mode": "crc", 2026-02-25T08:41:26.110 INFO:teuthology.orchestra.run.trial023.stdout: "ms_cluster_type": "", 2026-02-25T08:41:26.110 INFO:teuthology.orchestra.run.trial023.stdout: "ms_compress_secure": "false", 2026-02-25T08:41:26.110 INFO:teuthology.orchestra.run.trial023.stdout: "ms_connection_idle_timeout": "900", 2026-02-25T08:41:26.110 INFO:teuthology.orchestra.run.trial023.stdout: "ms_connection_ready_timeout": "10", 2026-02-25T08:41:26.110 INFO:teuthology.orchestra.run.trial023.stdout: "ms_crc_data": "true", 2026-02-25T08:41:26.110 INFO:teuthology.orchestra.run.trial023.stdout: "ms_crc_header": "true", 2026-02-25T08:41:26.110 INFO:teuthology.orchestra.run.trial023.stdout: "ms_die_on_bad_msg": "false", 2026-02-25T08:41:26.110 INFO:teuthology.orchestra.run.trial023.stdout: "ms_die_on_bug": "true", 2026-02-25T08:41:26.110 INFO:teuthology.orchestra.run.trial023.stdout: "ms_die_on_old_message": "true", 2026-02-25T08:41:26.110 INFO:teuthology.orchestra.run.trial023.stdout: "ms_die_on_skipped_message": "false", 2026-02-25T08:41:26.110 INFO:teuthology.orchestra.run.trial023.stdout: "ms_die_on_unhandled_msg": "false", 2026-02-25T08:41:26.110 INFO:teuthology.orchestra.run.trial023.stdout: "ms_dispatch_throttle_bytes": "104857600", 2026-02-25T08:41:26.110 INFO:teuthology.orchestra.run.trial023.stdout: "ms_dpdk_coremask": "0xF", 2026-02-25T08:41:26.110 INFO:teuthology.orchestra.run.trial023.stdout: "ms_dpdk_debug_allow_loopback": "false", 2026-02-25T08:41:26.110 INFO:teuthology.orchestra.run.trial023.stdout: "ms_dpdk_devs_allowlist": "", 2026-02-25T08:41:26.110 INFO:teuthology.orchestra.run.trial023.stdout: "ms_dpdk_enable_tso": "true", 2026-02-25T08:41:26.110 INFO:teuthology.orchestra.run.trial023.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2026-02-25T08:41:26.110 INFO:teuthology.orchestra.run.trial023.stdout: "ms_dpdk_host_ipv4_addr": "", 2026-02-25T08:41:26.110 INFO:teuthology.orchestra.run.trial023.stdout: "ms_dpdk_hugepages": "", 2026-02-25T08:41:26.110 INFO:teuthology.orchestra.run.trial023.stdout: "ms_dpdk_hw_flow_control": "true", 2026-02-25T08:41:26.110 INFO:teuthology.orchestra.run.trial023.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2026-02-25T08:41:26.110 INFO:teuthology.orchestra.run.trial023.stdout: "ms_dpdk_lro": "true", 2026-02-25T08:41:26.110 INFO:teuthology.orchestra.run.trial023.stdout: "ms_dpdk_memory_channel": "4", 2026-02-25T08:41:26.110 INFO:teuthology.orchestra.run.trial023.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2026-02-25T08:41:26.110 INFO:teuthology.orchestra.run.trial023.stdout: "ms_dpdk_pmd": "", 2026-02-25T08:41:26.110 INFO:teuthology.orchestra.run.trial023.stdout: "ms_dpdk_port_id": "0", 2026-02-25T08:41:26.110 INFO:teuthology.orchestra.run.trial023.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2026-02-25T08:41:26.110 INFO:teuthology.orchestra.run.trial023.stdout: "ms_dump_corrupt_message_level": "1", 2026-02-25T08:41:26.110 INFO:teuthology.orchestra.run.trial023.stdout: "ms_dump_on_send": "false", 2026-02-25T08:41:26.110 INFO:teuthology.orchestra.run.trial023.stdout: "ms_initial_backoff": "0.200000", 2026-02-25T08:41:26.110 INFO:teuthology.orchestra.run.trial023.stdout: "ms_inject_delay_max": "1.000000", 2026-02-25T08:41:26.111 INFO:teuthology.orchestra.run.trial023.stdout: "ms_inject_delay_probability": "0.000000", 2026-02-25T08:41:26.111 INFO:teuthology.orchestra.run.trial023.stdout: "ms_inject_delay_type": "", 2026-02-25T08:41:26.111 INFO:teuthology.orchestra.run.trial023.stdout: "ms_inject_internal_delays": "0.000000", 2026-02-25T08:41:26.111 INFO:teuthology.orchestra.run.trial023.stdout: "ms_inject_network_congestion": "0", 2026-02-25T08:41:26.111 INFO:teuthology.orchestra.run.trial023.stdout: "ms_inject_socket_failures": "0", 2026-02-25T08:41:26.111 INFO:teuthology.orchestra.run.trial023.stdout: "ms_learn_addr_from_peer": "true", 2026-02-25T08:41:26.111 INFO:teuthology.orchestra.run.trial023.stdout: "ms_max_accept_failures": "4", 2026-02-25T08:41:26.111 INFO:teuthology.orchestra.run.trial023.stdout: "ms_max_backoff": "15.000000", 2026-02-25T08:41:26.111 INFO:teuthology.orchestra.run.trial023.stdout: "ms_mon_client_mode": "crc", 2026-02-25T08:41:26.111 INFO:teuthology.orchestra.run.trial023.stdout: "ms_mon_cluster_mode": "crc", 2026-02-25T08:41:26.111 INFO:teuthology.orchestra.run.trial023.stdout: "ms_mon_service_mode": "crc", 2026-02-25T08:41:26.111 INFO:teuthology.orchestra.run.trial023.stdout: "ms_osd_compress_min_size": "1024", 2026-02-25T08:41:26.111 INFO:teuthology.orchestra.run.trial023.stdout: "ms_osd_compress_mode": "none", 2026-02-25T08:41:26.111 INFO:teuthology.orchestra.run.trial023.stdout: "ms_osd_compression_algorithm": "snappy", 2026-02-25T08:41:26.111 INFO:teuthology.orchestra.run.trial023.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2026-02-25T08:41:26.111 INFO:teuthology.orchestra.run.trial023.stdout: "ms_pq_min_cost": "65536", 2026-02-25T08:41:26.111 INFO:teuthology.orchestra.run.trial023.stdout: "ms_public_type": "", 2026-02-25T08:41:26.111 INFO:teuthology.orchestra.run.trial023.stdout: "ms_service_mode": "crc", 2026-02-25T08:41:26.111 INFO:teuthology.orchestra.run.trial023.stdout: "ms_tcp_listen_backlog": "512", 2026-02-25T08:41:26.111 INFO:teuthology.orchestra.run.trial023.stdout: "ms_tcp_nodelay": "true", 2026-02-25T08:41:26.111 INFO:teuthology.orchestra.run.trial023.stdout: "ms_tcp_prefetch_max_size": "65536", 2026-02-25T08:41:26.111 INFO:teuthology.orchestra.run.trial023.stdout: "ms_tcp_rcvbuf": "0", 2026-02-25T08:41:26.111 INFO:teuthology.orchestra.run.trial023.stdout: "ms_time_events_min_wait_interval": "1000", 2026-02-25T08:41:26.111 INFO:teuthology.orchestra.run.trial023.stdout: "ms_type": "async+posix", 2026-02-25T08:41:26.111 INFO:teuthology.orchestra.run.trial023.stdout: "no_config_file": "false", 2026-02-25T08:41:26.111 INFO:teuthology.orchestra.run.trial023.stdout: "nvmeof_mon_client_connect_panic": "30", 2026-02-25T08:41:26.111 INFO:teuthology.orchestra.run.trial023.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2026-02-25T08:41:26.111 INFO:teuthology.orchestra.run.trial023.stdout: "nvmeof_mon_client_tick_period": "1", 2026-02-25T08:41:26.111 INFO:teuthology.orchestra.run.trial023.stdout: "objecter_completion_locks_per_session": "32", 2026-02-25T08:41:26.111 INFO:teuthology.orchestra.run.trial023.stdout: "objecter_debug_inject_relock_delay": "false", 2026-02-25T08:41:26.111 INFO:teuthology.orchestra.run.trial023.stdout: "objecter_inflight_op_bytes": "104857600", 2026-02-25T08:41:26.112 INFO:teuthology.orchestra.run.trial023.stdout: "objecter_inflight_ops": "1024", 2026-02-25T08:41:26.112 INFO:teuthology.orchestra.run.trial023.stdout: "objecter_inject_no_watch_ping": "false", 2026-02-25T08:41:26.112 INFO:teuthology.orchestra.run.trial023.stdout: "objecter_retry_writes_after_first_reply": "false", 2026-02-25T08:41:26.112 INFO:teuthology.orchestra.run.trial023.stdout: "objecter_tick_interval": "5.000000", 2026-02-25T08:41:26.112 INFO:teuthology.orchestra.run.trial023.stdout: "objecter_timeout": "10.000000", 2026-02-25T08:41:26.112 INFO:teuthology.orchestra.run.trial023.stdout: "objectstore_blackhole": "false", 2026-02-25T08:41:26.112 INFO:teuthology.orchestra.run.trial023.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2026-02-25T08:41:26.112 INFO:teuthology.orchestra.run.trial023.stdout: "openssl_conf": "", 2026-02-25T08:41:26.112 INFO:teuthology.orchestra.run.trial023.stdout: "osd_agent_delay_time": "5.000000", 2026-02-25T08:41:26.112 INFO:teuthology.orchestra.run.trial023.stdout: "osd_agent_hist_halflife": "1000", 2026-02-25T08:41:26.112 INFO:teuthology.orchestra.run.trial023.stdout: "osd_agent_max_low_ops": "2", 2026-02-25T08:41:26.112 INFO:teuthology.orchestra.run.trial023.stdout: "osd_agent_max_ops": "4", 2026-02-25T08:41:26.112 INFO:teuthology.orchestra.run.trial023.stdout: "osd_agent_min_evict_effort": "0.100000", 2026-02-25T08:41:26.112 INFO:teuthology.orchestra.run.trial023.stdout: "osd_agent_quantize_effort": "0.100000", 2026-02-25T08:41:26.112 INFO:teuthology.orchestra.run.trial023.stdout: "osd_agent_slop": "0.020000", 2026-02-25T08:41:26.112 INFO:teuthology.orchestra.run.trial023.stdout: "osd_aggregated_slow_ops_logging": "true", 2026-02-25T08:41:26.112 INFO:teuthology.orchestra.run.trial023.stdout: "osd_allow_recovery_below_min_size": "true", 2026-02-25T08:41:26.112 INFO:teuthology.orchestra.run.trial023.stdout: "osd_asio_thread_count": "2", 2026-02-25T08:41:26.112 INFO:teuthology.orchestra.run.trial023.stdout: "osd_async_recovery_min_cost": "100", 2026-02-25T08:41:26.112 INFO:teuthology.orchestra.run.trial023.stdout: "osd_auto_mark_unfound_lost": "false", 2026-02-25T08:41:26.112 INFO:teuthology.orchestra.run.trial023.stdout: "osd_backfill_retry_interval": "30.000000", 2026-02-25T08:41:26.112 INFO:teuthology.orchestra.run.trial023.stdout: "osd_backfill_scan_max": "512", 2026-02-25T08:41:26.112 INFO:teuthology.orchestra.run.trial023.stdout: "osd_backfill_scan_min": "64", 2026-02-25T08:41:26.112 INFO:teuthology.orchestra.run.trial023.stdout: "osd_backoff_on_degraded": "false", 2026-02-25T08:41:26.112 INFO:teuthology.orchestra.run.trial023.stdout: "osd_backoff_on_peering": "false", 2026-02-25T08:41:26.112 INFO:teuthology.orchestra.run.trial023.stdout: "osd_backoff_on_unfound": "true", 2026-02-25T08:41:26.112 INFO:teuthology.orchestra.run.trial023.stdout: "osd_beacon_report_interval": "300", 2026-02-25T08:41:26.112 INFO:teuthology.orchestra.run.trial023.stdout: "osd_bench_duration": "30", 2026-02-25T08:41:26.112 INFO:teuthology.orchestra.run.trial023.stdout: "osd_bench_large_size_max_throughput": "104857600", 2026-02-25T08:41:26.112 INFO:teuthology.orchestra.run.trial023.stdout: "osd_bench_max_block_size": "67108864", 2026-02-25T08:41:26.112 INFO:teuthology.orchestra.run.trial023.stdout: "osd_bench_small_size_max_iops": "100", 2026-02-25T08:41:26.112 INFO:teuthology.orchestra.run.trial023.stdout: "osd_blkin_trace_all": "false", 2026-02-25T08:41:26.113 INFO:teuthology.orchestra.run.trial023.stdout: "osd_blocked_scrub_grace_period": "120", 2026-02-25T08:41:26.113 INFO:teuthology.orchestra.run.trial023.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2026-02-25T08:41:26.113 INFO:teuthology.orchestra.run.trial023.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2026-02-25T08:41:26.113 INFO:teuthology.orchestra.run.trial023.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2026-02-25T08:41:26.113 INFO:teuthology.orchestra.run.trial023.stdout: "osd_check_for_log_corruption": "false", 2026-02-25T08:41:26.113 INFO:teuthology.orchestra.run.trial023.stdout: "osd_check_max_object_name_len_on_startup": "true", 2026-02-25T08:41:26.113 INFO:teuthology.orchestra.run.trial023.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo sem_set", 2026-02-25T08:41:26.113 INFO:teuthology.orchestra.run.trial023.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2026-02-25T08:41:26.113 INFO:teuthology.orchestra.run.trial023.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo sem_set", 2026-02-25T08:41:26.113 INFO:teuthology.orchestra.run.trial023.stdout: "osd_class_update_on_start": "true", 2026-02-25T08:41:26.113 INFO:teuthology.orchestra.run.trial023.stdout: "osd_client_message_cap": "256", 2026-02-25T08:41:26.113 INFO:teuthology.orchestra.run.trial023.stdout: "osd_client_message_size_cap": "524288000", 2026-02-25T08:41:26.113 INFO:teuthology.orchestra.run.trial023.stdout: "osd_client_op_priority": "63", 2026-02-25T08:41:26.113 INFO:teuthology.orchestra.run.trial023.stdout: "osd_client_watch_timeout": "30", 2026-02-25T08:41:26.113 INFO:teuthology.orchestra.run.trial023.stdout: "osd_command_max_records": "256", 2026-02-25T08:41:26.113 INFO:teuthology.orchestra.run.trial023.stdout: "osd_compact_on_start": "false", 2026-02-25T08:41:26.113 INFO:teuthology.orchestra.run.trial023.stdout: "osd_copyfrom_max_chunk": "8388608", 2026-02-25T08:41:26.113 INFO:teuthology.orchestra.run.trial023.stdout: "osd_crush_chooseleaf_type": "0", 2026-02-25T08:41:26.113 INFO:teuthology.orchestra.run.trial023.stdout: "osd_crush_initial_weight": "-1.000000", 2026-02-25T08:41:26.113 INFO:teuthology.orchestra.run.trial023.stdout: "osd_crush_update_on_start": "true", 2026-02-25T08:41:26.113 INFO:teuthology.orchestra.run.trial023.stdout: "osd_crush_update_weight_set": "true", 2026-02-25T08:41:26.113 INFO:teuthology.orchestra.run.trial023.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2026-02-25T08:41:26.113 INFO:teuthology.orchestra.run.trial023.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2026-02-25T08:41:26.113 INFO:teuthology.orchestra.run.trial023.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2026-02-25T08:41:26.113 INFO:teuthology.orchestra.run.trial023.stdout: "osd_debug_drop_ping_duration": "0", 2026-02-25T08:41:26.113 INFO:teuthology.orchestra.run.trial023.stdout: "osd_debug_drop_ping_probability": "0.000000", 2026-02-25T08:41:26.113 INFO:teuthology.orchestra.run.trial023.stdout: "osd_debug_feed_pullee": "-1", 2026-02-25T08:41:26.113 INFO:teuthology.orchestra.run.trial023.stdout: "osd_debug_inject_copyfrom_error": "false", 2026-02-25T08:41:26.113 INFO:teuthology.orchestra.run.trial023.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2026-02-25T08:41:26.113 INFO:teuthology.orchestra.run.trial023.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2026-02-25T08:41:26.113 INFO:teuthology.orchestra.run.trial023.stdout: "osd_debug_misdirected_ops": "false", 2026-02-25T08:41:26.113 INFO:teuthology.orchestra.run.trial023.stdout: "osd_debug_no_acting_change": "false", 2026-02-25T08:41:26.113 INFO:teuthology.orchestra.run.trial023.stdout: "osd_debug_no_purge_strays": "false", 2026-02-25T08:41:26.114 INFO:teuthology.orchestra.run.trial023.stdout: "osd_debug_op_order": "false", 2026-02-25T08:41:26.114 INFO:teuthology.orchestra.run.trial023.stdout: "osd_debug_pg_log_writeout": "false", 2026-02-25T08:41:26.114 INFO:teuthology.orchestra.run.trial023.stdout: "osd_debug_pretend_recovery_active": "false", 2026-02-25T08:41:26.114 INFO:teuthology.orchestra.run.trial023.stdout: "osd_debug_random_push_read_error": "0.000000", 2026-02-25T08:41:26.114 INFO:teuthology.orchestra.run.trial023.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2026-02-25T08:41:26.114 INFO:teuthology.orchestra.run.trial023.stdout: "osd_debug_shutdown": "false", 2026-02-25T08:41:26.114 INFO:teuthology.orchestra.run.trial023.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2026-02-25T08:41:26.114 INFO:teuthology.orchestra.run.trial023.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2026-02-25T08:41:26.114 INFO:teuthology.orchestra.run.trial023.stdout: "osd_debug_trim_objects": "false", 2026-02-25T08:41:26.114 INFO:teuthology.orchestra.run.trial023.stdout: "osd_debug_verify_cached_snaps": "false", 2026-02-25T08:41:26.114 INFO:teuthology.orchestra.run.trial023.stdout: "osd_debug_verify_missing_on_start": "false", 2026-02-25T08:41:26.114 INFO:teuthology.orchestra.run.trial023.stdout: "osd_debug_verify_snaps": "false", 2026-02-25T08:41:26.114 INFO:teuthology.orchestra.run.trial023.stdout: "osd_debug_verify_stray_on_activate": "false", 2026-02-25T08:41:26.114 INFO:teuthology.orchestra.run.trial023.stdout: "osd_deep_scrub_interval": "604800.000000", 2026-02-25T08:41:26.114 INFO:teuthology.orchestra.run.trial023.stdout: "osd_deep_scrub_interval_cv": "0.200000", 2026-02-25T08:41:26.114 INFO:teuthology.orchestra.run.trial023.stdout: "osd_deep_scrub_keys": "1024", 2026-02-25T08:41:26.114 INFO:teuthology.orchestra.run.trial023.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2026-02-25T08:41:26.114 INFO:teuthology.orchestra.run.trial023.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2026-02-25T08:41:26.114 INFO:teuthology.orchestra.run.trial023.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2026-02-25T08:41:26.114 INFO:teuthology.orchestra.run.trial023.stdout: "osd_deep_scrub_stride": "4194304", 2026-02-25T08:41:26.114 INFO:teuthology.orchestra.run.trial023.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2026-02-25T08:41:26.114 INFO:teuthology.orchestra.run.trial023.stdout: "osd_default_data_pool_replay_window": "5", 2026-02-25T08:41:26.114 INFO:teuthology.orchestra.run.trial023.stdout: "osd_default_notify_timeout": "30", 2026-02-25T08:41:26.114 INFO:teuthology.orchestra.run.trial023.stdout: "osd_delete_sleep": "0.000000", 2026-02-25T08:41:26.114 INFO:teuthology.orchestra.run.trial023.stdout: "osd_delete_sleep_hdd": "5.000000", 2026-02-25T08:41:26.114 INFO:teuthology.orchestra.run.trial023.stdout: "osd_delete_sleep_hybrid": "1.000000", 2026-02-25T08:41:26.114 INFO:teuthology.orchestra.run.trial023.stdout: "osd_delete_sleep_ssd": "1.000000", 2026-02-25T08:41:26.114 INFO:teuthology.orchestra.run.trial023.stdout: "osd_discard_disconnected_ops": "true", 2026-02-25T08:41:26.114 INFO:teuthology.orchestra.run.trial023.stdout: "osd_ec_partial_reads": "true", 2026-02-25T08:41:26.114 INFO:teuthology.orchestra.run.trial023.stdout: "osd_enable_op_tracker": "true", 2026-02-25T08:41:26.114 INFO:teuthology.orchestra.run.trial023.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2026-02-25T08:41:26.114 INFO:teuthology.orchestra.run.trial023.stdout: "osd_extblkdev_plugins": "vdo", 2026-02-25T08:41:26.114 INFO:teuthology.orchestra.run.trial023.stdout: "osd_failsafe_full_ratio": "0.970000", 2026-02-25T08:41:26.115 INFO:teuthology.orchestra.run.trial023.stdout: "osd_fast_fail_on_connection_refused": "true", 2026-02-25T08:41:26.115 INFO:teuthology.orchestra.run.trial023.stdout: "osd_fast_info": "true", 2026-02-25T08:41:26.115 INFO:teuthology.orchestra.run.trial023.stdout: "osd_fast_shutdown": "true", 2026-02-25T08:41:26.115 INFO:teuthology.orchestra.run.trial023.stdout: "osd_fast_shutdown_notify_mon": "true", 2026-02-25T08:41:26.115 INFO:teuthology.orchestra.run.trial023.stdout: "osd_fast_shutdown_timeout": "15", 2026-02-25T08:41:26.115 INFO:teuthology.orchestra.run.trial023.stdout: "osd_find_best_info_ignore_history_les": "false", 2026-02-25T08:41:26.115 INFO:teuthology.orchestra.run.trial023.stdout: "osd_force_auth_primary_missing_objects": "100", 2026-02-25T08:41:26.115 INFO:teuthology.orchestra.run.trial023.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2026-02-25T08:41:26.115 INFO:teuthology.orchestra.run.trial023.stdout: "osd_function_tracing": "false", 2026-02-25T08:41:26.115 INFO:teuthology.orchestra.run.trial023.stdout: "osd_heartbeat_grace": "20", 2026-02-25T08:41:26.115 INFO:teuthology.orchestra.run.trial023.stdout: "osd_heartbeat_interval": "6", 2026-02-25T08:41:26.115 INFO:teuthology.orchestra.run.trial023.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2026-02-25T08:41:26.115 INFO:teuthology.orchestra.run.trial023.stdout: "osd_heartbeat_min_peers": "10", 2026-02-25T08:41:26.115 INFO:teuthology.orchestra.run.trial023.stdout: "osd_heartbeat_min_size": "2000", 2026-02-25T08:41:26.115 INFO:teuthology.orchestra.run.trial023.stdout: "osd_heartbeat_stale": "600", 2026-02-25T08:41:26.115 INFO:teuthology.orchestra.run.trial023.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2026-02-25T08:41:26.115 INFO:teuthology.orchestra.run.trial023.stdout: "osd_hit_set_max_size": "100000", 2026-02-25T08:41:26.115 INFO:teuthology.orchestra.run.trial023.stdout: "osd_hit_set_min_size": "1000", 2026-02-25T08:41:26.115 INFO:teuthology.orchestra.run.trial023.stdout: "osd_hit_set_namespace": ".ceph-internal", 2026-02-25T08:41:26.115 INFO:teuthology.orchestra.run.trial023.stdout: "osd_ignore_stale_divergent_priors": "false", 2026-02-25T08:41:26.115 INFO:teuthology.orchestra.run.trial023.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2026-02-25T08:41:26.115 INFO:teuthology.orchestra.run.trial023.stdout: "osd_inject_failure_on_pg_removal": "false", 2026-02-25T08:41:26.115 INFO:teuthology.orchestra.run.trial023.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2026-02-25T08:41:26.115 INFO:teuthology.orchestra.run.trial023.stdout: "osd_journal_flush_on_shutdown": "true", 2026-02-25T08:41:26.115 INFO:teuthology.orchestra.run.trial023.stdout: "osd_journal_size": "5120", 2026-02-25T08:41:26.115 INFO:teuthology.orchestra.run.trial023.stdout: "osd_kill_backfill_at": "0", 2026-02-25T08:41:26.115 INFO:teuthology.orchestra.run.trial023.stdout: "osd_loop_before_reset_tphandle": "64", 2026-02-25T08:41:26.115 INFO:teuthology.orchestra.run.trial023.stdout: "osd_map_cache_size": "50", 2026-02-25T08:41:26.115 INFO:teuthology.orchestra.run.trial023.stdout: "osd_map_dedup": "true", 2026-02-25T08:41:26.115 INFO:teuthology.orchestra.run.trial023.stdout: "osd_map_message_max": "40", 2026-02-25T08:41:26.115 INFO:teuthology.orchestra.run.trial023.stdout: "osd_map_message_max_bytes": "10485760", 2026-02-25T08:41:26.115 INFO:teuthology.orchestra.run.trial023.stdout: "osd_map_share_max_epochs": "40", 2026-02-25T08:41:26.115 INFO:teuthology.orchestra.run.trial023.stdout: "osd_max_attr_name_len": "100", 2026-02-25T08:41:26.115 INFO:teuthology.orchestra.run.trial023.stdout: "osd_max_attr_size": "0", 2026-02-25T08:41:26.115 INFO:teuthology.orchestra.run.trial023.stdout: "osd_max_backfills": "1", 2026-02-25T08:41:26.116 INFO:teuthology.orchestra.run.trial023.stdout: "osd_max_markdown_count": "5", 2026-02-25T08:41:26.116 INFO:teuthology.orchestra.run.trial023.stdout: "osd_max_markdown_period": "600", 2026-02-25T08:41:26.116 INFO:teuthology.orchestra.run.trial023.stdout: "osd_max_object_name_len": "2048", 2026-02-25T08:41:26.116 INFO:teuthology.orchestra.run.trial023.stdout: "osd_max_object_namespace_len": "256", 2026-02-25T08:41:26.116 INFO:teuthology.orchestra.run.trial023.stdout: "osd_max_object_size": "134217728", 2026-02-25T08:41:26.116 INFO:teuthology.orchestra.run.trial023.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2026-02-25T08:41:26.116 INFO:teuthology.orchestra.run.trial023.stdout: "osd_max_omap_entries_per_request": "1024", 2026-02-25T08:41:26.116 INFO:teuthology.orchestra.run.trial023.stdout: "osd_max_pg_blocked_by": "16", 2026-02-25T08:41:26.116 INFO:teuthology.orchestra.run.trial023.stdout: "osd_max_pg_log_entries": "2", 2026-02-25T08:41:26.116 INFO:teuthology.orchestra.run.trial023.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2026-02-25T08:41:26.116 INFO:teuthology.orchestra.run.trial023.stdout: "osd_max_pgls": "1024", 2026-02-25T08:41:26.116 INFO:teuthology.orchestra.run.trial023.stdout: "osd_max_push_cost": "8388608", 2026-02-25T08:41:26.116 INFO:teuthology.orchestra.run.trial023.stdout: "osd_max_push_objects": "10", 2026-02-25T08:41:26.116 INFO:teuthology.orchestra.run.trial023.stdout: "osd_max_scrubs": "3", 2026-02-25T08:41:26.116 INFO:teuthology.orchestra.run.trial023.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2026-02-25T08:41:26.116 INFO:teuthology.orchestra.run.trial023.stdout: "osd_max_trimming_pgs": "2", 2026-02-25T08:41:26.116 INFO:teuthology.orchestra.run.trial023.stdout: "osd_max_write_op_reply_len": "64", 2026-02-25T08:41:26.116 INFO:teuthology.orchestra.run.trial023.stdout: "osd_max_write_size": "90", 2026-02-25T08:41:26.116 INFO:teuthology.orchestra.run.trial023.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2026-02-25T08:41:26.116 INFO:teuthology.orchestra.run.trial023.stdout: "osd_mclock_iops_capacity_low_threshold_hdd": "50.000000", 2026-02-25T08:41:26.116 INFO:teuthology.orchestra.run.trial023.stdout: "osd_mclock_iops_capacity_low_threshold_ssd": "1000.000000", 2026-02-25T08:41:26.116 INFO:teuthology.orchestra.run.trial023.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2026-02-25T08:41:26.116 INFO:teuthology.orchestra.run.trial023.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2026-02-25T08:41:26.116 INFO:teuthology.orchestra.run.trial023.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2026-02-25T08:41:26.116 INFO:teuthology.orchestra.run.trial023.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2026-02-25T08:41:26.116 INFO:teuthology.orchestra.run.trial023.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2026-02-25T08:41:26.116 INFO:teuthology.orchestra.run.trial023.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2026-02-25T08:41:26.116 INFO:teuthology.orchestra.run.trial023.stdout: "osd_mclock_override_recovery_settings": "false", 2026-02-25T08:41:26.116 INFO:teuthology.orchestra.run.trial023.stdout: "osd_mclock_profile": "balanced", 2026-02-25T08:41:26.116 INFO:teuthology.orchestra.run.trial023.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2026-02-25T08:41:26.116 INFO:teuthology.orchestra.run.trial023.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2026-02-25T08:41:26.116 INFO:teuthology.orchestra.run.trial023.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2026-02-25T08:41:26.116 INFO:teuthology.orchestra.run.trial023.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2026-02-25T08:41:26.116 INFO:teuthology.orchestra.run.trial023.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2026-02-25T08:41:26.116 INFO:teuthology.orchestra.run.trial023.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2026-02-25T08:41:26.117 INFO:teuthology.orchestra.run.trial023.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2026-02-25T08:41:26.117 INFO:teuthology.orchestra.run.trial023.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2026-02-25T08:41:26.117 INFO:teuthology.orchestra.run.trial023.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2026-02-25T08:41:26.117 INFO:teuthology.orchestra.run.trial023.stdout: "osd_mclock_scheduler_client_wgt": "1", 2026-02-25T08:41:26.117 INFO:teuthology.orchestra.run.trial023.stdout: "osd_mclock_skip_benchmark": "false", 2026-02-25T08:41:26.117 INFO:teuthology.orchestra.run.trial023.stdout: "osd_memory_base": "805306368", 2026-02-25T08:41:26.117 INFO:teuthology.orchestra.run.trial023.stdout: "osd_memory_cache_min": "134217728", 2026-02-25T08:41:26.117 INFO:teuthology.orchestra.run.trial023.stdout: "osd_memory_cache_resize_interval": "1.000000", 2026-02-25T08:41:26.117 INFO:teuthology.orchestra.run.trial023.stdout: "osd_memory_expected_fragmentation": "0.150000", 2026-02-25T08:41:26.117 INFO:teuthology.orchestra.run.trial023.stdout: "osd_memory_target": "4294967296", 2026-02-25T08:41:26.117 INFO:teuthology.orchestra.run.trial023.stdout: "osd_memory_target_autotune": "false", 2026-02-25T08:41:26.117 INFO:teuthology.orchestra.run.trial023.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2026-02-25T08:41:26.117 INFO:teuthology.orchestra.run.trial023.stdout: "osd_min_pg_log_entries": "1", 2026-02-25T08:41:26.117 INFO:teuthology.orchestra.run.trial023.stdout: "osd_min_recovery_priority": "0", 2026-02-25T08:41:26.117 INFO:teuthology.orchestra.run.trial023.stdout: "osd_mon_heartbeat_interval": "30", 2026-02-25T08:41:26.117 INFO:teuthology.orchestra.run.trial023.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2026-02-25T08:41:26.117 INFO:teuthology.orchestra.run.trial023.stdout: "osd_mon_report_interval": "5", 2026-02-25T08:41:26.117 INFO:teuthology.orchestra.run.trial023.stdout: "osd_mon_report_max_in_flight": "2", 2026-02-25T08:41:26.117 INFO:teuthology.orchestra.run.trial023.stdout: "osd_mon_shutdown_timeout": "5.000000", 2026-02-25T08:41:26.117 INFO:teuthology.orchestra.run.trial023.stdout: "osd_num_cache_shards": "32", 2026-02-25T08:41:26.117 INFO:teuthology.orchestra.run.trial023.stdout: "osd_num_op_tracker_shard": "32", 2026-02-25T08:41:26.117 INFO:teuthology.orchestra.run.trial023.stdout: "osd_numa_auto_affinity": "true", 2026-02-25T08:41:26.117 INFO:teuthology.orchestra.run.trial023.stdout: "osd_numa_node": "-1", 2026-02-25T08:41:26.117 INFO:teuthology.orchestra.run.trial023.stdout: "osd_numa_prefer_iface": "true", 2026-02-25T08:41:26.117 INFO:teuthology.orchestra.run.trial023.stdout: "osd_object_clean_region_max_num_intervals": "10", 2026-02-25T08:41:26.117 INFO:teuthology.orchestra.run.trial023.stdout: "osd_objecter_finishers": "1", 2026-02-25T08:41:26.117 INFO:teuthology.orchestra.run.trial023.stdout: "osd_objectstore": "bluestore", 2026-02-25T08:41:26.117 INFO:teuthology.orchestra.run.trial023.stdout: "osd_objectstore_fuse": "false", 2026-02-25T08:41:26.117 INFO:teuthology.orchestra.run.trial023.stdout: "osd_objectstore_ideal_list_max": "64", 2026-02-25T08:41:26.117 INFO:teuthology.orchestra.run.trial023.stdout: "osd_objectstore_tracing": "false", 2026-02-25T08:41:26.117 INFO:teuthology.orchestra.run.trial023.stdout: "osd_op_complaint_time": "30.000000", 2026-02-25T08:41:26.117 INFO:teuthology.orchestra.run.trial023.stdout: "osd_op_history_duration": "600", 2026-02-25T08:41:26.117 INFO:teuthology.orchestra.run.trial023.stdout: "osd_op_history_size": "20", 2026-02-25T08:41:26.117 INFO:teuthology.orchestra.run.trial023.stdout: "osd_op_history_slow_op_size": "20", 2026-02-25T08:41:26.117 INFO:teuthology.orchestra.run.trial023.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2026-02-25T08:41:26.118 INFO:teuthology.orchestra.run.trial023.stdout: "osd_op_log_threshold": "5", 2026-02-25T08:41:26.118 INFO:teuthology.orchestra.run.trial023.stdout: "osd_op_num_shards": "0", 2026-02-25T08:41:26.118 INFO:teuthology.orchestra.run.trial023.stdout: "osd_op_num_shards_hdd": "1", 2026-02-25T08:41:26.118 INFO:teuthology.orchestra.run.trial023.stdout: "osd_op_num_shards_ssd": "8", 2026-02-25T08:41:26.118 INFO:teuthology.orchestra.run.trial023.stdout: "osd_op_num_threads_per_shard": "0", 2026-02-25T08:41:26.118 INFO:teuthology.orchestra.run.trial023.stdout: "osd_op_num_threads_per_shard_hdd": "5", 2026-02-25T08:41:26.118 INFO:teuthology.orchestra.run.trial023.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2026-02-25T08:41:26.118 INFO:teuthology.orchestra.run.trial023.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2026-02-25T08:41:26.118 INFO:teuthology.orchestra.run.trial023.stdout: "osd_op_pq_min_cost": "65536", 2026-02-25T08:41:26.118 INFO:teuthology.orchestra.run.trial023.stdout: "osd_op_queue": "mclock_scheduler", 2026-02-25T08:41:26.118 INFO:teuthology.orchestra.run.trial023.stdout: "osd_op_queue_cut_off": "high", 2026-02-25T08:41:26.118 INFO:teuthology.orchestra.run.trial023.stdout: "osd_op_thread_suicide_timeout": "150", 2026-02-25T08:41:26.118 INFO:teuthology.orchestra.run.trial023.stdout: "osd_op_thread_timeout": "15", 2026-02-25T08:41:26.118 INFO:teuthology.orchestra.run.trial023.stdout: "osd_open_classes_on_start": "true", 2026-02-25T08:41:26.118 INFO:teuthology.orchestra.run.trial023.stdout: "osd_os_flags": "0", 2026-02-25T08:41:26.118 INFO:teuthology.orchestra.run.trial023.stdout: "osd_peering_op_priority": "255", 2026-02-25T08:41:26.118 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pg_delete_cost": "1048576", 2026-02-25T08:41:26.118 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pg_delete_priority": "5", 2026-02-25T08:41:26.118 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2026-02-25T08:41:26.118 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2026-02-25T08:41:26.118 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pg_log_dups_tracked": "3000", 2026-02-25T08:41:26.118 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pg_log_trim_max": "10000", 2026-02-25T08:41:26.118 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pg_log_trim_min": "0", 2026-02-25T08:41:26.118 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2026-02-25T08:41:26.118 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pg_object_context_cache_count": "64", 2026-02-25T08:41:26.118 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2026-02-25T08:41:26.118 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2026-02-25T08:41:26.118 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2026-02-25T08:41:26.118 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pool_default_cache_min_evict_age": "0", 2026-02-25T08:41:26.118 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pool_default_cache_min_flush_age": "0", 2026-02-25T08:41:26.118 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2026-02-25T08:41:26.118 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2026-02-25T08:41:26.118 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2026-02-25T08:41:26.118 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pool_default_crimson": "true", 2026-02-25T08:41:26.118 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pool_default_crush_rule": "-1", 2026-02-25T08:41:26.118 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pool_default_ec_fast_read": "false", 2026-02-25T08:41:26.119 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pool_default_erasure_code_profile": "plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2026-02-25T08:41:26.119 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pool_default_flag_bulk": "false", 2026-02-25T08:41:26.119 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pool_default_flag_ec_optimizations": "false", 2026-02-25T08:41:26.119 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pool_default_flag_hashpspool": "true", 2026-02-25T08:41:26.119 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pool_default_flag_nodelete": "false", 2026-02-25T08:41:26.119 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pool_default_flag_nopgchange": "false", 2026-02-25T08:41:26.119 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pool_default_flag_nosizechange": "false", 2026-02-25T08:41:26.119 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pool_default_flags": "0", 2026-02-25T08:41:26.119 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2026-02-25T08:41:26.119 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pool_default_min_size": "2", 2026-02-25T08:41:26.119 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2026-02-25T08:41:26.119 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pool_default_pg_num": "32", 2026-02-25T08:41:26.119 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pool_default_pgp_num": "0", 2026-02-25T08:41:26.119 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2026-02-25T08:41:26.119 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pool_default_read_ratio": "70", 2026-02-25T08:41:26.119 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pool_default_size": "3", 2026-02-25T08:41:26.119 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pool_default_type": "replicated", 2026-02-25T08:41:26.119 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pool_erasure_code_stripe_unit": "0", 2026-02-25T08:41:26.119 INFO:teuthology.orchestra.run.trial023.stdout: "osd_pool_use_gmt_hitset": "true", 2026-02-25T08:41:26.119 INFO:teuthology.orchestra.run.trial023.stdout: "osd_push_per_object_cost": "1000", 2026-02-25T08:41:26.119 INFO:teuthology.orchestra.run.trial023.stdout: "osd_read_ec_check_for_errors": "false", 2026-02-25T08:41:26.119 INFO:teuthology.orchestra.run.trial023.stdout: "osd_recover_clone_overlap": "true", 2026-02-25T08:41:26.119 INFO:teuthology.orchestra.run.trial023.stdout: "osd_recover_clone_overlap_limit": "10", 2026-02-25T08:41:26.119 INFO:teuthology.orchestra.run.trial023.stdout: "osd_recovery_cost": "20971520", 2026-02-25T08:41:26.119 INFO:teuthology.orchestra.run.trial023.stdout: "osd_recovery_delay_start": "0.000000", 2026-02-25T08:41:26.119 INFO:teuthology.orchestra.run.trial023.stdout: "osd_recovery_max_active": "0", 2026-02-25T08:41:26.119 INFO:teuthology.orchestra.run.trial023.stdout: "osd_recovery_max_active_hdd": "3", 2026-02-25T08:41:26.119 INFO:teuthology.orchestra.run.trial023.stdout: "osd_recovery_max_active_ssd": "10", 2026-02-25T08:41:26.119 INFO:teuthology.orchestra.run.trial023.stdout: "osd_recovery_max_chunk": "8388608", 2026-02-25T08:41:26.119 INFO:teuthology.orchestra.run.trial023.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2026-02-25T08:41:26.119 INFO:teuthology.orchestra.run.trial023.stdout: "osd_recovery_max_single_start": "1", 2026-02-25T08:41:26.119 INFO:teuthology.orchestra.run.trial023.stdout: "osd_recovery_op_priority": "3", 2026-02-25T08:41:26.119 INFO:teuthology.orchestra.run.trial023.stdout: "osd_recovery_op_warn_multiple": "16", 2026-02-25T08:41:26.119 INFO:teuthology.orchestra.run.trial023.stdout: "osd_recovery_priority": "5", 2026-02-25T08:41:26.119 INFO:teuthology.orchestra.run.trial023.stdout: "osd_recovery_retry_interval": "30.000000", 2026-02-25T08:41:26.119 INFO:teuthology.orchestra.run.trial023.stdout: "osd_recovery_sleep": "0.000000", 2026-02-25T08:41:26.120 INFO:teuthology.orchestra.run.trial023.stdout: "osd_recovery_sleep_degraded": "0.000000", 2026-02-25T08:41:26.120 INFO:teuthology.orchestra.run.trial023.stdout: "osd_recovery_sleep_degraded_hdd": "0.100000", 2026-02-25T08:41:26.120 INFO:teuthology.orchestra.run.trial023.stdout: "osd_recovery_sleep_degraded_hybrid": "0.025000", 2026-02-25T08:41:26.120 INFO:teuthology.orchestra.run.trial023.stdout: "osd_recovery_sleep_degraded_ssd": "0.000000", 2026-02-25T08:41:26.120 INFO:teuthology.orchestra.run.trial023.stdout: "osd_recovery_sleep_hdd": "0.100000", 2026-02-25T08:41:26.120 INFO:teuthology.orchestra.run.trial023.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2026-02-25T08:41:26.120 INFO:teuthology.orchestra.run.trial023.stdout: "osd_recovery_sleep_ssd": "0.000000", 2026-02-25T08:41:26.120 INFO:teuthology.orchestra.run.trial023.stdout: "osd_requested_scrub_priority": "5", 2026-02-25T08:41:26.120 INFO:teuthology.orchestra.run.trial023.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2026-02-25T08:41:26.120 INFO:teuthology.orchestra.run.trial023.stdout: "osd_rollback_to_cluster_snap": "", 2026-02-25T08:41:26.120 INFO:teuthology.orchestra.run.trial023.stdout: "osd_scrub_auto_repair": "false", 2026-02-25T08:41:26.120 INFO:teuthology.orchestra.run.trial023.stdout: "osd_scrub_auto_repair_num_errors": "5", 2026-02-25T08:41:26.120 INFO:teuthology.orchestra.run.trial023.stdout: "osd_scrub_backoff_ratio": "0.660000", 2026-02-25T08:41:26.120 INFO:teuthology.orchestra.run.trial023.stdout: "osd_scrub_begin_hour": "0", 2026-02-25T08:41:26.120 INFO:teuthology.orchestra.run.trial023.stdout: "osd_scrub_begin_week_day": "0", 2026-02-25T08:41:26.120 INFO:teuthology.orchestra.run.trial023.stdout: "osd_scrub_chunk_max": "15", 2026-02-25T08:41:26.120 INFO:teuthology.orchestra.run.trial023.stdout: "osd_scrub_chunk_min": "5", 2026-02-25T08:41:26.120 INFO:teuthology.orchestra.run.trial023.stdout: "osd_scrub_cost": "52428800", 2026-02-25T08:41:26.120 INFO:teuthology.orchestra.run.trial023.stdout: "osd_scrub_disable_reservation_queuing": "false", 2026-02-25T08:41:26.120 INFO:teuthology.orchestra.run.trial023.stdout: "osd_scrub_during_recovery": "false", 2026-02-25T08:41:26.120 INFO:teuthology.orchestra.run.trial023.stdout: "osd_scrub_end_hour": "0", 2026-02-25T08:41:26.120 INFO:teuthology.orchestra.run.trial023.stdout: "osd_scrub_end_week_day": "0", 2026-02-25T08:41:26.120 INFO:teuthology.orchestra.run.trial023.stdout: "osd_scrub_event_cost": "4096", 2026-02-25T08:41:26.120 INFO:teuthology.orchestra.run.trial023.stdout: "osd_scrub_extended_sleep": "0.000000", 2026-02-25T08:41:26.120 INFO:teuthology.orchestra.run.trial023.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2026-02-25T08:41:26.120 INFO:teuthology.orchestra.run.trial023.stdout: "osd_scrub_invalid_stats": "true", 2026-02-25T08:41:26.120 INFO:teuthology.orchestra.run.trial023.stdout: "osd_scrub_load_threshold": "10.000000", 2026-02-25T08:41:26.120 INFO:teuthology.orchestra.run.trial023.stdout: "osd_scrub_max_interval": "604800.000000", 2026-02-25T08:41:26.120 INFO:teuthology.orchestra.run.trial023.stdout: "osd_scrub_max_preemptions": "5", 2026-02-25T08:41:26.120 INFO:teuthology.orchestra.run.trial023.stdout: "osd_scrub_min_interval": "86400.000000", 2026-02-25T08:41:26.120 INFO:teuthology.orchestra.run.trial023.stdout: "osd_scrub_priority": "5", 2026-02-25T08:41:26.120 INFO:teuthology.orchestra.run.trial023.stdout: "osd_scrub_retry_after_noscrub": "60", 2026-02-25T08:41:26.120 INFO:teuthology.orchestra.run.trial023.stdout: "osd_scrub_retry_delay": "30", 2026-02-25T08:41:26.120 INFO:teuthology.orchestra.run.trial023.stdout: "osd_scrub_retry_new_interval": "10", 2026-02-25T08:41:26.120 INFO:teuthology.orchestra.run.trial023.stdout: "osd_scrub_retry_pg_state": "60", 2026-02-25T08:41:26.120 INFO:teuthology.orchestra.run.trial023.stdout: "osd_scrub_retry_trimming": "10", 2026-02-25T08:41:26.120 INFO:teuthology.orchestra.run.trial023.stdout: "osd_scrub_sleep": "0.000000", 2026-02-25T08:41:26.120 INFO:teuthology.orchestra.run.trial023.stdout: "osd_shallow_scrub_chunk_max": "100", 2026-02-25T08:41:26.121 INFO:teuthology.orchestra.run.trial023.stdout: "osd_shallow_scrub_chunk_min": "50", 2026-02-25T08:41:26.121 INFO:teuthology.orchestra.run.trial023.stdout: "osd_shutdown_pgref_assert": "false", 2026-02-25T08:41:26.121 INFO:teuthology.orchestra.run.trial023.stdout: "osd_skip_check_past_interval_bounds": "false", 2026-02-25T08:41:26.121 INFO:teuthology.orchestra.run.trial023.stdout: "osd_skip_data_digest": "false", 2026-02-25T08:41:26.121 INFO:teuthology.orchestra.run.trial023.stdout: "osd_smart_report_timeout": "5", 2026-02-25T08:41:26.121 INFO:teuthology.orchestra.run.trial023.stdout: "osd_snap_trim_cost": "1048576", 2026-02-25T08:41:26.121 INFO:teuthology.orchestra.run.trial023.stdout: "osd_snap_trim_priority": "5", 2026-02-25T08:41:26.121 INFO:teuthology.orchestra.run.trial023.stdout: "osd_snap_trim_sleep": "0.000000", 2026-02-25T08:41:26.121 INFO:teuthology.orchestra.run.trial023.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2026-02-25T08:41:26.121 INFO:teuthology.orchestra.run.trial023.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2026-02-25T08:41:26.121 INFO:teuthology.orchestra.run.trial023.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2026-02-25T08:41:26.121 INFO:teuthology.orchestra.run.trial023.stdout: "osd_stats_update_period_not_scrubbing": "120", 2026-02-25T08:41:26.121 INFO:teuthology.orchestra.run.trial023.stdout: "osd_stats_update_period_scrubbing": "15", 2026-02-25T08:41:26.121 INFO:teuthology.orchestra.run.trial023.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2026-02-25T08:41:26.121 INFO:teuthology.orchestra.run.trial023.stdout: "osd_target_transaction_size": "30", 2026-02-25T08:41:26.121 INFO:teuthology.orchestra.run.trial023.stdout: "osd_tier_default_cache_hit_set_count": "4", 2026-02-25T08:41:26.121 INFO:teuthology.orchestra.run.trial023.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2026-02-25T08:41:26.121 INFO:teuthology.orchestra.run.trial023.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2026-02-25T08:41:26.121 INFO:teuthology.orchestra.run.trial023.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2026-02-25T08:41:26.121 INFO:teuthology.orchestra.run.trial023.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2026-02-25T08:41:26.121 INFO:teuthology.orchestra.run.trial023.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2026-02-25T08:41:26.121 INFO:teuthology.orchestra.run.trial023.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2026-02-25T08:41:26.121 INFO:teuthology.orchestra.run.trial023.stdout: "osd_tier_default_cache_mode": "writeback", 2026-02-25T08:41:26.121 INFO:teuthology.orchestra.run.trial023.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2026-02-25T08:41:26.121 INFO:teuthology.orchestra.run.trial023.stdout: "osd_tier_promote_max_objects_sec": "25", 2026-02-25T08:41:26.121 INFO:teuthology.orchestra.run.trial023.stdout: "osd_tracing": "false", 2026-02-25T08:41:26.121 INFO:teuthology.orchestra.run.trial023.stdout: "osd_use_stale_snap": "false", 2026-02-25T08:41:26.121 INFO:teuthology.orchestra.run.trial023.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2026-02-25T08:41:26.121 INFO:teuthology.orchestra.run.trial023.stdout: "osdc_blkin_trace_all": "false", 2026-02-25T08:41:26.121 INFO:teuthology.orchestra.run.trial023.stdout: "paxos_kill_at": "0", 2026-02-25T08:41:26.121 INFO:teuthology.orchestra.run.trial023.stdout: "paxos_max_join_drift": "10", 2026-02-25T08:41:26.121 INFO:teuthology.orchestra.run.trial023.stdout: "paxos_min": "500", 2026-02-25T08:41:26.121 INFO:teuthology.orchestra.run.trial023.stdout: "paxos_min_wait": "0.050000", 2026-02-25T08:41:26.121 INFO:teuthology.orchestra.run.trial023.stdout: "paxos_propose_interval": "1.000000", 2026-02-25T08:41:26.121 INFO:teuthology.orchestra.run.trial023.stdout: "paxos_service_trim_max": "500", 2026-02-25T08:41:26.121 INFO:teuthology.orchestra.run.trial023.stdout: "paxos_service_trim_max_multiplier": "20", 2026-02-25T08:41:26.121 INFO:teuthology.orchestra.run.trial023.stdout: "paxos_service_trim_min": "10", 2026-02-25T08:41:26.122 INFO:teuthology.orchestra.run.trial023.stdout: "paxos_stash_full_interval": "25", 2026-02-25T08:41:26.122 INFO:teuthology.orchestra.run.trial023.stdout: "paxos_trim_max": "500", 2026-02-25T08:41:26.122 INFO:teuthology.orchestra.run.trial023.stdout: "paxos_trim_min": "250", 2026-02-25T08:41:26.122 INFO:teuthology.orchestra.run.trial023.stdout: "perf": "true", 2026-02-25T08:41:26.122 INFO:teuthology.orchestra.run.trial023.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2026-02-25T08:41:26.122 INFO:teuthology.orchestra.run.trial023.stdout: "plugin_crypto_accelerator": "crypto_isal", 2026-02-25T08:41:26.122 INFO:teuthology.orchestra.run.trial023.stdout: "plugin_dir": "/usr/lib64/ceph", 2026-02-25T08:41:26.122 INFO:teuthology.orchestra.run.trial023.stdout: "pool_availability_update_interval": "1.000000", 2026-02-25T08:41:26.122 INFO:teuthology.orchestra.run.trial023.stdout: "public_addr": "-", 2026-02-25T08:41:26.122 INFO:teuthology.orchestra.run.trial023.stdout: "public_addrv": "", 2026-02-25T08:41:26.122 INFO:teuthology.orchestra.run.trial023.stdout: "public_bind_addr": "-", 2026-02-25T08:41:26.122 INFO:teuthology.orchestra.run.trial023.stdout: "public_network": "", 2026-02-25T08:41:26.122 INFO:teuthology.orchestra.run.trial023.stdout: "public_network_interface": "", 2026-02-25T08:41:26.122 INFO:teuthology.orchestra.run.trial023.stdout: "qat_compressor_busy_polling": "false", 2026-02-25T08:41:26.122 INFO:teuthology.orchestra.run.trial023.stdout: "qat_compressor_enabled": "false", 2026-02-25T08:41:26.122 INFO:teuthology.orchestra.run.trial023.stdout: "qat_compressor_session_max_number": "256", 2026-02-25T08:41:26.122 INFO:teuthology.orchestra.run.trial023.stdout: "rados_mon_op_timeout": "0", 2026-02-25T08:41:26.122 INFO:teuthology.orchestra.run.trial023.stdout: "rados_osd_op_timeout": "0", 2026-02-25T08:41:26.122 INFO:teuthology.orchestra.run.trial023.stdout: "rados_replica_read_policy": "default", 2026-02-25T08:41:26.122 INFO:teuthology.orchestra.run.trial023.stdout: "rados_replica_read_policy_on_objclass": "false", 2026-02-25T08:41:26.122 INFO:teuthology.orchestra.run.trial023.stdout: "rados_tracing": "false", 2026-02-25T08:41:26.122 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_atime_update_interval": "60", 2026-02-25T08:41:26.122 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2026-02-25T08:41:26.122 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_balance_parent_reads": "false", 2026-02-25T08:41:26.122 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_balance_snap_reads": "false", 2026-02-25T08:41:26.122 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_blkin_trace_all": "false", 2026-02-25T08:41:26.122 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_blocklist_expire_seconds": "0", 2026-02-25T08:41:26.122 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_blocklist_on_break_lock": "true", 2026-02-25T08:41:26.122 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_cache": "true", 2026-02-25T08:41:26.122 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_cache_block_writes_upfront": "false", 2026-02-25T08:41:26.122 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_cache_max_dirty": "25165824", 2026-02-25T08:41:26.122 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_cache_max_dirty_age": "1.000000", 2026-02-25T08:41:26.122 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_cache_max_dirty_object": "0", 2026-02-25T08:41:26.122 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_cache_policy": "writearound", 2026-02-25T08:41:26.122 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_cache_size": "33554432", 2026-02-25T08:41:26.122 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_cache_target_dirty": "16777216", 2026-02-25T08:41:26.122 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_cache_writethrough_until_flush": "true", 2026-02-25T08:41:26.122 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_clone_copy_on_read": "false", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_compression_hint": "none", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_concurrent_management_ops": "10", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_config_pool_override_update_timestamp": "0", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_default_clone_format": "auto", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_default_data_pool": "", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_default_features": "61", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_default_format": "2", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_default_map_options": "", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_default_order": "22", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_default_pool": "rbd", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_default_stripe_count": "0", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_default_stripe_unit": "0", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_disable_zero_copy_writes": "true", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_discard_granularity_bytes": "65536", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_discard_on_zeroed_write_same": "true", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_enable_alloc_hint": "true", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_io_scheduler": "simple", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_journal_commit_age": "5.000000", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_journal_max_payload_bytes": "16384", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_journal_object_flush_age": "0.000000", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_journal_object_flush_bytes": "1048576", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_journal_object_flush_interval": "0", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_journal_order": "24", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_journal_pool": "", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_journal_splay_width": "4", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_localize_parent_reads": "false", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_localize_snap_reads": "false", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2026-02-25T08:41:26.123 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_mirror_image_policy_type": "simple", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_mirror_image_state_check_interval": "30", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_mirror_memory_autotune": "true", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_mirror_memory_base": "805306368", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_mirror_memory_cache_min": "134217728", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_mirror_memory_target": "4294967296", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_mirror_perf_stats_prio": "5", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_mirroring_delete_delay": "0", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_mirroring_replay_delay": "0", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_move_to_trash_on_remove": "false", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_mtime_update_interval": "60", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_non_blocking_aio": "true", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_op_thread_timeout": "60", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_op_threads": "1", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_parent_cache_enabled": "false", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_persistent_cache_mode": "disabled", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_persistent_cache_path": "/tmp", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_persistent_cache_size": "1073741824", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_plugins": "", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_qos_bps_burst": "0", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_qos_bps_burst_seconds": "1", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_qos_bps_limit": "0", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_qos_exclude_ops": "0", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_qos_iops_burst": "0", 2026-02-25T08:41:26.124 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_qos_iops_burst_seconds": "1", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_qos_iops_limit": "0", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_qos_read_bps_burst": "0", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_qos_read_bps_limit": "0", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_qos_read_iops_burst": "0", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_qos_read_iops_limit": "0", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_qos_schedule_tick_min": "50", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_qos_write_bps_burst": "0", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_qos_write_bps_limit": "0", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_qos_write_iops_burst": "0", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_qos_write_iops_limit": "0", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_quiesce_notification_attempts": "10", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_read_from_replica_policy": "default", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_readahead_max_bytes": "524288", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_readahead_trigger_requests": "10", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_request_timed_out_seconds": "30", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_skip_partial_discard": "true", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_tracing": "false", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_validate_names": "true", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rbd_validate_pool": "true", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "restapi_base_url": "", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "restapi_log_level": "", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_account_default_quota_max_objects": "-1", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_account_default_quota_max_size": "-1", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_acl_grants_max_num": "100", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_admin_entry": "admin", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_asio_assert_yielding": "false", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_backend_store": "rados", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_barbican_url": "", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_beast_enable_async": "true", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_bucket_counters_cache": "false", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_bucket_counters_cache_size": "10000", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_bucket_default_quota_max_size": "-1", 2026-02-25T08:41:26.125 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_bucket_eexist_override": "false", 2026-02-25T08:41:26.126 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_bucket_index_max_aio": "128", 2026-02-25T08:41:26.126 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2026-02-25T08:41:26.126 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_bucket_logging_obj_roll_time": "300", 2026-02-25T08:41:26.126 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_bucket_persistent_notif_num_shards": "11", 2026-02-25T08:41:26.126 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_bucket_quota_cache_size": "10000", 2026-02-25T08:41:26.126 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_bucket_quota_ttl": "600", 2026-02-25T08:41:26.126 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_bucket_sync_spawn_window": "20", 2026-02-25T08:41:26.126 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_cache_enabled": "true", 2026-02-25T08:41:26.126 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_cache_expiry_interval": "900", 2026-02-25T08:41:26.126 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_cache_lru_size": "25000", 2026-02-25T08:41:26.126 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_config_store": "rados", 2026-02-25T08:41:26.126 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_content_length_compat": "false", 2026-02-25T08:41:26.126 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_copy_obj_progress": "true", 2026-02-25T08:41:26.126 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2026-02-25T08:41:26.126 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_cors_rules_max_num": "100", 2026-02-25T08:41:26.126 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_cross_domain_policy": "", 2026-02-25T08:41:26.126 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_crypt_default_encryption_key": "", 2026-02-25T08:41:26.126 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_crypt_kmip_addr": "", 2026-02-25T08:41:26.126 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_crypt_kmip_ca_path": "", 2026-02-25T08:41:26.126 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_crypt_kmip_client_cert": "", 2026-02-25T08:41:26.126 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_crypt_kmip_client_key": "", 2026-02-25T08:41:26.126 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_crypt_kmip_kms_key_template": "", 2026-02-25T08:41:26.126 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_crypt_kmip_password": "", 2026-02-25T08:41:26.126 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2026-02-25T08:41:26.126 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_crypt_kmip_username": "", 2026-02-25T08:41:26.126 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_crypt_require_ssl": "true", 2026-02-25T08:41:26.126 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2026-02-25T08:41:26.126 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2026-02-25T08:41:26.126 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_crypt_sse_s3_backend": "vault", 2026-02-25T08:41:26.126 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2026-02-25T08:41:26.126 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2026-02-25T08:41:26.126 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2026-02-25T08:41:26.126 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2026-02-25T08:41:26.126 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2026-02-25T08:41:26.126 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2026-02-25T08:41:26.126 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_crypt_suppress_logs": "true", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_crypt_vault_addr": "", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_crypt_vault_auth": "token", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_crypt_vault_namespace": "", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_crypt_vault_prefix": "", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_crypt_vault_secret_engine": "transit", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_crypt_vault_ssl_cacert": "", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_crypt_vault_token_file": "", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_crypt_vault_verify_ssl": "true", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_curl_buffersize": "524288", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_curl_low_speed_limit": "1024", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_curl_low_speed_time": "300", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_curl_tcp_keepalive": "0", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_curl_wait_timeout_ms": "1000", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_d3n_l1_fadvise": "4", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_d3n_libaio_aio_num": "64", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_d3n_libaio_aio_threads": "20", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_d4n_backend_address": "127.0.0.1:6379", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_d4n_cache_cleaning_interval": "1000", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_d4n_l1_datacache_disk_reserve": "1073741824", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_d4n_l1_fadvise": "4", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_d4n_l1_write_open_flags": "4096", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_d4n_libaio_aio_num": "64", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_d4n_libaio_aio_threads": "20", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_d4n_local_rgw_address": "127.0.0.1:8000", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2026-02-25T08:41:26.127 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_data_log_changes_size": "1000", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_data_log_num_shards": "128", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_data_log_window": "30", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_data_notify_interval_msec": "0", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_data_sync_poll_interval": "20", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_data_sync_spawn_window": "20", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_debug_inject_set_olh_err": "0", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_default_data_log_backing": "fifo", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_default_realm_info_oid": "default.realm", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_default_region_info_oid": "default.region", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_default_zone_info_oid": "default.zone", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_defer_to_bucket_acls": "", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_delete_multi_obj_max_num": "1000", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_disable_s3select": "false", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_dmclock_admin_lim": "0.000000", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_dmclock_admin_res": "100.000000", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_dmclock_auth_lim": "0.000000", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_dmclock_auth_res": "200.000000", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_dmclock_data_lim": "0.000000", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_dmclock_data_res": "500.000000", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_dmclock_data_wgt": "500.000000", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_dmclock_metadata_res": "500.000000", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_dns_name": "", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_dns_s3website_name": "", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_dynamic_resharding": "true", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_enable_gc_threads": "true", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_enable_jwks_url_verification": "false", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_enable_lc_threads": "true", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_enable_mdsearch": "true", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_enable_ops_log": "false", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_enable_quota_threads": "true", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_enable_restore_threads": "true", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_enable_static_website": "false", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_enable_usage_log": "false", 2026-02-25T08:41:26.128 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_enforce_swift_acls": "true", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_exit_timeout_secs": "120", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_expose_bucket": "false", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_extended_http_attrs": "", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_filter": "none", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_frontends": "beast port=7480", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_gc_max_concurrent_io": "10", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_gc_max_deferred": "50", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_gc_max_objs": "32", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_gc_max_queue_size": "134213632", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_gc_max_trim_chunk": "16", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_gc_obj_min_wait": "7200", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_gc_processor_max_time": "3600", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_gc_processor_period": "3600", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_get_obj_max_req_size": "4194304", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_get_obj_window_size": "16777216", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_graceful_stop": "false", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_healthcheck_disabling_path": "", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_http_notif_connection_timeout": "5", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_http_notif_max_inflight": "8192", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_http_notif_message_timeout": "10", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_ignore_get_invalid_range": "false", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_init_timeout": "300", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_kafka_connection_idle": "300", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_kafka_message_timeout": "5000", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_kafka_sleep_timeout": "10", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_keystone_accepted_admin_roles": "", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_keystone_accepted_reader_roles": "", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_keystone_admin_domain": "", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_keystone_admin_password": "", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_keystone_admin_password_path": "", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_keystone_admin_project": "", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_keystone_admin_tenant": "", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_keystone_admin_user": "", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_keystone_barbican_domain": "", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_keystone_barbican_password": "", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_keystone_barbican_project": "", 2026-02-25T08:41:26.129 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_keystone_barbican_tenant": "", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_keystone_barbican_user": "", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_keystone_implicit_tenants": "false", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_keystone_service_token_enabled": "false", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_keystone_token_cache_size": "10000", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_keystone_token_cache_ttl": "300", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_keystone_url": "", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_keystone_verify_ssl": "true", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_lc_debug_interval": "-1", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_lc_list_cnt": "1000", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_lc_lock_max_time": "90", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_lc_max_objs": "32", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_lc_max_rules": "1000", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_lc_max_worker": "3", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_lc_max_wp_worker": "128", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_lc_ordered_list_threshold": "500", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_lc_thread_delay": "0", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_ldap_dnattr": "uid", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_ldap_searchfilter": "", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_ldap_uri": "", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_lfuda_sync_frequency": "60", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_list_bucket_min_readahead": "1000", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_list_buckets_max_chunk": "1000", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_log_http_headers": "", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_log_nonexistent_bucket": "false", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_log_object_name_utc": "false", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_lua_enable": "true", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_lua_max_memory_per_state": "128000", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_lua_max_runtime_per_state": "1000", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_max_attr_name_len": "0", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_max_attr_size": "0", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_max_attrs_num_in_req": "0", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_max_chunk_size": "4194304", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_max_concurrent_requests": "1024", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_max_control_aio": "8", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2026-02-25T08:41:26.130 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_max_dynamic_shards": "1999", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_max_listing_results": "5000", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_max_notify_retries": "10", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_max_objs_per_shard": "100000", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_max_put_param_size": "1048576", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_max_put_size": "5368709120", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_max_slo_entries": "1000", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_md_log_max_shards": "64", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_md_notify_interval_msec": "200", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_meta_sync_poll_interval": "20", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_meta_sync_spawn_window": "20", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_mime_types_file": "/etc/mime.types", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_mp_lock_max_time": "600", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_multi_obj_del_max_aio": "16", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_multipart_min_part_size": "5242880", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_multipart_part_upload_limit": "10000", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_nfs_fhcache_partitions": "3", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_nfs_fhcache_size": "2017", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_nfs_frontends": "rgw-nfs", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_nfs_lru_lanes": "5", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_nfs_max_gc": "300", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_nfs_namespace_expire_secs": "300", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_nfs_run_gc_threads": "false", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_nfs_run_lc_threads": "false", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_nfs_run_quota_threads": "false", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_nfs_run_restore_threads": "false", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_nfs_run_sync_thread": "false", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_nfs_s3_fast_attrs": "false", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_nfs_write_completion_interval_s": "10", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_num_async_rados_threads": "32", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_num_control_oids": "8", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_numa_node": "-1", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_obj_stripe_size": "4194304", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_obj_tombstone_cache_size": "1000", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_objexp_chunk_size": "100", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_objexp_gc_interval": "600", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_objexp_hints_num_shards": "127", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_olh_pending_timeout_sec": "3600", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_op_thread_suicide_timeout": "0", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_op_thread_timeout": "600", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_op_tracing": "false", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_opa_token": "", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_opa_url": "", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_opa_verify_ssl": "true", 2026-02-25T08:41:26.131 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_ops_log_data_backlog": "5242880", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_ops_log_rados": "false", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_ops_log_socket_path": "", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_override_bucket_index_max_shards": "0", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_parquet_buffer_size": "16777216", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_period_push_interval": "2.000000", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_period_push_interval_max": "30.000000", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_period_root_pool": ".rgw.root", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_policy_reject_invalid_principals": "true", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_posix_cache_lanes": "3", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_posix_cache_lmdb_count": "3", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_posix_cache_max_buckets": "100", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_posix_cache_partitions": "3", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_posix_userdb_dir": "/var/lib/ceph/radosgw", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_print_continue": "true", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_print_prohibited_content_length": "false", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_put_obj_max_window_size": "67108864", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_put_obj_min_window_size": "16777216", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_rados_pool_recovery_priority": "5", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_rados_tracing": "false", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_ratelimit_interval": "60", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_realm": "", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_realm_id": "", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_realm_root_pool": ".rgw.root", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_redis_connection_pool_size": "512", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_region": "", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_region_root_pool": ".rgw.root", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_relaxed_region_enforcement": "false", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_relaxed_s3_bucket_names": "false", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_request_uri": "", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_reshard_batch_size": "64", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_reshard_bucket_lock_duration": "360", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_reshard_debug_interval": "-1", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_reshard_max_aio": "128", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_reshard_num_logs": "16", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_reshard_progress_judge_interval": "120", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_reshard_progress_judge_ratio": "0.500000", 2026-02-25T08:41:26.132 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_reshard_thread_interval": "600", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_reshardlog_threshold": "30000", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_resolve_cname": "false", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_rest_getusage_op_compat": "false", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_restore_debug_interval": "-1", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_restore_lock_max_time": "90", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_restore_max_objs": "32", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_restore_processor_period": "900", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_run_sync_thread": "true", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_s3_auth_disable_signature_url": "false", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_s3_auth_order": "sts, external, local", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_s3_auth_use_keystone": "false", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_s3_auth_use_ldap": "false", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_s3_auth_use_rados": "true", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_s3_auth_use_sts": "false", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_s3_client_max_sig_ver": "-1", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_s3_success_create_obj_status": "0", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_safe_max_objects_per_shard": "102400", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_scheduler_type": "throttler", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_script_uri": "", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_service_provider_name": "", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_shard_warning_threshold": "90.000000", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_sts_client_id": "", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_sts_client_secret": "", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_sts_entry": "sts", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_sts_key": "", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_sts_max_session_duration": "43200", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_sts_min_session_duration": "900", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_sts_token_introspection_url": "", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_swift_account_in_url": "false", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_swift_auth_entry": "auth", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_swift_auth_url": "", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_swift_custom_header": "", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_swift_enforce_content_length": "false", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_swift_need_stats": "true", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_swift_tenant_name": "", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_swift_token_expiration": "86400", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_swift_url": "", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_swift_url_prefix": "swift", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_swift_versioning_enabled": "false", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_sync_lease_period": "120", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_sync_log_trim_interval": "1200", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_sync_log_trim_max_buckets": "16", 2026-02-25T08:41:26.133 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_sync_obj_etag_verify": "false", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_sync_trace_history_size": "4096", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_sync_trace_per_node_log_size": "32", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_thread_pool_size": "512", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_topic_persistency_max_retries": "0", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_topic_persistency_sleep_duration": "0", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_topic_persistency_time_to_live": "0", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_topic_require_publish_policy": "false", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_torrent_comment": "", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_torrent_createby": "", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_torrent_encoding": "", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_torrent_flag": "false", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_torrent_max_size": "5368709120", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_torrent_origin": "", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_torrent_sha_unit": "524288", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_torrent_tracker": "", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_trust_forwarded_https": "false", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_usage_log_flush_threshold": "1024", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_usage_log_key_transition": "true", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_usage_log_tick_interval": "30", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_usage_max_shards": "32", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_usage_max_user_shards": "1", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_use_opa_authz": "false", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_user_counters_cache": "false", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_user_counters_cache_size": "10000", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_user_default_quota_max_objects": "-1", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_user_default_quota_max_size": "-1", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_user_max_buckets": "1000", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_user_policies_max_num": "100", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_user_quota_sync_idle_users": "false", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_user_quota_sync_interval": "86400", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_user_quota_sync_wait_time": "86400", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_user_unique_email": "true", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_verify_ssl": "true", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_website_routing_rules_max_num": "50", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_zone": "", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_zone_id": "", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_zone_root_pool": ".rgw.root", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_zonegroup": "", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_zonegroup_id": "", 2026-02-25T08:41:26.134 INFO:teuthology.orchestra.run.trial023.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "rgwlc_auto_session_clear": "true", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "rgwlc_skip_bucket_step": "false", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "rocksdb_block_size": "4096", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "rocksdb_bloom_bits_per_key": "20", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "rocksdb_cache_row_ratio": "0.000000", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "rocksdb_cache_shard_bits": "4", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "rocksdb_cache_size": "536870912", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "rocksdb_cache_type": "binned_lru", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "rocksdb_cf_compact_on_deletion": "true", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "rocksdb_collect_compaction_stats": "false", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "rocksdb_collect_extended_stats": "false", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "rocksdb_collect_memory_stats": "false", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "rocksdb_delete_range_threshold": "1048576", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "rocksdb_index_type": "binary_search", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "rocksdb_log_to_ceph_log": "true", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "rocksdb_metadata_block_size": "4096", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "rocksdb_partition_filters": "false", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "rocksdb_perf": "false", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "rotating_keys_bootstrap_timeout": "30", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "rotating_keys_renewal_timeout": "10", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "run_dir": "/var/run/ceph", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "seastore_block_create": "true", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "seastore_cachepin_2q_in_ratio": "0.500000", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "seastore_cachepin_2q_out_ratio": "0.500000", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "seastore_cachepin_size_pershard": "2147483648", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "seastore_cachepin_type": "LRU", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "seastore_cbjournal_size": "5368709120", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "seastore_cold_tier_generations": "3", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "seastore_data_delta_based_overwrite": "0", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "seastore_default_max_object_size": "16777216", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "seastore_default_object_metadata_reservation": "16777216", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "seastore_device_size": "53687091200", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "seastore_disable_end_to_end_data_protection": "true", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "seastore_full_integrity_check": "false", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "seastore_hot_tier_generations": "5", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "seastore_journal_batch_capacity": "16", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "seastore_journal_batch_flush_size": "16777216", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "seastore_journal_iodepth_limit": "5", 2026-02-25T08:41:26.135 INFO:teuthology.orchestra.run.trial023.stdout: "seastore_main_device_type": "SSD", 2026-02-25T08:41:26.136 INFO:teuthology.orchestra.run.trial023.stdout: "seastore_max_concurrent_transactions": "128", 2026-02-25T08:41:26.136 INFO:teuthology.orchestra.run.trial023.stdout: "seastore_max_data_allocation_size": "0", 2026-02-25T08:41:26.136 INFO:teuthology.orchestra.run.trial023.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2026-02-25T08:41:26.136 INFO:teuthology.orchestra.run.trial023.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2026-02-25T08:41:26.136 INFO:teuthology.orchestra.run.trial023.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2026-02-25T08:41:26.136 INFO:teuthology.orchestra.run.trial023.stdout: "seastore_segment_cleaner_gc_formula": "cost_benefit", 2026-02-25T08:41:26.136 INFO:teuthology.orchestra.run.trial023.stdout: "seastore_segment_size": "67108864", 2026-02-25T08:41:26.136 INFO:teuthology.orchestra.run.trial023.stdout: "service_unique_id": "", 2026-02-25T08:41:26.136 INFO:teuthology.orchestra.run.trial023.stdout: "set_keepcaps": "false", 2026-02-25T08:41:26.136 INFO:teuthology.orchestra.run.trial023.stdout: "setgroup": "", 2026-02-25T08:41:26.136 INFO:teuthology.orchestra.run.trial023.stdout: "setuser": "", 2026-02-25T08:41:26.136 INFO:teuthology.orchestra.run.trial023.stdout: "setuser_match_path": "", 2026-02-25T08:41:26.136 INFO:teuthology.orchestra.run.trial023.stdout: "subv_metrics_window_interval": "30", 2026-02-25T08:41:26.136 INFO:teuthology.orchestra.run.trial023.stdout: "target_max_misplaced_ratio": "0.050000", 2026-02-25T08:41:26.136 INFO:teuthology.orchestra.run.trial023.stdout: "thp": "false", 2026-02-25T08:41:26.136 INFO:teuthology.orchestra.run.trial023.stdout: "threadpool_default_timeout": "60", 2026-02-25T08:41:26.136 INFO:teuthology.orchestra.run.trial023.stdout: "threadpool_empty_queue_max_wait": "2", 2026-02-25T08:41:26.136 INFO:teuthology.orchestra.run.trial023.stdout: "throttler_perf_counter": "true", 2026-02-25T08:41:26.136 INFO:teuthology.orchestra.run.trial023.stdout: "tmp_dir": "/tmp", 2026-02-25T08:41:26.136 INFO:teuthology.orchestra.run.trial023.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX", 2026-02-25T08:41:26.136 INFO:teuthology.orchestra.run.trial023.stdout: "uadk_compressor_enabled": "false", 2026-02-25T08:41:26.136 INFO:teuthology.orchestra.run.trial023.stdout: "uadk_wd_sync_ctx_num": "2" 2026-02-25T08:41:26.136 INFO:teuthology.orchestra.run.trial023.stdout:} 2026-02-25T08:41:26.137 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2026-02-25T08:41:26.210 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:26.208+0000 7f7243a5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:26.213 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:26.212+0000 7f7243a5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:26.221 INFO:teuthology.orchestra.run.trial023.stdout:{} 2026-02-25T08:41:26.222 INFO:teuthology.orchestra.run.trial023.stderr:mon_osd_down_out_interval = '' (not observed, change may require restart) 2026-02-25T08:41:26.228 DEBUG:teuthology.orchestra.run.trial023:> type ceph-objectstore-tool 2026-02-25T08:41:26.243 INFO:teuthology.orchestra.run.trial023.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2026-02-25T08:41:26.244 DEBUG:teuthology.orchestra.run.trial030:> type ceph-objectstore-tool 2026-02-25T08:41:26.262 INFO:teuthology.orchestra.run.trial030.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2026-02-25T08:41:26.262 INFO:teuthology.run_tasks:Running task rados... 2026-02-25T08:41:26.277 INFO:tasks.rados:Beginning rados... 2026-02-25T08:41:26.277 INFO:tasks.rados:config is {'clients': ['client.0'], 'objects': 50, 'op_weights': {'copy_from': 0, 'delete': 50, 'read': 100, 'rollback': 0, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000} 2026-02-25T08:41:26.277 INFO:tasks.rados:overrides is {'admin_socket': {'branch': 'wip-logstoremanager'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd target transaction size': 5, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Degraded data redundancy', 'is down', 'osds down', 'pg .*? is .*?degraded.*?, acting', 'pg .*? is stuck', 'pg degraded', 'PG_BACKFILL_FULL', 'Low space hindering backfill .*? backfill_toofull', 'OSD_ROOT_DOWN', 'pgs degraded', 'pgs undersized', 'is active.*backfill_toofull.*'], 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'c32469671c6b0222f786dd36c34eb702f780a11d'}} 2026-02-25T08:41:26.277 INFO:tasks.rados:config is {'clients': ['client.0'], 'objects': 50, 'op_weights': {'copy_from': 0, 'delete': 50, 'read': 100, 'rollback': 0, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000} 2026-02-25T08:41:26.277 DEBUG:teuthology.run_tasks:Unwinding manager rados 2026-02-25T08:41:26.290 INFO:tasks.rados:joining rados 2026-02-25T08:41:26.290 INFO:tasks.thrashosds.thrasher:starting do_thrash 2026-02-25T08:41:26.291 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2026-02-25T08:41:26.291 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-25T08:41:26.291 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:41:26.291 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2026-02-25T08:41:26.291 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:41:26.292 INFO:tasks.rados:clients are ['client.0', 'client.1'] 2026-02-25T08:41:26.292 INFO:tasks.rados:starting run 0 out of 1 2026-02-25T08:41:26.292 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2026-02-25T08:41:26.292 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2026-02-25T08:41:26.370 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:26.367+0000 7f11563f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:26.370 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:26.367+0000 7fbd737cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:26.372 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:26.367+0000 7f228fc5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:26.373 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:26.372+0000 7f11563f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:26.373 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:26.372+0000 7fbd737cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:26.375 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:26.374+0000 7f228fc5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:26.484 INFO:teuthology.orchestra.run.trial023.stdout: 2026-02-25T08:41:26.485 INFO:teuthology.orchestra.run.trial023.stderr:dumped all 2026-02-25T08:41:26.494 INFO:teuthology.orchestra.run.trial023.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2026-02-25T08:41:26.016544+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209715136,"kb_used":83890120,"kb_used_data":83890120,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":125825016,"statfs":{"total":214748299264,"available":128844816384,"internally_reserved":0,"allocated":85903482880,"data_stored":85903482880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.833410"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.191682+0000","last_change":"2026-02-25T08:41:23.195394+0000","last_active":"2026-02-25T08:41:23.191682+0000","last_peered":"2026-02-25T08:41:23.191682+0000","last_clean":"2026-02-25T08:41:23.191682+0000","last_became_active":"2026-02-25T08:41:21.223518+0000","last_became_peered":"2026-02-25T08:41:21.223518+0000","last_unstale":"2026-02-25T08:41:23.191682+0000","last_undegraded":"2026-02-25T08:41:23.191682+0000","last_fullsized":"2026-02-25T08:41:23.191682+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.190363+0000","last_change":"2026-02-25T08:41:23.192509+0000","last_active":"2026-02-25T08:41:23.190363+0000","last_peered":"2026-02-25T08:41:23.190363+0000","last_clean":"2026-02-25T08:41:23.190363+0000","last_became_active":"2026-02-25T08:41:21.240067+0000","last_became_peered":"2026-02-25T08:41:21.240067+0000","last_unstale":"2026-02-25T08:41:23.190363+0000","last_undegraded":"2026-02-25T08:41:23.190363+0000","last_fullsized":"2026-02-25T08:41:23.190363+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.190372+0000","last_change":"2026-02-25T08:41:23.195131+0000","last_active":"2026-02-25T08:41:23.190372+0000","last_peered":"2026-02-25T08:41:23.190372+0000","last_clean":"2026-02-25T08:41:23.190372+0000","last_became_active":"2026-02-25T08:41:21.226683+0000","last_became_peered":"2026-02-25T08:41:21.226683+0000","last_unstale":"2026-02-25T08:41:23.190372+0000","last_undegraded":"2026-02-25T08:41:23.190372+0000","last_fullsized":"2026-02-25T08:41:23.190372+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.190142+0000","last_change":"2026-02-25T08:41:23.192156+0000","last_active":"2026-02-25T08:41:23.190142+0000","last_peered":"2026-02-25T08:41:23.190142+0000","last_clean":"2026-02-25T08:41:23.190142+0000","last_became_active":"2026-02-25T08:41:21.216122+0000","last_became_peered":"2026-02-25T08:41:21.216122+0000","last_unstale":"2026-02-25T08:41:23.190142+0000","last_undegraded":"2026-02-25T08:41:23.190142+0000","last_fullsized":"2026-02-25T08:41:23.190142+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"13'52","reported_seq":82,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.189947+0000","last_change":"2026-02-25T08:41:21.235650+0000","last_active":"2026-02-25T08:41:23.189947+0000","last_peered":"2026-02-25T08:41:23.189947+0000","last_clean":"2026-02-25T08:41:23.189947+0000","last_became_active":"2026-02-25T08:41:21.226864+0000","last_became_peered":"2026-02-25T08:41:21.226864+0000","last_unstale":"2026-02-25T08:41:23.189947+0000","last_undegraded":"2026-02-25T08:41:23.189947+0000","last_fullsized":"2026-02-25T08:41:23.189947+0000","mapping_epoch":12,"log_start":"13'48","ondisk_log_start":"13'48","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":48,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.188719+0000","last_change":"2026-02-25T08:41:23.191189+0000","last_active":"2026-02-25T08:41:23.188719+0000","last_peered":"2026-02-25T08:41:23.188719+0000","last_clean":"2026-02-25T08:41:23.188719+0000","last_became_active":"2026-02-25T08:41:21.216133+0000","last_became_peered":"2026-02-25T08:41:21.216133+0000","last_unstale":"2026-02-25T08:41:23.188719+0000","last_undegraded":"2026-02-25T08:41:23.188719+0000","last_fullsized":"2026-02-25T08:41:23.188719+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"14'1","reported_seq":18,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.192134+0000","last_change":"2026-02-25T08:41:23.192134+0000","last_active":"2026-02-25T08:41:23.192134+0000","last_peered":"2026-02-25T08:41:23.192134+0000","last_clean":"2026-02-25T08:41:23.192134+0000","last_became_active":"2026-02-25T08:41:21.215058+0000","last_became_peered":"2026-02-25T08:41:21.215058+0000","last_unstale":"2026-02-25T08:41:23.192134+0000","last_undegraded":"2026-02-25T08:41:23.192134+0000","last_fullsized":"2026-02-25T08:41:23.192134+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.188725+0000","last_change":"2026-02-25T08:41:23.191858+0000","last_active":"2026-02-25T08:41:23.188725+0000","last_peered":"2026-02-25T08:41:23.188725+0000","last_clean":"2026-02-25T08:41:23.188725+0000","last_became_active":"2026-02-25T08:41:21.215894+0000","last_became_peered":"2026-02-25T08:41:21.215894+0000","last_unstale":"2026-02-25T08:41:23.188725+0000","last_undegraded":"2026-02-25T08:41:23.188725+0000","last_fullsized":"2026-02-25T08:41:23.188725+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:23.188713+0000","last_change":"2026-02-25T08:41:23.191742+0000","last_active":"2026-02-25T08:41:23.188713+0000","last_peered":"2026-02-25T08:41:23.188713+0000","last_clean":"2026-02-25T08:41:23.188713+0000","last_became_active":"2026-02-25T08:41:21.218886+0000","last_became_peered":"2026-02-25T08:41:21.218886+0000","last_unstale":"2026-02-25T08:41:23.188713+0000","last_undegraded":"2026-02-25T08:41:23.188713+0000","last_fullsized":"2026-02-25T08:41:23.188713+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:20.163293+0000","last_clean_scrub_stamp":"2026-02-25T08:41:20.163293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32210972672,"internally_reserved":0,"allocated":21476102144,"data_stored":21476102144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96633774080,"internally_reserved":0,"allocated":64427450368,"data_stored":64427450368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":24,"acting":24,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972724,"kb_used_data":20972724,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456060,"statfs":{"total":53687074816,"available":32211005440,"internally_reserved":0,"allocated":21476069376,"data_stored":21476069376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972688,"kb_used_data":20972688,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456096,"statfs":{"total":53687074816,"available":32211042304,"internally_reserved":0,"allocated":21476032512,"data_stored":21476032512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20971952,"kb_used_data":20971952,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456832,"statfs":{"total":53687074816,"available":32211795968,"internally_reserved":0,"allocated":21475278848,"data_stored":21475278848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972756,"kb_used_data":20972756,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456028,"statfs":{"total":53687074816,"available":32210972672,"internally_reserved":0,"allocated":21476102144,"data_stored":21476102144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53687074816,"available":32210972672,"internally_reserved":0,"allocated":21476102144,"data_stored":21476102144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53687074816,"available":32211795968,"internally_reserved":0,"allocated":21475278848,"data_stored":21475278848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":53687074816,"available":32211005440,"internally_reserved":0,"allocated":21476069376,"data_stored":21476069376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53687074816,"available":32210972672,"internally_reserved":0,"allocated":21476102144,"data_stored":21476102144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T08:41:27.376 INFO:teuthology.orchestra.run.trial023.stderr:pool 'unique_pool_0' created 2026-02-25T08:41:27.389 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2026-02-25T08:41:27.475 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:27.469+0000 7fedfd45a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:27.478 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:27.477+0000 7fedfd45a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:28.380 INFO:teuthology.orchestra.run.trial023.stderr:enabled application 'rados' on pool 'unique_pool_0' 2026-02-25T08:41:28.440 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:41:28.525 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:28.522+0000 7f5f7cd0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:28.527 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:28.526+0000 7f5f7cd0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:29.396 DEBUG:teuthology.orchestra.run.trial023:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0 2026-02-25T08:41:29.445 INFO:tasks.rados.rados.0.trial023.stdout:adding op weight read -> 100 2026-02-25T08:41:29.445 INFO:tasks.rados.rados.0.trial023.stdout:adding op weight write -> 50 2026-02-25T08:41:29.445 INFO:tasks.rados.rados.0.trial023.stdout:adding op weight delete -> 50 2026-02-25T08:41:29.445 INFO:tasks.rados.rados.0.trial023.stdout:adding op weight snap_create -> 50 2026-02-25T08:41:29.445 INFO:tasks.rados.rados.0.trial023.stdout:adding op weight snap_remove -> 50 2026-02-25T08:41:29.445 INFO:tasks.rados.rados.0.trial023.stdout:adding op weight write_excl -> 50 2026-02-25T08:41:29.446 INFO:tasks.rados.rados.0.trial023.stdout:ceph version 20.3.0-5338-g6aeb6975 (6aeb6975bc9a4790a399c9cc70dba9635d2a7819) tentacle (dev - Debug) 2026-02-25T08:41:29.446 INFO:tasks.rados.rados.0.trial023.stdout:Configuration: 2026-02-25T08:41:29.446 INFO:tasks.rados.rados.0.trial023.stdout: Number of operations: 4000 2026-02-25T08:41:29.446 INFO:tasks.rados.rados.0.trial023.stdout: Number of objects: 50 2026-02-25T08:41:29.446 INFO:tasks.rados.rados.0.trial023.stdout: Max in flight operations: 16 2026-02-25T08:41:29.446 INFO:tasks.rados.rados.0.trial023.stdout: Object size (in bytes): 4000000 2026-02-25T08:41:29.446 INFO:tasks.rados.rados.0.trial023.stdout: Write stride min: 400000 2026-02-25T08:41:29.446 INFO:tasks.rados.rados.0.trial023.stdout: Write stride max: 800000 2026-02-25T08:41:30.388 INFO:tasks.rados.rados.0.trial023.stdout:1: write initial oid 1 2026-02-25T08:41:30.388 INFO:tasks.rados.rados.0.trial023.stdout:1: seq_num 0 ranges {416807=475249,1542129=474678} 2026-02-25T08:41:30.392 INFO:tasks.rados.rados.0.trial023.stdout:1: writing trial02342524-1 from 416807 to 892056 tid 1 2026-02-25T08:41:30.396 INFO:tasks.rados.rados.0.trial023.stdout:1: writing trial02342524-1 from 1542129 to 2016807 tid 2 2026-02-25T08:41:30.396 INFO:tasks.rados.rados.0.trial023.stdout:2: write initial oid 2 2026-02-25T08:41:30.397 INFO:tasks.rados.rados.0.trial023.stdout:2: seq_num 1 ranges {416807=475249,1542129=474678} 2026-02-25T08:41:30.400 INFO:tasks.rados.rados.0.trial023.stdout:2: writing trial02342524-2 from 416807 to 892056 tid 1 2026-02-25T08:41:30.404 INFO:tasks.rados.rados.0.trial023.stdout:2: writing trial02342524-2 from 1542129 to 2016807 tid 2 2026-02-25T08:41:30.404 INFO:tasks.rados.rados.0.trial023.stdout:3: write initial oid 3 2026-02-25T08:41:30.405 INFO:tasks.rados.rados.0.trial023.stdout:3: seq_num 2 ranges {433614=550498,1600611=433003} 2026-02-25T08:41:30.409 INFO:tasks.rados.rados.0.trial023.stdout:3: writing trial02342524-3 from 433614 to 984112 tid 1 2026-02-25T08:41:30.414 INFO:tasks.rados.rados.0.trial023.stdout:3: writing trial02342524-3 from 1600611 to 2033614 tid 2 2026-02-25T08:41:30.414 INFO:tasks.rados.rados.0.trial023.stdout:4: write initial oid 4 2026-02-25T08:41:30.414 INFO:tasks.rados.rados.0.trial023.stdout:4: seq_num 3 ranges {450421=625747,1659093=391328} 2026-02-25T08:41:30.418 INFO:tasks.rados.rados.0.trial023.stdout:4: writing trial02342524-4 from 450421 to 1076168 tid 1 2026-02-25T08:41:30.422 INFO:tasks.rados.rados.0.trial023.stdout:4: writing trial02342524-4 from 1659093 to 2050421 tid 2 2026-02-25T08:41:30.422 INFO:tasks.rados.rados.0.trial023.stdout:5: write initial oid 5 2026-02-25T08:41:30.422 INFO:tasks.rados.rados.0.trial023.stdout:5: seq_num 4 ranges {467228=700996,1717575=349653} 2026-02-25T08:41:30.427 INFO:tasks.rados.rados.0.trial023.stdout:5: writing trial02342524-5 from 467228 to 1168224 tid 1 2026-02-25T08:41:30.430 INFO:tasks.rados.rados.0.trial023.stdout:5: writing trial02342524-5 from 1717575 to 2067228 tid 2 2026-02-25T08:41:30.431 INFO:tasks.rados.rados.0.trial023.stdout:6: write initial oid 6 2026-02-25T08:41:30.431 INFO:tasks.rados.rados.0.trial023.stdout:6: seq_num 5 ranges {484035=776245,2059704=24331} 2026-02-25T08:41:30.435 INFO:tasks.rados.rados.0.trial023.stdout:6: writing trial02342524-6 from 484035 to 1260280 tid 1 2026-02-25T08:41:30.438 INFO:tasks.rados.rados.0.trial023.stdout:6: writing trial02342524-6 from 2059704 to 2084035 tid 2 2026-02-25T08:41:30.438 INFO:tasks.rados.rados.0.trial023.stdout:7: write initial oid 7 2026-02-25T08:41:30.438 INFO:tasks.rados.rados.0.trial023.stdout:7: seq_num 6 ranges {500842=451494,1718186=382656} 2026-02-25T08:41:30.442 INFO:tasks.rados.rados.0.trial023.stdout:7: writing trial02342524-7 from 500842 to 952336 tid 1 2026-02-25T08:41:30.447 INFO:tasks.rados.rados.0.trial023.stdout:7: writing trial02342524-7 from 1718186 to 2100842 tid 2 2026-02-25T08:41:30.447 INFO:tasks.rados.rados.0.trial023.stdout:8: write initial oid 8 2026-02-25T08:41:30.447 INFO:tasks.rados.rados.0.trial023.stdout:8: seq_num 7 ranges {517649=526743,1776668=340981} 2026-02-25T08:41:30.451 INFO:tasks.rados.rados.0.trial023.stdout:8: writing trial02342524-8 from 517649 to 1044392 tid 1 2026-02-25T08:41:30.457 INFO:tasks.rados.rados.0.trial023.stdout:8: writing trial02342524-8 from 1776668 to 2117649 tid 2 2026-02-25T08:41:30.457 INFO:tasks.rados.rados.0.trial023.stdout:9: write initial oid 9 2026-02-25T08:41:30.457 INFO:tasks.rados.rados.0.trial023.stdout:9: seq_num 8 ranges {534456=718345,1951503=182953} 2026-02-25T08:41:30.461 INFO:tasks.rados.rados.0.trial023.stdout:9: writing trial02342524-9 from 534456 to 1252801 tid 1 2026-02-25T08:41:30.465 INFO:tasks.rados.rados.0.trial023.stdout:9: writing trial02342524-9 from 1951503 to 2134456 tid 2 2026-02-25T08:41:30.465 INFO:tasks.rados.rados.0.trial023.stdout:10: write initial oid 10 2026-02-25T08:41:30.465 INFO:tasks.rados.rados.0.trial023.stdout:10: seq_num 9 ranges {551263=793594,1893632=257631} 2026-02-25T08:41:30.470 INFO:tasks.rados.rados.0.trial023.stdout:10: writing trial02342524-10 from 551263 to 1344857 tid 1 2026-02-25T08:41:30.474 INFO:tasks.rados.rados.0.trial023.stdout:10: writing trial02342524-10 from 1893632 to 2151263 tid 2 2026-02-25T08:41:30.474 INFO:tasks.rados.rados.0.trial023.stdout:11: write initial oid 11 2026-02-25T08:41:30.474 INFO:tasks.rados.rados.0.trial023.stdout:11: seq_num 10 ranges {568070=468843,1552114=615956} 2026-02-25T08:41:30.478 INFO:tasks.rados.rados.0.trial023.stdout:11: writing trial02342524-11 from 568070 to 1036913 tid 1 2026-02-25T08:41:30.482 INFO:tasks.rados.rados.0.trial023.stdout:11: writing trial02342524-11 from 1552114 to 2168070 tid 2 2026-02-25T08:41:30.482 INFO:tasks.rados.rados.0.trial023.stdout:12: write initial oid 12 2026-02-25T08:41:30.482 INFO:tasks.rados.rados.0.trial023.stdout:12: seq_num 11 ranges {584877=544092,1610596=562003} 2026-02-25T08:41:30.487 INFO:tasks.rados.rados.0.trial023.stdout:12: writing trial02342524-12 from 584877 to 1128969 tid 1 2026-02-25T08:41:30.491 INFO:tasks.rados.rados.0.trial023.stdout:12: writing trial02342524-12 from 1610596 to 2172599 tid 2 2026-02-25T08:41:30.491 INFO:tasks.rados.rados.0.trial023.stdout:13: write initial oid 13 2026-02-25T08:41:30.491 INFO:tasks.rados.rados.0.trial023.stdout:13: seq_num 12 ranges {601684=619341,1669078=532606} 2026-02-25T08:41:30.496 INFO:tasks.rados.rados.0.trial023.stdout:13: writing trial02342524-13 from 601684 to 1221025 tid 1 2026-02-25T08:41:30.500 INFO:tasks.rados.rados.0.trial023.stdout:13: writing trial02342524-13 from 1669078 to 2201684 tid 2 2026-02-25T08:41:30.500 INFO:tasks.rados.rados.0.trial023.stdout:14: write initial oid 14 2026-02-25T08:41:30.500 INFO:tasks.rados.rados.0.trial023.stdout:14: seq_num 13 ranges {618491=694590,2011207=207284} 2026-02-25T08:41:30.505 INFO:tasks.rados.rados.0.trial023.stdout:14: writing trial02342524-14 from 618491 to 1313081 tid 1 2026-02-25T08:41:30.509 INFO:tasks.rados.rados.0.trial023.stdout:14: writing trial02342524-14 from 2011207 to 2218491 tid 2 2026-02-25T08:41:30.509 INFO:tasks.rados.rados.0.trial023.stdout:15: write initial oid 15 2026-02-25T08:41:30.509 INFO:tasks.rados.rados.0.trial023.stdout:15: seq_num 14 ranges {635298=769839,2069689=165609} 2026-02-25T08:41:30.514 INFO:tasks.rados.rados.0.trial023.stdout:15: writing trial02342524-15 from 635298 to 1405137 tid 1 2026-02-25T08:41:30.518 INFO:tasks.rados.rados.0.trial023.stdout:15: writing trial02342524-15 from 2069689 to 2235298 tid 2 2026-02-25T08:41:30.518 INFO:tasks.rados.rados.0.trial023.stdout:16: write initial oid 16 2026-02-25T08:41:30.518 INFO:tasks.rados.rados.0.trial023.stdout:16: seq_num 15 ranges {652105=445088,1728171=452988} 2026-02-25T08:41:30.521 INFO:tasks.rados.rados.0.trial023.stdout:16: writing trial02342524-16 from 652105 to 1097193 tid 1 2026-02-25T08:41:30.527 INFO:tasks.rados.rados.0.trial023.stdout:16: writing trial02342524-16 from 1728171 to 2181159 tid 2 2026-02-25T08:41:30.527 INFO:tasks.rados.rados.0.trial023.stdout: waiting on 16 2026-02-25T08:41:30.527 INFO:tasks.rados.rados.0.trial023.stdout:1: finishing write tid 1 to trial02342524-1 2026-02-25T08:41:30.527 INFO:tasks.rados.rados.0.trial023.stdout:3: finishing write tid 1 to trial02342524-3 2026-02-25T08:41:30.527 INFO:tasks.rados.rados.0.trial023.stdout:1: finishing write tid 2 to trial02342524-1 2026-02-25T08:41:30.527 INFO:tasks.rados.rados.0.trial023.stdout:3: finishing write tid 2 to trial02342524-3 2026-02-25T08:41:30.527 INFO:tasks.rados.rados.0.trial023.stdout:5: finishing write tid 1 to trial02342524-5 2026-02-25T08:41:30.527 INFO:tasks.rados.rados.0.trial023.stdout:3: finishing write tid 3 to trial02342524-3 2026-02-25T08:41:30.527 INFO:tasks.rados.rados.0.trial023.stdout:3: finishing write tid 4 to trial02342524-3 2026-02-25T08:41:30.527 INFO:tasks.rados.rados.0.trial023.stdout:3: oid 3 updating version 0 to 1 2026-02-25T08:41:30.528 INFO:tasks.rados.rados.0.trial023.stdout:3: oid 3 updating version 1 to 3 2026-02-25T08:41:30.528 INFO:tasks.rados.rados.0.trial023.stdout:3: oid 3 version 3 is already newer than 3 2026-02-25T08:41:30.528 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2026-02-25T08:41:30.528 INFO:tasks.rados.rados.0.trial023.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2026-02-25T08:41:30.528 INFO:tasks.rados.rados.0.trial023.stdout:1: finishing write tid 3 to trial02342524-1 2026-02-25T08:41:30.528 INFO:tasks.rados.rados.0.trial023.stdout:5: finishing write tid 2 to trial02342524-5 2026-02-25T08:41:30.528 INFO:tasks.rados.rados.0.trial023.stdout:4: finishing write tid 1 to trial02342524-4 2026-02-25T08:41:30.528 INFO:tasks.rados.rados.0.trial023.stdout:3: done (15 left) 2026-02-25T08:41:30.528 INFO:tasks.rados.rados.0.trial023.stdout:17: write initial oid 17 2026-02-25T08:41:30.528 INFO:tasks.rados.rados.0.trial023.stdout:17: seq_num 16 ranges {668912=636690,1903006=365906} 2026-02-25T08:41:30.531 INFO:tasks.rados.rados.0.trial023.stdout:17: writing trial02342524-17 from 668912 to 1305602 tid 1 2026-02-25T08:41:30.535 INFO:tasks.rados.rados.0.trial023.stdout:17: writing trial02342524-17 from 1903006 to 2268912 tid 2 2026-02-25T08:41:30.535 INFO:tasks.rados.rados.0.trial023.stdout: waiting on 16 2026-02-25T08:41:30.535 INFO:tasks.rados.rados.0.trial023.stdout:5: finishing write tid 3 to trial02342524-5 2026-02-25T08:41:30.535 INFO:tasks.rados.rados.0.trial023.stdout:5: finishing write tid 4 to trial02342524-5 2026-02-25T08:41:30.536 INFO:tasks.rados.rados.0.trial023.stdout:5: oid 5 updating version 0 to 1 2026-02-25T08:41:30.536 INFO:tasks.rados.rados.0.trial023.stdout:5: oid 5 updating version 1 to 3 2026-02-25T08:41:30.536 INFO:tasks.rados.rados.0.trial023.stdout:5: oid 5 version 3 is already newer than 3 2026-02-25T08:41:30.536 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2026-02-25T08:41:30.536 INFO:tasks.rados.rados.0.trial023.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2026-02-25T08:41:30.536 INFO:tasks.rados.rados.0.trial023.stdout:1: finishing write tid 4 to trial02342524-1 2026-02-25T08:41:30.536 INFO:tasks.rados.rados.0.trial023.stdout:1: oid 1 updating version 0 to 3 2026-02-25T08:41:30.536 INFO:tasks.rados.rados.0.trial023.stdout:1: oid 1 version 3 is already newer than 3 2026-02-25T08:41:30.536 INFO:tasks.rados.rados.0.trial023.stdout:1: oid 1 version 3 is already newer than 1 2026-02-25T08:41:30.536 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2026-02-25T08:41:30.536 INFO:tasks.rados.rados.0.trial023.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2026-02-25T08:41:30.536 INFO:tasks.rados.rados.0.trial023.stdout:8: finishing write tid 1 to trial02342524-8 2026-02-25T08:41:30.537 INFO:tasks.rados.rados.0.trial023.stdout:1: done (15 left) 2026-02-25T08:41:30.537 INFO:tasks.rados.rados.0.trial023.stdout:5: done (14 left) 2026-02-25T08:41:30.537 INFO:tasks.rados.rados.0.trial023.stdout:18: write initial oid 18 2026-02-25T08:41:30.537 INFO:tasks.rados.rados.0.trial023.stdout:18: seq_num 17 ranges {685719=711939,1845135=440584} 2026-02-25T08:41:30.540 INFO:tasks.rados.rados.0.trial023.stdout:18: writing trial02342524-18 from 685719 to 1397658 tid 1 2026-02-25T08:41:30.544 INFO:tasks.rados.rados.0.trial023.stdout:18: writing trial02342524-18 from 1845135 to 2285719 tid 2 2026-02-25T08:41:30.544 INFO:tasks.rados.rados.0.trial023.stdout:19: write initial oid 19 2026-02-25T08:41:30.544 INFO:tasks.rados.rados.0.trial023.stdout:19: seq_num 18 ranges {702526=787188,1903617=398909} 2026-02-25T08:41:30.550 INFO:tasks.rados.rados.0.trial023.stdout:19: writing trial02342524-19 from 702526 to 1489714 tid 1 2026-02-25T08:41:30.553 INFO:tasks.rados.rados.0.trial023.stdout:19: writing trial02342524-19 from 1903617 to 2302526 tid 2 2026-02-25T08:41:30.554 INFO:tasks.rados.rados.0.trial023.stdout: waiting on 16 2026-02-25T08:41:30.554 INFO:tasks.rados.rados.0.trial023.stdout:10: finishing write tid 1 to trial02342524-10 2026-02-25T08:41:30.554 INFO:tasks.rados.rados.0.trial023.stdout:4: finishing write tid 2 to trial02342524-4 2026-02-25T08:41:30.554 INFO:tasks.rados.rados.0.trial023.stdout:7: finishing write tid 1 to trial02342524-7 2026-02-25T08:41:30.554 INFO:tasks.rados.rados.0.trial023.stdout:10: finishing write tid 2 to trial02342524-10 2026-02-25T08:41:30.554 INFO:tasks.rados.rados.0.trial023.stdout:4: finishing write tid 3 to trial02342524-4 2026-02-25T08:41:30.554 INFO:tasks.rados.rados.0.trial023.stdout:9: finishing write tid 1 to trial02342524-9 2026-02-25T08:41:30.554 INFO:tasks.rados.rados.0.trial023.stdout:4: finishing write tid 4 to trial02342524-4 2026-02-25T08:41:30.554 INFO:tasks.rados.rados.0.trial023.stdout:4: oid 4 updating version 0 to 4 2026-02-25T08:41:30.554 INFO:tasks.rados.rados.0.trial023.stdout:4: oid 4 updating version 4 to 6 2026-02-25T08:41:30.555 INFO:tasks.rados.rados.0.trial023.stdout:4: oid 4 version 6 is already newer than 6 2026-02-25T08:41:30.555 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 4 v 6 (ObjNum 3 snap 0 seq_num 3) dirty exists 2026-02-25T08:41:30.555 INFO:tasks.rados.rados.0.trial023.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2026-02-25T08:41:30.555 INFO:tasks.rados.rados.0.trial023.stdout:10: finishing write tid 3 to trial02342524-10 2026-02-25T08:41:30.555 INFO:tasks.rados.rados.0.trial023.stdout:11: finishing write tid 1 to trial02342524-11 2026-02-25T08:41:30.555 INFO:tasks.rados.rados.0.trial023.stdout:4: done (15 left) 2026-02-25T08:41:30.555 INFO:tasks.rados.rados.0.trial023.stdout:20: write initial oid 20 2026-02-25T08:41:30.555 INFO:tasks.rados.rados.0.trial023.stdout:20: seq_num 19 ranges {719333=462437,1962099=357234} 2026-02-25T08:41:30.557 INFO:tasks.rados.rados.0.trial023.stdout:20: writing trial02342524-20 from 719333 to 1181770 tid 1 2026-02-25T08:41:30.563 INFO:tasks.rados.rados.0.trial023.stdout:20: writing trial02342524-20 from 1962099 to 2319333 tid 2 2026-02-25T08:41:30.563 INFO:tasks.rados.rados.0.trial023.stdout: waiting on 16 2026-02-25T08:41:30.563 INFO:tasks.rados.rados.0.trial023.stdout:10: finishing write tid 4 to trial02342524-10 2026-02-25T08:41:30.563 INFO:tasks.rados.rados.0.trial023.stdout:10: oid 10 updating version 0 to 3 2026-02-25T08:41:30.563 INFO:tasks.rados.rados.0.trial023.stdout:10: oid 10 version 3 is already newer than 3 2026-02-25T08:41:30.563 INFO:tasks.rados.rados.0.trial023.stdout:10: oid 10 version 3 is already newer than 1 2026-02-25T08:41:30.563 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 10 v 3 (ObjNum 9 snap 0 seq_num 9) dirty exists 2026-02-25T08:41:30.563 INFO:tasks.rados.rados.0.trial023.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2026-02-25T08:41:30.563 INFO:tasks.rados.rados.0.trial023.stdout:8: finishing write tid 2 to trial02342524-8 2026-02-25T08:41:30.563 INFO:tasks.rados.rados.0.trial023.stdout:8: finishing write tid 3 to trial02342524-8 2026-02-25T08:41:30.564 INFO:tasks.rados.rados.0.trial023.stdout:8: finishing write tid 4 to trial02342524-8 2026-02-25T08:41:30.564 INFO:tasks.rados.rados.0.trial023.stdout:8: oid 8 updating version 0 to 1 2026-02-25T08:41:30.564 INFO:tasks.rados.rados.0.trial023.stdout:8: oid 8 updating version 1 to 3 2026-02-25T08:41:30.564 INFO:tasks.rados.rados.0.trial023.stdout:8: oid 8 version 3 is already newer than 3 2026-02-25T08:41:30.564 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 8 v 3 (ObjNum 7 snap 0 seq_num 7) dirty exists 2026-02-25T08:41:30.564 INFO:tasks.rados.rados.0.trial023.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2026-02-25T08:41:30.564 INFO:tasks.rados.rados.0.trial023.stdout:15: finishing write tid 1 to trial02342524-15 2026-02-25T08:41:30.564 INFO:tasks.rados.rados.0.trial023.stdout:2: finishing write tid 1 to trial02342524-2 2026-02-25T08:41:30.564 INFO:tasks.rados.rados.0.trial023.stdout:9: finishing write tid 2 to trial02342524-9 2026-02-25T08:41:30.564 INFO:tasks.rados.rados.0.trial023.stdout:13: finishing write tid 1 to trial02342524-13 2026-02-25T08:41:30.564 INFO:tasks.rados.rados.0.trial023.stdout:8: done (15 left) 2026-02-25T08:41:30.564 INFO:tasks.rados.rados.0.trial023.stdout:10: done (14 left) 2026-02-25T08:41:30.564 INFO:tasks.rados.rados.0.trial023.stdout:21: write initial oid 21 2026-02-25T08:41:30.565 INFO:tasks.rados.rados.0.trial023.stdout:16: finishing write tid 1 to trial02342524-16 2026-02-25T08:41:30.565 INFO:tasks.rados.rados.0.trial023.stdout:9: finishing write tid 3 to trial02342524-9 2026-02-25T08:41:30.565 INFO:tasks.rados.rados.0.trial023.stdout:9: finishing write tid 4 to trial02342524-9 2026-02-25T08:41:30.565 INFO:tasks.rados.rados.0.trial023.stdout:9: oid 9 updating version 0 to 3 2026-02-25T08:41:30.565 INFO:tasks.rados.rados.0.trial023.stdout:9: oid 9 version 3 is already newer than 1 2026-02-25T08:41:30.565 INFO:tasks.rados.rados.0.trial023.stdout:9: oid 9 version 3 is already newer than 3 2026-02-25T08:41:30.565 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 9 v 3 (ObjNum 8 snap 0 seq_num 8) dirty exists 2026-02-25T08:41:30.565 INFO:tasks.rados.rados.0.trial023.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2026-02-25T08:41:30.565 INFO:tasks.rados.rados.0.trial023.stdout:21: seq_num 20 ranges {736140=537686,2020581=315559} 2026-02-25T08:41:30.567 INFO:tasks.rados.rados.0.trial023.stdout:21: writing trial02342524-21 from 736140 to 1273826 tid 1 2026-02-25T08:41:30.571 INFO:tasks.rados.rados.0.trial023.stdout:21: writing trial02342524-21 from 2020581 to 2336140 tid 2 2026-02-25T08:41:30.571 INFO:tasks.rados.rados.0.trial023.stdout:9: done (14 left) 2026-02-25T08:41:30.571 INFO:tasks.rados.rados.0.trial023.stdout:22: write initial oid 22 2026-02-25T08:41:30.571 INFO:tasks.rados.rados.0.trial023.stdout:22: seq_num 21 ranges {752947=612935,1962710=390237} 2026-02-25T08:41:30.576 INFO:tasks.rados.rados.0.trial023.stdout:22: writing trial02342524-22 from 752947 to 1365882 tid 1 2026-02-25T08:41:30.580 INFO:tasks.rados.rados.0.trial023.stdout:22: writing trial02342524-22 from 1962710 to 2352947 tid 2 2026-02-25T08:41:30.580 INFO:tasks.rados.rados.0.trial023.stdout:23: write initial oid 23 2026-02-25T08:41:30.580 INFO:tasks.rados.rados.0.trial023.stdout:23: seq_num 22 ranges {769754=688184,2021192=348562} 2026-02-25T08:41:30.585 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:41:30.586 INFO:tasks.rados.rados.0.trial023.stdout:23: writing trial02342524-23 from 769754 to 1457938 tid 1 2026-02-25T08:41:30.589 INFO:tasks.rados.rados.0.trial023.stdout:23: writing trial02342524-23 from 2021192 to 2369754 tid 2 2026-02-25T08:41:30.589 INFO:tasks.rados.rados.0.trial023.stdout: waiting on 16 2026-02-25T08:41:30.589 INFO:tasks.rados.rados.0.trial023.stdout:11: finishing write tid 2 to trial02342524-11 2026-02-25T08:41:30.589 INFO:tasks.rados.rados.0.trial023.stdout:13: finishing write tid 2 to trial02342524-13 2026-02-25T08:41:30.589 INFO:tasks.rados.rados.0.trial023.stdout:13: finishing write tid 3 to trial02342524-13 2026-02-25T08:41:30.589 INFO:tasks.rados.rados.0.trial023.stdout:17: finishing write tid 1 to trial02342524-17 2026-02-25T08:41:30.589 INFO:tasks.rados.rados.0.trial023.stdout:13: finishing write tid 4 to trial02342524-13 2026-02-25T08:41:30.589 INFO:tasks.rados.rados.0.trial023.stdout:13: oid 13 updating version 0 to 6 2026-02-25T08:41:30.589 INFO:tasks.rados.rados.0.trial023.stdout:13: oid 13 version 6 is already newer than 6 2026-02-25T08:41:30.589 INFO:tasks.rados.rados.0.trial023.stdout:13: oid 13 version 6 is already newer than 4 2026-02-25T08:41:30.589 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 13 v 6 (ObjNum 12 snap 0 seq_num 12) dirty exists 2026-02-25T08:41:30.589 INFO:tasks.rados.rados.0.trial023.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2026-02-25T08:41:30.589 INFO:tasks.rados.rados.0.trial023.stdout:15: finishing write tid 2 to trial02342524-15 2026-02-25T08:41:30.589 INFO:tasks.rados.rados.0.trial023.stdout:7: finishing write tid 2 to trial02342524-7 2026-02-25T08:41:30.589 INFO:tasks.rados.rados.0.trial023.stdout:15: finishing write tid 3 to trial02342524-15 2026-02-25T08:41:30.589 INFO:tasks.rados.rados.0.trial023.stdout:13: done (15 left) 2026-02-25T08:41:30.590 INFO:tasks.rados.rados.0.trial023.stdout:24: write initial oid 24 2026-02-25T08:41:30.590 INFO:tasks.rados.rados.0.trial023.stdout:24: seq_num 23 ranges {786561=479786,1796027=467664} 2026-02-25T08:41:30.594 INFO:tasks.rados.rados.0.trial023.stdout:24: writing trial02342524-24 from 786561 to 1266347 tid 1 2026-02-25T08:41:30.598 INFO:tasks.rados.rados.0.trial023.stdout:24: writing trial02342524-24 from 1796027 to 2263691 tid 2 2026-02-25T08:41:30.598 INFO:tasks.rados.rados.0.trial023.stdout: waiting on 16 2026-02-25T08:41:30.598 INFO:tasks.rados.rados.0.trial023.stdout:15: finishing write tid 4 to trial02342524-15 2026-02-25T08:41:30.598 INFO:tasks.rados.rados.0.trial023.stdout:15: oid 15 updating version 0 to 1 2026-02-25T08:41:30.598 INFO:tasks.rados.rados.0.trial023.stdout:15: oid 15 updating version 1 to 3 2026-02-25T08:41:30.598 INFO:tasks.rados.rados.0.trial023.stdout:15: oid 15 version 3 is already newer than 3 2026-02-25T08:41:30.598 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 15 v 3 (ObjNum 14 snap 0 seq_num 14) dirty exists 2026-02-25T08:41:30.598 INFO:tasks.rados.rados.0.trial023.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2026-02-25T08:41:30.598 INFO:tasks.rados.rados.0.trial023.stdout:16: finishing write tid 2 to trial02342524-16 2026-02-25T08:41:30.598 INFO:tasks.rados.rados.0.trial023.stdout:11: finishing write tid 3 to trial02342524-11 2026-02-25T08:41:30.598 INFO:tasks.rados.rados.0.trial023.stdout:15: done (15 left) 2026-02-25T08:41:30.598 INFO:tasks.rados.rados.0.trial023.stdout:25: write initial oid 25 2026-02-25T08:41:30.598 INFO:tasks.rados.rados.0.trial023.stdout:25: seq_num 24 ranges {403368=555035,1454509=727675} 2026-02-25T08:41:30.601 INFO:tasks.rados.rados.0.trial023.stdout:25: writing trial02342524-25 from 403368 to 958403 tid 1 2026-02-25T08:41:30.606 INFO:tasks.rados.rados.0.trial023.stdout:25: writing trial02342524-25 from 1454509 to 2182184 tid 2 2026-02-25T08:41:30.606 INFO:tasks.rados.rados.0.trial023.stdout: waiting on 16 2026-02-25T08:41:30.606 INFO:tasks.rados.rados.0.trial023.stdout:11: finishing write tid 4 to trial02342524-11 2026-02-25T08:41:30.606 INFO:tasks.rados.rados.0.trial023.stdout:11: oid 11 updating version 0 to 7 2026-02-25T08:41:30.606 INFO:tasks.rados.rados.0.trial023.stdout:11: oid 11 updating version 7 to 9 2026-02-25T08:41:30.606 INFO:tasks.rados.rados.0.trial023.stdout:11: oid 11 version 9 is already newer than 9 2026-02-25T08:41:30.606 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 11 v 9 (ObjNum 10 snap 0 seq_num 10) dirty exists 2026-02-25T08:41:30.606 INFO:tasks.rados.rados.0.trial023.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2026-02-25T08:41:30.606 INFO:tasks.rados.rados.0.trial023.stdout:11: done (15 left) 2026-02-25T08:41:30.607 INFO:tasks.rados.rados.0.trial023.stdout:26: write initial oid 26 2026-02-25T08:41:30.607 INFO:tasks.rados.rados.0.trial023.stdout:26: seq_num 25 ranges {420175=630284,1796638=471333} 2026-02-25T08:41:30.610 INFO:tasks.rados.rados.0.trial023.stdout:26: writing trial02342524-26 from 420175 to 1050459 tid 1 2026-02-25T08:41:30.615 INFO:tasks.rados.rados.0.trial023.stdout:26: writing trial02342524-26 from 1796638 to 2267971 tid 2 2026-02-25T08:41:30.615 INFO:tasks.rados.rados.0.trial023.stdout: waiting on 16 2026-02-25T08:41:30.615 INFO:tasks.rados.rados.0.trial023.stdout:16: finishing write tid 3 to trial02342524-16 2026-02-25T08:41:30.615 INFO:tasks.rados.rados.0.trial023.stdout:7: finishing write tid 3 to trial02342524-7 2026-02-25T08:41:30.615 INFO:tasks.rados.rados.0.trial023.stdout:16: finishing write tid 4 to trial02342524-16 2026-02-25T08:41:30.615 INFO:tasks.rados.rados.0.trial023.stdout:16: oid 16 updating version 0 to 1 2026-02-25T08:41:30.615 INFO:tasks.rados.rados.0.trial023.stdout:16: oid 16 updating version 1 to 3 2026-02-25T08:41:30.615 INFO:tasks.rados.rados.0.trial023.stdout:16: oid 16 version 3 is already newer than 3 2026-02-25T08:41:30.615 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 16 v 3 (ObjNum 15 snap 0 seq_num 15) dirty exists 2026-02-25T08:41:30.615 INFO:tasks.rados.rados.0.trial023.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2026-02-25T08:41:30.615 INFO:tasks.rados.rados.0.trial023.stdout:23: finishing write tid 1 to trial02342524-23 2026-02-25T08:41:30.615 INFO:tasks.rados.rados.0.trial023.stdout:16: done (15 left) 2026-02-25T08:41:30.615 INFO:tasks.rados.rados.0.trial023.stdout:27: write initial oid 27 2026-02-25T08:41:30.616 INFO:tasks.rados.rados.0.trial023.stdout:27: seq_num 26 ranges {436982=705533,1855120=581862} 2026-02-25T08:41:30.620 INFO:tasks.rados.rados.0.trial023.stdout:27: writing trial02342524-27 from 436982 to 1142515 tid 1 2026-02-25T08:41:30.626 INFO:tasks.rados.rados.0.trial023.stdout:27: writing trial02342524-27 from 1855120 to 2436982 tid 2 2026-02-25T08:41:30.626 INFO:tasks.rados.rados.0.trial023.stdout: waiting on 16 2026-02-25T08:41:30.626 INFO:tasks.rados.rados.0.trial023.stdout:18: finishing write tid 1 to trial02342524-18 2026-02-25T08:41:30.626 INFO:tasks.rados.rados.0.trial023.stdout:17: finishing write tid 2 to trial02342524-17 2026-02-25T08:41:30.627 INFO:tasks.rados.rados.0.trial023.stdout:7: finishing write tid 4 to trial02342524-7 2026-02-25T08:41:30.627 INFO:tasks.rados.rados.0.trial023.stdout:7: oid 7 updating version 0 to 3 2026-02-25T08:41:30.627 INFO:tasks.rados.rados.0.trial023.stdout:7: oid 7 version 3 is already newer than 1 2026-02-25T08:41:30.627 INFO:tasks.rados.rados.0.trial023.stdout:7: oid 7 version 3 is already newer than 3 2026-02-25T08:41:30.627 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 7 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2026-02-25T08:41:30.627 INFO:tasks.rados.rados.0.trial023.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2026-02-25T08:41:30.627 INFO:tasks.rados.rados.0.trial023.stdout:23: finishing write tid 2 to trial02342524-23 2026-02-25T08:41:30.627 INFO:tasks.rados.rados.0.trial023.stdout:7: done (15 left) 2026-02-25T08:41:30.627 INFO:tasks.rados.rados.0.trial023.stdout:28: write initial oid 28 2026-02-25T08:41:30.627 INFO:tasks.rados.rados.0.trial023.stdout:28: seq_num 27 ranges {453789=780782,1913602=475002} 2026-02-25T08:41:30.630 INFO:tasks.rados.rados.0.trial023.stdout:28: writing trial02342524-28 from 453789 to 1234571 tid 1 2026-02-25T08:41:30.635 INFO:tasks.rados.rados.0.trial023.stdout:28: writing trial02342524-28 from 1913602 to 2388604 tid 2 2026-02-25T08:41:30.635 INFO:tasks.rados.rados.0.trial023.stdout: waiting on 16 2026-02-25T08:41:30.635 INFO:tasks.rados.rados.0.trial023.stdout:17: finishing write tid 3 to trial02342524-17 2026-02-25T08:41:30.635 INFO:tasks.rados.rados.0.trial023.stdout:23: finishing write tid 3 to trial02342524-23 2026-02-25T08:41:30.636 INFO:tasks.rados.rados.0.trial023.stdout:22: finishing write tid 1 to trial02342524-22 2026-02-25T08:41:30.636 INFO:tasks.rados.rados.0.trial023.stdout:17: finishing write tid 4 to trial02342524-17 2026-02-25T08:41:30.636 INFO:tasks.rados.rados.0.trial023.stdout:17: oid 17 updating version 0 to 3 2026-02-25T08:41:30.636 INFO:tasks.rados.rados.0.trial023.stdout:17: oid 17 version 3 is already newer than 3 2026-02-25T08:41:30.636 INFO:tasks.rados.rados.0.trial023.stdout:17: oid 17 version 3 is already newer than 1 2026-02-25T08:41:30.636 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 17 v 3 (ObjNum 16 snap 0 seq_num 16) dirty exists 2026-02-25T08:41:30.636 INFO:tasks.rados.rados.0.trial023.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2026-02-25T08:41:30.636 INFO:tasks.rados.rados.0.trial023.stdout:23: finishing write tid 4 to trial02342524-23 2026-02-25T08:41:30.636 INFO:tasks.rados.rados.0.trial023.stdout:23: oid 23 updating version 0 to 7 2026-02-25T08:41:30.636 INFO:tasks.rados.rados.0.trial023.stdout:23: oid 23 updating version 7 to 9 2026-02-25T08:41:30.636 INFO:tasks.rados.rados.0.trial023.stdout:23: oid 23 version 9 is already newer than 9 2026-02-25T08:41:30.636 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 23 v 9 (ObjNum 22 snap 0 seq_num 22) dirty exists 2026-02-25T08:41:30.637 INFO:tasks.rados.rados.0.trial023.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2026-02-25T08:41:30.637 INFO:tasks.rados.rados.0.trial023.stdout:2: finishing write tid 2 to trial02342524-2 2026-02-25T08:41:30.637 INFO:tasks.rados.rados.0.trial023.stdout:17: done (15 left) 2026-02-25T08:41:30.637 INFO:tasks.rados.rados.0.trial023.stdout:23: done (14 left) 2026-02-25T08:41:30.637 INFO:tasks.rados.rados.0.trial023.stdout:29: write initial oid 29 2026-02-25T08:41:30.637 INFO:tasks.rados.rados.0.trial023.stdout:29: seq_num 28 ranges {470596=456031,1572084=618660} 2026-02-25T08:41:30.638 INFO:tasks.rados.rados.0.trial023.stdout:29: writing trial02342524-29 from 470596 to 926627 tid 1 2026-02-25T08:41:30.643 INFO:tasks.rados.rados.0.trial023.stdout:29: writing trial02342524-29 from 1572084 to 2190744 tid 2 2026-02-25T08:41:30.643 INFO:tasks.rados.rados.0.trial023.stdout:30: write initial oid 30 2026-02-25T08:41:30.644 INFO:tasks.rados.rados.0.trial023.stdout:30: seq_num 29 ranges {487403=531280,1514213=478671} 2026-02-25T08:41:30.647 INFO:tasks.rados.rados.0.trial023.stdout:30: writing trial02342524-30 from 487403 to 1018683 tid 1 2026-02-25T08:41:30.652 INFO:tasks.rados.rados.0.trial023.stdout:30: writing trial02342524-30 from 1514213 to 1992884 tid 2 2026-02-25T08:41:30.652 INFO:tasks.rados.rados.0.trial023.stdout: waiting on 16 2026-02-25T08:41:30.652 INFO:tasks.rados.rados.0.trial023.stdout:26: finishing write tid 1 to trial02342524-26 2026-02-25T08:41:30.652 INFO:tasks.rados.rados.0.trial023.stdout:18: finishing write tid 2 to trial02342524-18 2026-02-25T08:41:30.652 INFO:tasks.rados.rados.0.trial023.stdout:18: finishing write tid 3 to trial02342524-18 2026-02-25T08:41:30.652 INFO:tasks.rados.rados.0.trial023.stdout:18: finishing write tid 4 to trial02342524-18 2026-02-25T08:41:30.652 INFO:tasks.rados.rados.0.trial023.stdout:18: oid 18 updating version 0 to 6 2026-02-25T08:41:30.653 INFO:tasks.rados.rados.0.trial023.stdout:18: oid 18 version 6 is already newer than 6 2026-02-25T08:41:30.653 INFO:tasks.rados.rados.0.trial023.stdout:18: oid 18 version 6 is already newer than 4 2026-02-25T08:41:30.653 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 18 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2026-02-25T08:41:30.653 INFO:tasks.rados.rados.0.trial023.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2026-02-25T08:41:30.653 INFO:tasks.rados.rados.0.trial023.stdout:18: done (15 left) 2026-02-25T08:41:30.653 INFO:tasks.rados.rados.0.trial023.stdout:31: write initial oid 31 2026-02-25T08:41:30.653 INFO:tasks.rados.rados.0.trial023.stdout:31: seq_num 30 ranges {504210=606529,1572695=622329} 2026-02-25T08:41:30.656 INFO:tasks.rados.rados.0.trial023.stdout:31: writing trial02342524-31 from 504210 to 1110739 tid 1 2026-02-25T08:41:30.660 INFO:tasks.rados.rados.0.trial023.stdout:31: writing trial02342524-31 from 1572695 to 2195024 tid 2 2026-02-25T08:41:30.660 INFO:tasks.rados.rados.0.trial023.stdout: waiting on 16 2026-02-25T08:41:30.660 INFO:tasks.rados.rados.0.trial023.stdout:2: finishing write tid 3 to trial02342524-2 2026-02-25T08:41:30.660 INFO:tasks.rados.rados.0.trial023.stdout:27: finishing write tid 1 to trial02342524-27 2026-02-25T08:41:30.661 INFO:tasks.rados.rados.0.trial023.stdout:20: finishing write tid 1 to trial02342524-20 2026-02-25T08:41:30.662 INFO:tasks.rados.rados.0.trial023.stdout:28: finishing write tid 1 to trial02342524-28 2026-02-25T08:41:30.663 INFO:tasks.rados.rados.0.trial023.stdout:26: finishing write tid 2 to trial02342524-26 2026-02-25T08:41:30.663 INFO:tasks.rados.rados.0.trial023.stdout:22: finishing write tid 2 to trial02342524-22 2026-02-25T08:41:30.670 INFO:tasks.rados.rados.0.trial023.stdout:19: finishing write tid 1 to trial02342524-19 2026-02-25T08:41:30.671 INFO:tasks.rados.rados.0.trial023.stdout:2: finishing write tid 4 to trial02342524-2 2026-02-25T08:41:30.671 INFO:tasks.rados.rados.0.trial023.stdout:2: oid 2 updating version 0 to 3 2026-02-25T08:41:30.672 INFO:tasks.rados.rados.0.trial023.stdout:2: oid 2 version 3 is already newer than 1 2026-02-25T08:41:30.672 INFO:tasks.rados.rados.0.trial023.stdout:2: oid 2 version 3 is already newer than 3 2026-02-25T08:41:30.672 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2026-02-25T08:41:30.672 INFO:tasks.rados.rados.0.trial023.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2026-02-25T08:41:30.672 INFO:tasks.rados.rados.0.trial023.stdout:2: done (15 left) 2026-02-25T08:41:30.672 INFO:tasks.rados.rados.0.trial023.stdout:32: write initial oid 32 2026-02-25T08:41:30.672 INFO:tasks.rados.rados.0.trial023.stdout:32: seq_num 31 ranges {521017=798131,1747530=482340} 2026-02-25T08:41:30.672 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:30.670+0000 7f3d640b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:30.676 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:30.675+0000 7f3d640b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:30.676 INFO:tasks.rados.rados.0.trial023.stdout:32: writing trial02342524-32 from 521017 to 1319148 tid 1 2026-02-25T08:41:30.681 INFO:tasks.rados.rados.0.trial023.stdout:32: writing trial02342524-32 from 1747530 to 2229870 tid 2 2026-02-25T08:41:30.681 INFO:tasks.rados.rados.0.trial023.stdout: waiting on 16 2026-02-25T08:41:30.681 INFO:tasks.rados.rados.0.trial023.stdout:22: finishing write tid 3 to trial02342524-22 2026-02-25T08:41:30.681 INFO:tasks.rados.rados.0.trial023.stdout:22: finishing write tid 4 to trial02342524-22 2026-02-25T08:41:30.681 INFO:tasks.rados.rados.0.trial023.stdout:22: oid 22 updating version 0 to 3 2026-02-25T08:41:30.681 INFO:tasks.rados.rados.0.trial023.stdout:22: oid 22 version 3 is already newer than 3 2026-02-25T08:41:30.681 INFO:tasks.rados.rados.0.trial023.stdout:22: oid 22 version 3 is already newer than 1 2026-02-25T08:41:30.682 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 22 v 3 (ObjNum 21 snap 0 seq_num 21) dirty exists 2026-02-25T08:41:30.682 INFO:tasks.rados.rados.0.trial023.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2026-02-25T08:41:30.682 INFO:tasks.rados.rados.0.trial023.stdout:22: done (15 left) 2026-02-25T08:41:30.682 INFO:tasks.rados.rados.0.trial023.stdout:33: write initial oid 33 2026-02-25T08:41:30.682 INFO:tasks.rados.rados.0.trial023.stdout:33: seq_num 32 ranges {537824=473380,1806012=625998} 2026-02-25T08:41:30.684 INFO:tasks.rados.rados.0.trial023.stdout:33: writing trial02342524-33 from 537824 to 1011204 tid 1 2026-02-25T08:41:30.690 INFO:tasks.rados.rados.0.trial023.stdout:33: writing trial02342524-33 from 1806012 to 2432010 tid 2 2026-02-25T08:41:30.690 INFO:tasks.rados.rados.0.trial023.stdout: waiting on 16 2026-02-25T08:41:30.690 INFO:tasks.rados.rados.0.trial023.stdout:27: finishing write tid 2 to trial02342524-27 2026-02-25T08:41:30.690 INFO:tasks.rados.rados.0.trial023.stdout:14: finishing write tid 1 to trial02342524-14 2026-02-25T08:41:30.693 INFO:tasks.rados.rados.0.trial023.stdout:27: finishing write tid 3 to trial02342524-27 2026-02-25T08:41:30.693 INFO:tasks.rados.rados.0.trial023.stdout:26: finishing write tid 3 to trial02342524-26 2026-02-25T08:41:30.695 INFO:tasks.rados.rados.0.trial023.stdout:27: finishing write tid 4 to trial02342524-27 2026-02-25T08:41:30.695 INFO:tasks.rados.rados.0.trial023.stdout:27: oid 27 updating version 0 to 6 2026-02-25T08:41:30.695 INFO:tasks.rados.rados.0.trial023.stdout:27: oid 27 version 6 is already newer than 4 2026-02-25T08:41:30.695 INFO:tasks.rados.rados.0.trial023.stdout:27: oid 27 version 6 is already newer than 6 2026-02-25T08:41:30.695 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 27 v 6 (ObjNum 26 snap 0 seq_num 26) dirty exists 2026-02-25T08:41:30.695 INFO:tasks.rados.rados.0.trial023.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2026-02-25T08:41:30.696 INFO:tasks.rados.rados.0.trial023.stdout:27: done (15 left) 2026-02-25T08:41:30.696 INFO:tasks.rados.rados.0.trial023.stdout:34: write initial oid 34 2026-02-25T08:41:30.696 INFO:tasks.rados.rados.0.trial023.stdout:34: seq_num 33 ranges {554631=548629,1748141=486009} 2026-02-25T08:41:30.699 INFO:tasks.rados.rados.0.trial023.stdout:34: writing trial02342524-34 from 554631 to 1103260 tid 1 2026-02-25T08:41:30.704 INFO:tasks.rados.rados.0.trial023.stdout:34: writing trial02342524-34 from 1748141 to 2234150 tid 2 2026-02-25T08:41:30.704 INFO:tasks.rados.rados.0.trial023.stdout: waiting on 16 2026-02-25T08:41:30.704 INFO:tasks.rados.rados.0.trial023.stdout:26: finishing write tid 4 to trial02342524-26 2026-02-25T08:41:30.704 INFO:tasks.rados.rados.0.trial023.stdout:26: oid 26 updating version 0 to 10 2026-02-25T08:41:30.705 INFO:tasks.rados.rados.0.trial023.stdout:26: oid 26 updating version 10 to 12 2026-02-25T08:41:30.705 INFO:tasks.rados.rados.0.trial023.stdout:26: oid 26 version 12 is already newer than 12 2026-02-25T08:41:30.705 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 26 v 12 (ObjNum 25 snap 0 seq_num 25) dirty exists 2026-02-25T08:41:30.705 INFO:tasks.rados.rados.0.trial023.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2026-02-25T08:41:30.705 INFO:tasks.rados.rados.0.trial023.stdout:26: done (15 left) 2026-02-25T08:41:30.705 INFO:tasks.rados.rados.0.trial023.stdout:35: write initial oid 35 2026-02-25T08:41:30.705 INFO:tasks.rados.rados.0.trial023.stdout:29: finishing write tid 1 to trial02342524-29 2026-02-25T08:41:30.705 INFO:tasks.rados.rados.0.trial023.stdout:35: seq_num 34 ranges {571438=623878,1806623=629667} 2026-02-25T08:41:30.708 INFO:tasks.rados.rados.0.trial023.stdout:35: writing trial02342524-35 from 571438 to 1195316 tid 1 2026-02-25T08:41:30.714 INFO:tasks.rados.rados.0.trial023.stdout:35: writing trial02342524-35 from 1806623 to 2436290 tid 2 2026-02-25T08:41:30.714 INFO:tasks.rados.rados.0.trial023.stdout: waiting on 16 2026-02-25T08:41:30.714 INFO:tasks.rados.rados.0.trial023.stdout:28: finishing write tid 2 to trial02342524-28 2026-02-25T08:41:30.714 INFO:tasks.rados.rados.0.trial023.stdout:28: finishing write tid 3 to trial02342524-28 2026-02-25T08:41:30.714 INFO:tasks.rados.rados.0.trial023.stdout:28: finishing write tid 4 to trial02342524-28 2026-02-25T08:41:30.714 INFO:tasks.rados.rados.0.trial023.stdout:28: oid 28 updating version 0 to 3 2026-02-25T08:41:30.714 INFO:tasks.rados.rados.0.trial023.stdout:28: oid 28 version 3 is already newer than 3 2026-02-25T08:41:30.715 INFO:tasks.rados.rados.0.trial023.stdout:28: oid 28 version 3 is already newer than 1 2026-02-25T08:41:30.715 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 28 v 3 (ObjNum 27 snap 0 seq_num 27) dirty exists 2026-02-25T08:41:30.715 INFO:tasks.rados.rados.0.trial023.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2026-02-25T08:41:30.715 INFO:tasks.rados.rados.0.trial023.stdout:20: finishing write tid 2 to trial02342524-20 2026-02-25T08:41:30.715 INFO:tasks.rados.rados.0.trial023.stdout:19: finishing write tid 2 to trial02342524-19 2026-02-25T08:41:30.715 INFO:tasks.rados.rados.0.trial023.stdout:28: done (15 left) 2026-02-25T08:41:30.715 INFO:tasks.rados.rados.0.trial023.stdout:36: write initial oid 36 2026-02-25T08:41:30.715 INFO:tasks.rados.rados.0.trial023.stdout:36: seq_num 35 ranges {588245=699127,1865105=489678} 2026-02-25T08:41:30.718 INFO:tasks.rados.rados.0.trial023.stdout:36: writing trial02342524-36 from 588245 to 1287372 tid 1 2026-02-25T08:41:30.722 INFO:tasks.rados.rados.0.trial023.stdout:36: writing trial02342524-36 from 1865105 to 2354783 tid 2 2026-02-25T08:41:30.723 INFO:tasks.rados.rados.0.trial023.stdout: waiting on 16 2026-02-25T08:41:30.723 INFO:tasks.rados.rados.0.trial023.stdout:29: finishing write tid 2 to trial02342524-29 2026-02-25T08:41:30.723 INFO:tasks.rados.rados.0.trial023.stdout:19: finishing write tid 3 to trial02342524-19 2026-02-25T08:41:30.723 INFO:tasks.rados.rados.0.trial023.stdout:24: finishing write tid 1 to trial02342524-24 2026-02-25T08:41:30.723 INFO:tasks.rados.rados.0.trial023.stdout:29: finishing write tid 3 to trial02342524-29 2026-02-25T08:41:30.724 INFO:tasks.rados.rados.0.trial023.stdout:20: finishing write tid 3 to trial02342524-20 2026-02-25T08:41:30.724 INFO:tasks.rados.rados.0.trial023.stdout:29: finishing write tid 4 to trial02342524-29 2026-02-25T08:41:30.724 INFO:tasks.rados.rados.0.trial023.stdout:29: oid 29 updating version 0 to 15 2026-02-25T08:41:30.724 INFO:tasks.rados.rados.0.trial023.stdout:29: oid 29 version 15 is already newer than 15 2026-02-25T08:41:30.724 INFO:tasks.rados.rados.0.trial023.stdout:29: oid 29 version 15 is already newer than 13 2026-02-25T08:41:30.724 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 29 v 15 (ObjNum 28 snap 0 seq_num 28) dirty exists 2026-02-25T08:41:30.725 INFO:tasks.rados.rados.0.trial023.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2026-02-25T08:41:30.725 INFO:tasks.rados.rados.0.trial023.stdout:29: done (15 left) 2026-02-25T08:41:30.725 INFO:tasks.rados.rados.0.trial023.stdout:37: write initial oid 37 2026-02-25T08:41:30.725 INFO:tasks.rados.rados.0.trial023.stdout:37: seq_num 36 ranges {605052=774376,1923587=633336} 2026-02-25T08:41:30.729 INFO:tasks.rados.rados.0.trial023.stdout:37: writing trial02342524-37 from 605052 to 1379428 tid 1 2026-02-25T08:41:30.734 INFO:tasks.rados.rados.0.trial023.stdout:37: writing trial02342524-37 from 1923587 to 2556923 tid 2 2026-02-25T08:41:30.734 INFO:tasks.rados.rados.0.trial023.stdout: waiting on 16 2026-02-25T08:41:30.734 INFO:tasks.rados.rados.0.trial023.stdout:19: finishing write tid 4 to trial02342524-19 2026-02-25T08:41:30.734 INFO:tasks.rados.rados.0.trial023.stdout:19: oid 19 updating version 0 to 7 2026-02-25T08:41:30.734 INFO:tasks.rados.rados.0.trial023.stdout:19: oid 19 updating version 7 to 9 2026-02-25T08:41:30.734 INFO:tasks.rados.rados.0.trial023.stdout:19: oid 19 version 9 is already newer than 9 2026-02-25T08:41:30.734 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 19 v 9 (ObjNum 18 snap 0 seq_num 18) dirty exists 2026-02-25T08:41:30.734 INFO:tasks.rados.rados.0.trial023.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2026-02-25T08:41:30.734 INFO:tasks.rados.rados.0.trial023.stdout:19: done (15 left) 2026-02-25T08:41:30.734 INFO:tasks.rados.rados.0.trial023.stdout:38: write initial oid 38 2026-02-25T08:41:30.735 INFO:tasks.rados.rados.0.trial023.stdout:38: seq_num 37 ranges {621859=449625,1865716=756143} 2026-02-25T08:41:30.737 INFO:tasks.rados.rados.0.trial023.stdout:38: writing trial02342524-38 from 621859 to 1071484 tid 1 2026-02-25T08:41:30.744 INFO:tasks.rados.rados.0.trial023.stdout:38: writing trial02342524-38 from 1865716 to 2621859 tid 2 2026-02-25T08:41:30.744 INFO:tasks.rados.rados.0.trial023.stdout: waiting on 16 2026-02-25T08:41:30.744 INFO:tasks.rados.rados.0.trial023.stdout:20: finishing write tid 4 to trial02342524-20 2026-02-25T08:41:30.744 INFO:tasks.rados.rados.0.trial023.stdout:20: oid 20 updating version 0 to 12 2026-02-25T08:41:30.744 INFO:tasks.rados.rados.0.trial023.stdout:20: oid 20 version 12 is already newer than 12 2026-02-25T08:41:30.744 INFO:tasks.rados.rados.0.trial023.stdout:20: oid 20 version 12 is already newer than 10 2026-02-25T08:41:30.744 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 20 v 12 (ObjNum 19 snap 0 seq_num 19) dirty exists 2026-02-25T08:41:30.744 INFO:tasks.rados.rados.0.trial023.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2026-02-25T08:41:30.745 INFO:tasks.rados.rados.0.trial023.stdout:20: done (15 left) 2026-02-25T08:41:30.745 INFO:tasks.rados.rados.0.trial023.stdout:39: write initial oid 39 2026-02-25T08:41:30.745 INFO:tasks.rados.rados.0.trial023.stdout:39: seq_num 38 ranges {638666=524874,1924198=637005} 2026-02-25T08:41:30.748 INFO:tasks.rados.rados.0.trial023.stdout:39: writing trial02342524-39 from 638666 to 1163540 tid 1 2026-02-25T08:41:30.754 INFO:tasks.rados.rados.0.trial023.stdout:39: writing trial02342524-39 from 1924198 to 2561203 tid 2 2026-02-25T08:41:30.755 INFO:tasks.rados.rados.0.trial023.stdout: waiting on 16 2026-02-25T08:41:30.755 INFO:tasks.rados.rados.0.trial023.stdout:35: finishing write tid 1 to trial02342524-35 2026-02-25T08:41:30.755 INFO:tasks.rados.rados.0.trial023.stdout:21: finishing write tid 1 to trial02342524-21 2026-02-25T08:41:30.755 INFO:tasks.rados.rados.0.trial023.stdout:33: finishing write tid 1 to trial02342524-33 2026-02-25T08:41:30.756 INFO:tasks.rados.rados.0.trial023.stdout:36: finishing write tid 1 to trial02342524-36 2026-02-25T08:41:30.763 INFO:tasks.rados.rados.0.trial023.stdout:34: finishing write tid 1 to trial02342524-34 2026-02-25T08:41:30.765 INFO:tasks.rados.rados.0.trial023.stdout:12: finishing write tid 1 to trial02342524-12 2026-02-25T08:41:30.766 INFO:tasks.rados.rados.0.trial023.stdout:35: finishing write tid 2 to trial02342524-35 2026-02-25T08:41:30.775 INFO:tasks.rados.rados.0.trial023.stdout:37: finishing write tid 1 to trial02342524-37 2026-02-25T08:41:30.780 INFO:tasks.rados.rados.0.trial023.stdout:35: finishing write tid 3 to trial02342524-35 2026-02-25T08:41:30.781 INFO:tasks.rados.rados.0.trial023.stdout:36: finishing write tid 2 to trial02342524-36 2026-02-25T08:41:30.781 INFO:tasks.rados.rados.0.trial023.stdout:33: finishing write tid 2 to trial02342524-33 2026-02-25T08:41:30.783 INFO:tasks.rados.rados.0.trial023.stdout:35: finishing write tid 4 to trial02342524-35 2026-02-25T08:41:30.783 INFO:tasks.rados.rados.0.trial023.stdout:35: oid 35 updating version 0 to 4 2026-02-25T08:41:30.783 INFO:tasks.rados.rados.0.trial023.stdout:35: oid 35 updating version 4 to 6 2026-02-25T08:41:30.783 INFO:tasks.rados.rados.0.trial023.stdout:35: oid 35 version 6 is already newer than 6 2026-02-25T08:41:30.783 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 35 v 6 (ObjNum 34 snap 0 seq_num 34) dirty exists 2026-02-25T08:41:30.783 INFO:tasks.rados.rados.0.trial023.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2026-02-25T08:41:30.783 INFO:tasks.rados.rados.0.trial023.stdout:35: done (15 left) 2026-02-25T08:41:30.784 INFO:tasks.rados.rados.0.trial023.stdout:40: write initial oid 40 2026-02-25T08:41:30.784 INFO:tasks.rados.rados.0.trial023.stdout:40: seq_num 39 ranges {655473=716476,2099033=556440} 2026-02-25T08:41:30.788 INFO:tasks.rados.rados.0.trial023.stdout:40: writing trial02342524-40 from 655473 to 1371949 tid 1 2026-02-25T08:41:30.793 INFO:tasks.rados.rados.0.trial023.stdout:40: writing trial02342524-40 from 2099033 to 2655473 tid 2 2026-02-25T08:41:30.793 INFO:tasks.rados.rados.0.trial023.stdout: waiting on 16 2026-02-25T08:41:30.793 INFO:tasks.rados.rados.0.trial023.stdout:39: finishing write tid 1 to trial02342524-39 2026-02-25T08:41:30.793 INFO:tasks.rados.rados.0.trial023.stdout:36: finishing write tid 3 to trial02342524-36 2026-02-25T08:41:30.795 INFO:tasks.rados.rados.0.trial023.stdout:37: finishing write tid 2 to trial02342524-37 2026-02-25T08:41:30.797 INFO:tasks.rados.rados.0.trial023.stdout:36: finishing write tid 4 to trial02342524-36 2026-02-25T08:41:30.797 INFO:tasks.rados.rados.0.trial023.stdout:36: oid 36 updating version 0 to 4 2026-02-25T08:41:30.797 INFO:tasks.rados.rados.0.trial023.stdout:36: oid 36 updating version 4 to 6 2026-02-25T08:41:30.797 INFO:tasks.rados.rados.0.trial023.stdout:36: oid 36 version 6 is already newer than 6 2026-02-25T08:41:30.797 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 36 v 6 (ObjNum 35 snap 0 seq_num 35) dirty exists 2026-02-25T08:41:30.797 INFO:tasks.rados.rados.0.trial023.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2026-02-25T08:41:30.797 INFO:tasks.rados.rados.0.trial023.stdout:36: done (15 left) 2026-02-25T08:41:30.797 INFO:tasks.rados.rados.0.trial023.stdout:41: write initial oid 41 2026-02-25T08:41:30.797 INFO:tasks.rados.rados.0.trial023.stdout:41: seq_num 40 ranges {672280=791725,2157515=514765} 2026-02-25T08:41:30.802 INFO:tasks.rados.rados.0.trial023.stdout:41: writing trial02342524-41 from 672280 to 1464005 tid 1 2026-02-25T08:41:30.808 INFO:tasks.rados.rados.0.trial023.stdout:41: writing trial02342524-41 from 2157515 to 2672280 tid 2 2026-02-25T08:41:30.808 INFO:tasks.rados.rados.0.trial023.stdout: waiting on 16 2026-02-25T08:41:30.808 INFO:tasks.rados.rados.0.trial023.stdout:21: finishing write tid 2 to trial02342524-21 2026-02-25T08:41:30.808 INFO:tasks.rados.rados.0.trial023.stdout:37: finishing write tid 3 to trial02342524-37 2026-02-25T08:41:30.808 INFO:tasks.rados.rados.0.trial023.stdout:37: finishing write tid 4 to trial02342524-37 2026-02-25T08:41:30.808 INFO:tasks.rados.rados.0.trial023.stdout:37: oid 37 updating version 0 to 18 2026-02-25T08:41:30.808 INFO:tasks.rados.rados.0.trial023.stdout:37: oid 37 version 18 is already newer than 16 2026-02-25T08:41:30.808 INFO:tasks.rados.rados.0.trial023.stdout:37: oid 37 version 18 is already newer than 18 2026-02-25T08:41:30.809 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 37 v 18 (ObjNum 36 snap 0 seq_num 36) dirty exists 2026-02-25T08:41:30.809 INFO:tasks.rados.rados.0.trial023.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2026-02-25T08:41:30.809 INFO:tasks.rados.rados.0.trial023.stdout:6: finishing write tid 1 to trial02342524-6 2026-02-25T08:41:30.809 INFO:tasks.rados.rados.0.trial023.stdout:25: finishing write tid 1 to trial02342524-25 2026-02-25T08:41:30.809 INFO:tasks.rados.rados.0.trial023.stdout:37: done (15 left) 2026-02-25T08:41:30.809 INFO:tasks.rados.rados.0.trial023.stdout:42: write initial oid 42 2026-02-25T08:41:30.809 INFO:tasks.rados.rados.0.trial023.stdout:42: seq_num 41 ranges {689087=466974,1699644=784332} 2026-02-25T08:41:30.811 INFO:tasks.rados.rados.0.trial023.stdout:42: writing trial02342524-42 from 689087 to 1156061 tid 1 2026-02-25T08:41:30.816 INFO:tasks.rados.rados.0.trial023.stdout:42: writing trial02342524-42 from 1699644 to 2483976 tid 2 2026-02-25T08:41:30.817 INFO:tasks.rados.rados.0.trial023.stdout: waiting on 16 2026-02-25T08:41:30.817 INFO:tasks.rados.rados.0.trial023.stdout:34: finishing write tid 2 to trial02342524-34 2026-02-25T08:41:30.817 INFO:tasks.rados.rados.0.trial023.stdout:33: finishing write tid 3 to trial02342524-33 2026-02-25T08:41:30.820 INFO:tasks.rados.rados.0.trial023.stdout:33: finishing write tid 4 to trial02342524-33 2026-02-25T08:41:30.820 INFO:tasks.rados.rados.0.trial023.stdout:33: oid 33 updating version 0 to 1 2026-02-25T08:41:30.821 INFO:tasks.rados.rados.0.trial023.stdout:33: oid 33 updating version 1 to 3 2026-02-25T08:41:30.821 INFO:tasks.rados.rados.0.trial023.stdout:33: oid 33 version 3 is already newer than 3 2026-02-25T08:41:30.821 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 33 v 3 (ObjNum 32 snap 0 seq_num 32) dirty exists 2026-02-25T08:41:30.821 INFO:tasks.rados.rados.0.trial023.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2026-02-25T08:41:30.821 INFO:tasks.rados.rados.0.trial023.stdout:33: done (15 left) 2026-02-25T08:41:30.821 INFO:tasks.rados.rados.0.trial023.stdout:43: write initial oid 43 2026-02-25T08:41:30.821 INFO:tasks.rados.rados.0.trial023.stdout:43: seq_num 42 ranges {705894=542223,1758126=644343} 2026-02-25T08:41:30.825 INFO:tasks.rados.rados.0.trial023.stdout:43: writing trial02342524-43 from 705894 to 1248117 tid 1 2026-02-25T08:41:30.830 INFO:tasks.rados.rados.0.trial023.stdout:43: writing trial02342524-43 from 1758126 to 2402469 tid 2 2026-02-25T08:41:30.830 INFO:tasks.rados.rados.0.trial023.stdout: waiting on 16 2026-02-25T08:41:30.830 INFO:tasks.rados.rados.0.trial023.stdout:21: finishing write tid 3 to trial02342524-21 2026-02-25T08:41:30.830 INFO:tasks.rados.rados.0.trial023.stdout:21: finishing write tid 4 to trial02342524-21 2026-02-25T08:41:30.830 INFO:tasks.rados.rados.0.trial023.stdout:21: oid 21 updating version 0 to 12 2026-02-25T08:41:30.830 INFO:tasks.rados.rados.0.trial023.stdout:21: oid 21 version 12 is already newer than 12 2026-02-25T08:41:30.830 INFO:tasks.rados.rados.0.trial023.stdout:21: oid 21 version 12 is already newer than 10 2026-02-25T08:41:30.830 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 21 v 12 (ObjNum 20 snap 0 seq_num 20) dirty exists 2026-02-25T08:41:30.830 INFO:tasks.rados.rados.0.trial023.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2026-02-25T08:41:30.830 INFO:tasks.rados.rados.0.trial023.stdout:34: finishing write tid 3 to trial02342524-34 2026-02-25T08:41:30.830 INFO:tasks.rados.rados.0.trial023.stdout:21: done (15 left) 2026-02-25T08:41:30.830 INFO:tasks.rados.rados.0.trial023.stdout:44: write initial oid 44 2026-02-25T08:41:30.830 INFO:tasks.rados.rados.0.trial023.stdout:44: seq_num 43 ranges {722701=617472,1816608=788001} 2026-02-25T08:41:30.835 INFO:tasks.rados.rados.0.trial023.stdout:44: writing trial02342524-44 from 722701 to 1340173 tid 1 2026-02-25T08:41:30.840 INFO:tasks.rados.rados.0.trial023.stdout:44: writing trial02342524-44 from 1816608 to 2604609 tid 2 2026-02-25T08:41:30.840 INFO:tasks.rados.rados.0.trial023.stdout: waiting on 16 2026-02-25T08:41:30.840 INFO:tasks.rados.rados.0.trial023.stdout:34: finishing write tid 4 to trial02342524-34 2026-02-25T08:41:30.840 INFO:tasks.rados.rados.0.trial023.stdout:34: oid 34 updating version 0 to 4 2026-02-25T08:41:30.840 INFO:tasks.rados.rados.0.trial023.stdout:34: oid 34 updating version 4 to 6 2026-02-25T08:41:30.840 INFO:tasks.rados.rados.0.trial023.stdout:34: oid 34 version 6 is already newer than 6 2026-02-25T08:41:30.840 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 34 v 6 (ObjNum 33 snap 0 seq_num 33) dirty exists 2026-02-25T08:41:30.840 INFO:tasks.rados.rados.0.trial023.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2026-02-25T08:41:30.840 INFO:tasks.rados.rados.0.trial023.stdout:34: done (15 left) 2026-02-25T08:41:30.840 INFO:tasks.rados.rados.0.trial023.stdout:45: write initial oid 45 2026-02-25T08:41:30.840 INFO:tasks.rados.rados.0.trial023.stdout:45: seq_num 44 ranges {739508=692721,1875090=648012} 2026-02-25T08:41:30.845 INFO:tasks.rados.rados.0.trial023.stdout:45: writing trial02342524-45 from 739508 to 1432229 tid 1 2026-02-25T08:41:30.849 INFO:tasks.rados.rados.0.trial023.stdout:45: writing trial02342524-45 from 1875090 to 2523102 tid 2 2026-02-25T08:41:30.850 INFO:tasks.rados.rados.0.trial023.stdout: waiting on 16 2026-02-25T08:41:30.850 INFO:tasks.rados.rados.0.trial023.stdout:40: finishing write tid 1 to trial02342524-40 2026-02-25T08:41:30.850 INFO:tasks.rados.rados.0.trial023.stdout:39: finishing write tid 2 to trial02342524-39 2026-02-25T08:41:30.850 INFO:tasks.rados.rados.0.trial023.stdout:14: finishing write tid 2 to trial02342524-14 2026-02-25T08:41:30.850 INFO:tasks.rados.rados.0.trial023.stdout:39: finishing write tid 3 to trial02342524-39 2026-02-25T08:41:30.850 INFO:tasks.rados.rados.0.trial023.stdout:39: finishing write tid 4 to trial02342524-39 2026-02-25T08:41:30.850 INFO:tasks.rados.rados.0.trial023.stdout:39: oid 39 updating version 0 to 6 2026-02-25T08:41:30.850 INFO:tasks.rados.rados.0.trial023.stdout:39: oid 39 version 6 is already newer than 4 2026-02-25T08:41:30.850 INFO:tasks.rados.rados.0.trial023.stdout:39: oid 39 version 6 is already newer than 6 2026-02-25T08:41:30.850 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 39 v 6 (ObjNum 38 snap 0 seq_num 38) dirty exists 2026-02-25T08:41:30.850 INFO:tasks.rados.rados.0.trial023.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2026-02-25T08:41:30.850 INFO:tasks.rados.rados.0.trial023.stdout:25: finishing write tid 2 to trial02342524-25 2026-02-25T08:41:30.850 INFO:tasks.rados.rados.0.trial023.stdout:38: finishing write tid 1 to trial02342524-38 2026-02-25T08:41:30.850 INFO:tasks.rados.rados.0.trial023.stdout:39: done (15 left) 2026-02-25T08:41:30.850 INFO:tasks.rados.rados.0.trial023.stdout:46: write initial oid 46 2026-02-25T08:41:30.850 INFO:tasks.rados.rados.0.trial023.stdout:46: seq_num 45 ranges {756315=767970,1933572=791670} 2026-02-25T08:41:30.855 INFO:tasks.rados.rados.0.trial023.stdout:46: writing trial02342524-46 from 756315 to 1524285 tid 1 2026-02-25T08:41:30.860 INFO:tasks.rados.rados.0.trial023.stdout:46: writing trial02342524-46 from 1933572 to 2725242 tid 2 2026-02-25T08:41:30.860 INFO:tasks.rados.rados.0.trial023.stdout: waiting on 16 2026-02-25T08:41:30.860 INFO:tasks.rados.rados.0.trial023.stdout:25: finishing write tid 3 to trial02342524-25 2026-02-25T08:41:30.861 INFO:tasks.rados.rados.0.trial023.stdout:25: finishing write tid 4 to trial02342524-25 2026-02-25T08:41:30.862 INFO:tasks.rados.rados.0.trial023.stdout:25: oid 25 updating version 0 to 15 2026-02-25T08:41:30.862 INFO:tasks.rados.rados.0.trial023.stdout:25: oid 25 version 15 is already newer than 13 2026-02-25T08:41:30.862 INFO:tasks.rados.rados.0.trial023.stdout:25: oid 25 version 15 is already newer than 15 2026-02-25T08:41:30.862 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 25 v 15 (ObjNum 24 snap 0 seq_num 24) dirty exists 2026-02-25T08:41:30.862 INFO:tasks.rados.rados.0.trial023.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2026-02-25T08:41:30.862 INFO:tasks.rados.rados.0.trial023.stdout:25: done (15 left) 2026-02-25T08:41:30.862 INFO:tasks.rados.rados.0.trial023.stdout:47: write initial oid 47 2026-02-25T08:41:30.862 INFO:tasks.rados.rados.0.trial023.stdout:47: seq_num 46 ranges {773122=559572,1992054=651681} 2026-02-25T08:41:30.866 INFO:tasks.rados.rados.0.trial023.stdout:47: writing trial02342524-47 from 773122 to 1332694 tid 1 2026-02-25T08:41:30.871 INFO:tasks.rados.rados.0.trial023.stdout:47: writing trial02342524-47 from 1992054 to 2643735 tid 2 2026-02-25T08:41:30.871 INFO:tasks.rados.rados.0.trial023.stdout: waiting on 16 2026-02-25T08:41:30.871 INFO:tasks.rados.rados.0.trial023.stdout:40: finishing write tid 2 to trial02342524-40 2026-02-25T08:41:30.871 INFO:tasks.rados.rados.0.trial023.stdout:6: finishing write tid 2 to trial02342524-6 2026-02-25T08:41:30.871 INFO:tasks.rados.rados.0.trial023.stdout:43: finishing write tid 1 to trial02342524-43 2026-02-25T08:41:30.873 INFO:tasks.rados.rados.0.trial023.stdout:44: finishing write tid 1 to trial02342524-44 2026-02-25T08:41:30.874 INFO:tasks.rados.rados.0.trial023.stdout:42: finishing write tid 1 to trial02342524-42 2026-02-25T08:41:30.875 INFO:tasks.rados.rados.0.trial023.stdout:38: finishing write tid 2 to trial02342524-38 2026-02-25T08:41:30.879 INFO:tasks.rados.rados.0.trial023.stdout:38: finishing write tid 3 to trial02342524-38 2026-02-25T08:41:30.882 INFO:tasks.rados.rados.0.trial023.stdout:6: finishing write tid 3 to trial02342524-6 2026-02-25T08:41:30.885 INFO:tasks.rados.rados.0.trial023.stdout:6: finishing write tid 4 to trial02342524-6 2026-02-25T08:41:30.885 INFO:tasks.rados.rados.0.trial023.stdout:6: oid 6 updating version 0 to 1 2026-02-25T08:41:30.885 INFO:tasks.rados.rados.0.trial023.stdout:6: oid 6 updating version 1 to 3 2026-02-25T08:41:30.885 INFO:tasks.rados.rados.0.trial023.stdout:6: oid 6 version 3 is already newer than 3 2026-02-25T08:41:30.885 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 6 v 3 (ObjNum 5 snap 0 seq_num 5) dirty exists 2026-02-25T08:41:30.885 INFO:tasks.rados.rados.0.trial023.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2026-02-25T08:41:30.885 INFO:tasks.rados.rados.0.trial023.stdout:6: done (15 left) 2026-02-25T08:41:30.886 INFO:tasks.rados.rados.0.trial023.stdout:48: write initial oid 48 2026-02-25T08:41:30.886 INFO:tasks.rados.rados.0.trial023.stdout:48: seq_num 47 ranges {789929=634821,2050536=739393} 2026-02-25T08:41:30.890 INFO:tasks.rados.rados.0.trial023.stdout:48: writing trial02342524-48 from 789929 to 1424750 tid 1 2026-02-25T08:41:30.896 INFO:tasks.rados.rados.0.trial023.stdout:48: writing trial02342524-48 from 2050536 to 2789929 tid 2 2026-02-25T08:41:30.896 INFO:tasks.rados.rados.0.trial023.stdout: waiting on 16 2026-02-25T08:41:30.896 INFO:tasks.rados.rados.0.trial023.stdout:38: finishing write tid 4 to trial02342524-38 2026-02-25T08:41:30.896 INFO:tasks.rados.rados.0.trial023.stdout:38: oid 38 updating version 0 to 13 2026-02-25T08:41:30.896 INFO:tasks.rados.rados.0.trial023.stdout:38: oid 38 updating version 13 to 15 2026-02-25T08:41:30.897 INFO:tasks.rados.rados.0.trial023.stdout:38: oid 38 version 15 is already newer than 15 2026-02-25T08:41:30.897 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 38 v 15 (ObjNum 37 snap 0 seq_num 37) dirty exists 2026-02-25T08:41:30.897 INFO:tasks.rados.rados.0.trial023.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2026-02-25T08:41:30.897 INFO:tasks.rados.rados.0.trial023.stdout:38: done (15 left) 2026-02-25T08:41:30.897 INFO:tasks.rados.rados.0.trial023.stdout:49: write initial oid 49 2026-02-25T08:41:30.897 INFO:tasks.rados.rados.0.trial023.stdout:49: seq_num 48 ranges {406736=710070,1709018=655350} 2026-02-25T08:41:30.900 INFO:tasks.rados.rados.0.trial023.stdout:49: writing trial02342524-49 from 406736 to 1116806 tid 1 2026-02-25T08:41:30.905 INFO:tasks.rados.rados.0.trial023.stdout:49: writing trial02342524-49 from 1709018 to 2364368 tid 2 2026-02-25T08:41:30.906 INFO:tasks.rados.rados.0.trial023.stdout: waiting on 16 2026-02-25T08:41:30.906 INFO:tasks.rados.rados.0.trial023.stdout:40: finishing write tid 3 to trial02342524-40 2026-02-25T08:41:30.906 INFO:tasks.rados.rados.0.trial023.stdout:40: finishing write tid 4 to trial02342524-40 2026-02-25T08:41:30.906 INFO:tasks.rados.rados.0.trial023.stdout:40: oid 40 updating version 0 to 7 2026-02-25T08:41:30.906 INFO:tasks.rados.rados.0.trial023.stdout:40: oid 40 updating version 7 to 9 2026-02-25T08:41:30.906 INFO:tasks.rados.rados.0.trial023.stdout:40: oid 40 version 9 is already newer than 9 2026-02-25T08:41:30.906 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 40 v 9 (ObjNum 39 snap 0 seq_num 39) dirty exists 2026-02-25T08:41:30.906 INFO:tasks.rados.rados.0.trial023.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2026-02-25T08:41:30.906 INFO:tasks.rados.rados.0.trial023.stdout:43: finishing write tid 2 to trial02342524-43 2026-02-25T08:41:30.906 INFO:tasks.rados.rados.0.trial023.stdout:40: done (15 left) 2026-02-25T08:41:30.906 INFO:tasks.rados.rados.0.trial023.stdout:50: write initial oid 50 2026-02-25T08:41:30.907 INFO:tasks.rados.rados.0.trial023.stdout:50: seq_num 49 ranges {423543=785319,1767500=799008} 2026-02-25T08:41:30.909 INFO:tasks.rados.rados.0.trial023.stdout:50: writing trial02342524-50 from 423543 to 1208862 tid 1 2026-02-25T08:41:30.915 INFO:tasks.rados.rados.0.trial023.stdout:50: writing trial02342524-50 from 1767500 to 2566508 tid 2 2026-02-25T08:41:30.915 INFO:tasks.rados.rados.0.trial023.stdout: waiting on 16 2026-02-25T08:41:30.915 INFO:tasks.rados.rados.0.trial023.stdout:43: finishing write tid 3 to trial02342524-43 2026-02-25T08:41:30.915 INFO:tasks.rados.rados.0.trial023.stdout:31: finishing write tid 1 to trial02342524-31 2026-02-25T08:41:30.916 INFO:tasks.rados.rados.0.trial023.stdout:44: finishing write tid 2 to trial02342524-44 2026-02-25T08:41:30.916 INFO:tasks.rados.rados.0.trial023.stdout:43: finishing write tid 4 to trial02342524-43 2026-02-25T08:41:30.916 INFO:tasks.rados.rados.0.trial023.stdout:43: oid 43 updating version 0 to 1 2026-02-25T08:41:30.916 INFO:tasks.rados.rados.0.trial023.stdout:43: oid 43 updating version 1 to 3 2026-02-25T08:41:30.916 INFO:tasks.rados.rados.0.trial023.stdout:43: oid 43 version 3 is already newer than 3 2026-02-25T08:41:30.916 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 43 v 3 (ObjNum 42 snap 0 seq_num 42) dirty exists 2026-02-25T08:41:30.916 INFO:tasks.rados.rados.0.trial023.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2026-02-25T08:41:30.916 INFO:tasks.rados.rados.0.trial023.stdout:45: finishing write tid 1 to trial02342524-45 2026-02-25T08:41:30.916 INFO:tasks.rados.rados.0.trial023.stdout:43: done (15 left) 2026-02-25T08:41:30.916 INFO:tasks.rados.rados.0.trial023.stdout:51: write (excl) oid 21 current snap is 0 2026-02-25T08:41:30.917 INFO:tasks.rados.rados.0.trial023.stdout:51: seq_num 50 ranges {440350=460568,1309629=542666,2323973=516377} 2026-02-25T08:41:30.918 INFO:tasks.rados.rados.0.trial023.stdout:51: writing trial02342524-21 from 440350 to 900918 tid 1 2026-02-25T08:41:30.922 INFO:tasks.rados.rados.0.trial023.stdout:51: writing trial02342524-21 from 1309629 to 1852295 tid 2 2026-02-25T08:41:30.926 INFO:tasks.rados.rados.0.trial023.stdout:51: writing trial02342524-21 from 2323973 to 2840350 tid 3 2026-02-25T08:41:30.926 INFO:tasks.rados.rados.0.trial023.stdout: waiting on 16 2026-02-25T08:41:30.926 INFO:tasks.rados.rados.0.trial023.stdout:44: finishing write tid 3 to trial02342524-44 2026-02-25T08:41:30.926 INFO:tasks.rados.rados.0.trial023.stdout:42: finishing write tid 2 to trial02342524-42 2026-02-25T08:41:30.927 INFO:tasks.rados.rados.0.trial023.stdout:48: finishing write tid 1 to trial02342524-48 2026-02-25T08:41:30.929 INFO:tasks.rados.rados.0.trial023.stdout:44: finishing write tid 4 to trial02342524-44 2026-02-25T08:41:30.929 INFO:tasks.rados.rados.0.trial023.stdout:44: oid 44 updating version 0 to 6 2026-02-25T08:41:30.929 INFO:tasks.rados.rados.0.trial023.stdout:44: oid 44 version 6 is already newer than 6 2026-02-25T08:41:30.929 INFO:tasks.rados.rados.0.trial023.stdout:44: oid 44 version 6 is already newer than 4 2026-02-25T08:41:30.929 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 44 v 6 (ObjNum 43 snap 0 seq_num 43) dirty exists 2026-02-25T08:41:30.929 INFO:tasks.rados.rados.0.trial023.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2026-02-25T08:41:30.929 INFO:tasks.rados.rados.0.trial023.stdout:44: done (15 left) 2026-02-25T08:41:30.929 INFO:tasks.rados.rados.0.trial023.stdout:52: write (excl) oid 16 current snap is 0 2026-02-25T08:41:30.930 INFO:tasks.rados.rados.0.trial023.stdout:52: seq_num 51 ranges {457157=535817,1768111=402677} 2026-02-25T08:41:30.932 INFO:tasks.rados.rados.0.trial023.stdout:52: writing trial02342524-16 from 457157 to 992974 tid 1 2026-02-25T08:41:30.937 INFO:tasks.rados.rados.0.trial023.stdout:52: writing trial02342524-16 from 1768111 to 2170788 tid 2 2026-02-25T08:41:30.937 INFO:tasks.rados.rados.0.trial023.stdout: waiting on 16 2026-02-25T08:41:30.943 INFO:tasks.rados.rados.0.trial023.stdout:14: finishing write tid 3 to trial02342524-14 2026-02-25T08:41:30.948 INFO:tasks.rados.rados.0.trial023.stdout:45: finishing write tid 2 to trial02342524-45 2026-02-25T08:41:30.950 INFO:tasks.rados.rados.0.trial023.stdout:45: finishing write tid 3 to trial02342524-45 2026-02-25T08:41:30.952 INFO:tasks.rados.rados.0.trial023.stdout:51: finishing write tid 1 to trial02342524-21 2026-02-25T08:41:30.956 INFO:tasks.rados.rados.0.trial023.stdout:14: finishing write tid 4 to trial02342524-14 2026-02-25T08:41:30.957 INFO:tasks.rados.rados.0.trial023.stdout:14: oid 14 updating version 0 to 6 2026-02-25T08:41:30.957 INFO:tasks.rados.rados.0.trial023.stdout:14: oid 14 version 6 is already newer than 6 2026-02-25T08:41:30.957 INFO:tasks.rados.rados.0.trial023.stdout:14: oid 14 version 6 is already newer than 4 2026-02-25T08:41:30.957 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 14 v 6 (ObjNum 13 snap 0 seq_num 13) dirty exists 2026-02-25T08:41:30.957 INFO:tasks.rados.rados.0.trial023.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2026-02-25T08:41:30.957 INFO:tasks.rados.rados.0.trial023.stdout:14: done (15 left) 2026-02-25T08:41:30.957 INFO:tasks.rados.rados.0.trial023.stdout:53: read oid 20 snap -1 2026-02-25T08:41:30.957 INFO:tasks.rados.rados.0.trial023.stdout:53: expect (ObjNum 19 snap 0 seq_num 19) 2026-02-25T08:41:30.957 INFO:tasks.rados.rados.0.trial023.stdout: waiting on 16 2026-02-25T08:41:30.957 INFO:tasks.rados.rados.0.trial023.stdout:45: finishing write tid 4 to trial02342524-45 2026-02-25T08:41:30.958 INFO:tasks.rados.rados.0.trial023.stdout:45: oid 45 updating version 0 to 18 2026-02-25T08:41:30.958 INFO:tasks.rados.rados.0.trial023.stdout:45: oid 45 version 18 is already newer than 16 2026-02-25T08:41:30.958 INFO:tasks.rados.rados.0.trial023.stdout:45: oid 45 version 18 is already newer than 18 2026-02-25T08:41:30.958 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 45 v 18 (ObjNum 44 snap 0 seq_num 44) dirty exists 2026-02-25T08:41:30.958 INFO:tasks.rados.rados.0.trial023.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2026-02-25T08:41:30.958 INFO:tasks.rados.rados.0.trial023.stdout:45: done (15 left) 2026-02-25T08:41:30.958 INFO:tasks.rados.rados.0.trial023.stdout:54: read oid 36 snap -1 2026-02-25T08:41:30.958 INFO:tasks.rados.rados.0.trial023.stdout:54: expect (ObjNum 35 snap 0 seq_num 35) 2026-02-25T08:41:30.958 INFO:tasks.rados.rados.0.trial023.stdout: waiting on 16 2026-02-25T08:41:30.963 INFO:tasks.rados.rados.0.trial023.stdout:42: finishing write tid 3 to trial02342524-42 2026-02-25T08:41:30.964 INFO:tasks.rados.rados.0.trial023.stdout:48: finishing write tid 2 to trial02342524-48 2026-02-25T08:41:30.965 INFO:tasks.rados.rados.0.trial023.stdout:48: finishing write tid 3 to trial02342524-48 2026-02-25T08:41:30.966 INFO:tasks.rados.rados.0.trial023.stdout:42: finishing write tid 4 to trial02342524-42 2026-02-25T08:41:30.966 INFO:tasks.rados.rados.0.trial023.stdout:42: oid 42 updating version 0 to 9 2026-02-25T08:41:30.966 INFO:tasks.rados.rados.0.trial023.stdout:42: oid 42 version 9 is already newer than 7 2026-02-25T08:41:30.966 INFO:tasks.rados.rados.0.trial023.stdout:42: oid 42 version 9 is already newer than 9 2026-02-25T08:41:30.966 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 42 v 9 (ObjNum 41 snap 0 seq_num 41) dirty exists 2026-02-25T08:41:30.967 INFO:tasks.rados.rados.0.trial023.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2026-02-25T08:41:30.967 INFO:tasks.rados.rados.0.trial023.stdout:42: done (15 left) 2026-02-25T08:41:30.967 INFO:tasks.rados.rados.0.trial023.stdout:55: read oid 26 snap -1 2026-02-25T08:41:30.967 INFO:tasks.rados.rados.0.trial023.stdout:55: expect (ObjNum 25 snap 0 seq_num 25) 2026-02-25T08:41:30.967 INFO:tasks.rados.rados.0.trial023.stdout: waiting on 16 2026-02-25T08:41:30.969 INFO:tasks.rados.rados.0.trial023.stdout:48: finishing write tid 4 to trial02342524-48 2026-02-25T08:41:30.969 INFO:tasks.rados.rados.0.trial023.stdout:48: oid 48 updating version 0 to 7 2026-02-25T08:41:30.969 INFO:tasks.rados.rados.0.trial023.stdout:48: oid 48 updating version 7 to 9 2026-02-25T08:41:30.969 INFO:tasks.rados.rados.0.trial023.stdout:48: oid 48 version 9 is already newer than 9 2026-02-25T08:41:30.969 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 48 v 9 (ObjNum 47 snap 0 seq_num 47) dirty exists 2026-02-25T08:41:30.969 INFO:tasks.rados.rados.0.trial023.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2026-02-25T08:41:30.969 INFO:tasks.rados.rados.0.trial023.stdout:48: done (15 left) 2026-02-25T08:41:30.970 INFO:tasks.rados.rados.0.trial023.stdout:56: read oid 3 snap -1 2026-02-25T08:41:30.970 INFO:tasks.rados.rados.0.trial023.stdout:56: expect (ObjNum 2 snap 0 seq_num 2) 2026-02-25T08:41:30.970 INFO:tasks.rados.rados.0.trial023.stdout: waiting on 16 2026-02-25T08:41:30.971 INFO:tasks.rados.rados.0.trial023.stdout:54: expect (ObjNum 35 snap 0 seq_num 35) 2026-02-25T08:41:30.991 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1358123 2026-02-25T08:41:31.015 INFO:tasks.rados.rados.0.trial023.stdout:31: finishing write tid 2 to trial02342524-31 2026-02-25T08:41:31.015 INFO:tasks.rados.rados.0.trial023.stdout:51: finishing write tid 2 to trial02342524-21 2026-02-25T08:41:31.015 INFO:tasks.rados.rados.0.trial023.stdout:46: finishing write tid 1 to trial02342524-46 2026-02-25T08:41:31.015 INFO:tasks.rados.rados.0.trial023.stdout:54: done (15 left) 2026-02-25T08:41:31.016 INFO:tasks.rados.rados.0.trial023.stdout:57: read oid 29 snap -1 2026-02-25T08:41:31.016 INFO:tasks.rados.rados.0.trial023.stdout:57: expect (ObjNum 28 snap 0 seq_num 28) 2026-02-25T08:41:31.016 INFO:tasks.rados.rados.0.trial023.stdout: waiting on 16 2026-02-25T08:41:31.016 INFO:tasks.rados.rados.0.trial023.stdout:51: finishing write tid 3 to trial02342524-21 2026-02-25T08:41:31.016 INFO:tasks.rados.rados.0.trial023.stdout:55: expect (ObjNum 25 snap 0 seq_num 25) 2026-02-25T08:41:31.035 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1637208 2026-02-25T08:41:31.046 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:41:31.068 INFO:tasks.rados.rados.0.trial023.stdout:12: finishing write tid 2 to trial02342524-12 2026-02-25T08:41:31.068 INFO:tasks.rados.rados.0.trial023.stdout:51: finishing write tid 4 to trial02342524-21 2026-02-25T08:41:31.068 INFO:tasks.rados.rados.0.trial023.stdout:46: finishing write tid 2 to trial02342524-46 2026-02-25T08:41:31.068 INFO:tasks.rados.rados.0.trial023.stdout:51: finishing write tid 5 to trial02342524-21 2026-02-25T08:41:31.068 INFO:tasks.rados.rados.0.trial023.stdout:51: oid 21 updating version 0 to 19 2026-02-25T08:41:31.069 INFO:tasks.rados.rados.0.trial023.stdout:51: oid 21 version 19 is already newer than 16 2026-02-25T08:41:31.069 INFO:tasks.rados.rados.0.trial023.stdout:51: oid 21 version 19 is already newer than 19 2026-02-25T08:41:31.069 INFO:tasks.rados.rados.0.trial023.stdout:51: oid 21 version 19 is already newer than 19 2026-02-25T08:41:31.069 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 21 v 19 (ObjNum 50 snap 0 seq_num 50) dirty exists 2026-02-25T08:41:31.069 INFO:tasks.rados.rados.0.trial023.stdout:51: left oid 21 (ObjNum 50 snap 0 seq_num 50) 2026-02-25T08:41:31.069 INFO:tasks.rados.rados.0.trial023.stdout:46: finishing write tid 3 to trial02342524-46 2026-02-25T08:41:31.069 INFO:tasks.rados.rados.0.trial023.stdout:51: done (15 left) 2026-02-25T08:41:31.069 INFO:tasks.rados.rados.0.trial023.stdout:55: done (14 left) 2026-02-25T08:41:31.069 INFO:tasks.rados.rados.0.trial023.stdout:58: delete oid 25 current snap is 0 2026-02-25T08:41:31.069 INFO:tasks.rados.rados.0.trial023.stdout:46: finishing write tid 4 to trial02342524-46 2026-02-25T08:41:31.069 INFO:tasks.rados.rados.0.trial023.stdout:46: oid 46 updating version 0 to 10 2026-02-25T08:41:31.069 INFO:tasks.rados.rados.0.trial023.stdout:46: oid 46 updating version 10 to 12 2026-02-25T08:41:31.070 INFO:tasks.rados.rados.0.trial023.stdout:46: oid 46 version 12 is already newer than 12 2026-02-25T08:41:31.070 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 46 v 12 (ObjNum 45 snap 0 seq_num 45) dirty exists 2026-02-25T08:41:31.070 INFO:tasks.rados.rados.0.trial023.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2026-02-25T08:41:31.070 INFO:tasks.rados.rados.0.trial023.stdout:52: finishing write tid 1 to trial02342524-16 2026-02-25T08:41:31.070 INFO:tasks.rados.rados.0.trial023.stdout:24: finishing write tid 2 to trial02342524-24 2026-02-25T08:41:31.070 INFO:tasks.rados.rados.0.trial023.stdout:52: finishing write tid 2 to trial02342524-16 2026-02-25T08:41:31.070 INFO:tasks.rados.rados.0.trial023.stdout:31: finishing write tid 3 to trial02342524-31 2026-02-25T08:41:31.070 INFO:tasks.rados.rados.0.trial023.stdout:31: finishing write tid 4 to trial02342524-31 2026-02-25T08:41:31.070 INFO:tasks.rados.rados.0.trial023.stdout:31: oid 31 updating version 0 to 6 2026-02-25T08:41:31.070 INFO:tasks.rados.rados.0.trial023.stdout:31: oid 31 version 6 is already newer than 4 2026-02-25T08:41:31.070 INFO:tasks.rados.rados.0.trial023.stdout:31: oid 31 version 6 is already newer than 6 2026-02-25T08:41:31.070 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 31 v 6 (ObjNum 30 snap 0 seq_num 30) dirty exists 2026-02-25T08:41:31.071 INFO:tasks.rados.rados.0.trial023.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2026-02-25T08:41:31.071 INFO:tasks.rados.rados.0.trial023.stdout:52: finishing write tid 3 to trial02342524-16 2026-02-25T08:41:31.071 INFO:tasks.rados.rados.0.trial023.stdout:57: expect (ObjNum 28 snap 0 seq_num 28) 2026-02-25T08:41:31.087 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:41:31.131 INFO:tasks.rados.rados.0.trial023.stdout:52: finishing write tid 4 to trial02342524-16 2026-02-25T08:41:31.131 INFO:tasks.rados.rados.0.trial023.stdout:52: oid 16 updating version 0 to 22 2026-02-25T08:41:31.131 INFO:tasks.rados.rados.0.trial023.stdout:52: oid 16 version 22 is already newer than 22 2026-02-25T08:41:31.131 INFO:tasks.rados.rados.0.trial023.stdout:52: oid 16 version 22 is already newer than 20 2026-02-25T08:41:31.131 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 16 v 22 (ObjNum 51 snap 0 seq_num 51) dirty exists 2026-02-25T08:41:31.132 INFO:tasks.rados.rados.0.trial023.stdout:52: left oid 16 (ObjNum 51 snap 0 seq_num 51) 2026-02-25T08:41:31.132 INFO:tasks.rados.rados.0.trial023.stdout:12: finishing write tid 3 to trial02342524-12 2026-02-25T08:41:31.132 INFO:tasks.rados.rados.0.trial023.stdout:31: done (14 left) 2026-02-25T08:41:31.132 INFO:tasks.rados.rados.0.trial023.stdout:46: done (13 left) 2026-02-25T08:41:31.132 INFO:tasks.rados.rados.0.trial023.stdout:52: done (12 left) 2026-02-25T08:41:31.132 INFO:tasks.rados.rados.0.trial023.stdout:57: done (11 left) 2026-02-25T08:41:31.132 INFO:tasks.rados.rados.0.trial023.stdout:58: done (10 left) 2026-02-25T08:41:31.132 INFO:tasks.rados.rados.0.trial023.stdout:59: write oid 14 current snap is 0 2026-02-25T08:41:31.132 INFO:tasks.rados.rados.0.trial023.stdout:50: finishing write tid 1 to trial02342524-50 2026-02-25T08:41:31.132 INFO:tasks.rados.rados.0.trial023.stdout:59: seq_num 53 ranges {490771=686315,1885075=406346} 2026-02-25T08:41:31.135 INFO:tasks.rados.rados.0.trial023.stdout:59: writing trial02342524-14 from 490771 to 1177086 tid 1 2026-02-25T08:41:31.140 INFO:tasks.rados.rados.0.trial023.stdout:59: writing trial02342524-14 from 1885075 to 2291421 tid 2 2026-02-25T08:41:31.140 INFO:tasks.rados.rados.0.trial023.stdout:60: write oid 29 current snap is 0 2026-02-25T08:41:31.140 INFO:tasks.rados.rados.0.trial023.stdout:12: finishing write tid 4 to trial02342524-12 2026-02-25T08:41:31.140 INFO:tasks.rados.rados.0.trial023.stdout:12: oid 12 updating version 0 to 6 2026-02-25T08:41:31.141 INFO:tasks.rados.rados.0.trial023.stdout:12: oid 12 version 6 is already newer than 4 2026-02-25T08:41:31.141 INFO:tasks.rados.rados.0.trial023.stdout:12: oid 12 version 6 is already newer than 6 2026-02-25T08:41:31.141 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 12 v 6 (ObjNum 11 snap 0 seq_num 11) dirty exists 2026-02-25T08:41:31.141 INFO:tasks.rados.rados.0.trial023.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2026-02-25T08:41:31.141 INFO:tasks.rados.rados.0.trial023.stdout:49: finishing write tid 1 to trial02342524-49 2026-02-25T08:41:31.141 INFO:tasks.rados.rados.0.trial023.stdout:50: finishing write tid 2 to trial02342524-50 2026-02-25T08:41:31.141 INFO:tasks.rados.rados.0.trial023.stdout:60: seq_num 54 ranges {507578=477917,1543557=550004,2833665=73913} 2026-02-25T08:41:31.143 INFO:tasks.rados.rados.0.trial023.stdout:60: writing trial02342524-29 from 507578 to 985495 tid 1 2026-02-25T08:41:31.148 INFO:tasks.rados.rados.0.trial023.stdout:60: writing trial02342524-29 from 1543557 to 2093561 tid 2 2026-02-25T08:41:31.152 INFO:tasks.rados.rados.0.trial023.stdout:60: writing trial02342524-29 from 2833665 to 2907578 tid 3 2026-02-25T08:41:31.152 INFO:tasks.rados.rados.0.trial023.stdout:12: done (11 left) 2026-02-25T08:41:31.152 INFO:tasks.rados.rados.0.trial023.stdout:61: read oid 38 snap -1 2026-02-25T08:41:31.152 INFO:tasks.rados.rados.0.trial023.stdout:61: expect (ObjNum 37 snap 0 seq_num 37) 2026-02-25T08:41:31.152 INFO:tasks.rados.rados.0.trial023.stdout:62: snap_create 2026-02-25T08:41:31.152 INFO:tasks.rados.rados.0.trial023.stdout:50: finishing write tid 3 to trial02342524-50 2026-02-25T08:41:31.152 INFO:tasks.rados.rados.0.trial023.stdout:50: finishing write tid 4 to trial02342524-50 2026-02-25T08:41:31.152 INFO:tasks.rados.rados.0.trial023.stdout:50: oid 50 updating version 0 to 21 2026-02-25T08:41:31.152 INFO:tasks.rados.rados.0.trial023.stdout:50: oid 50 version 21 is already newer than 21 2026-02-25T08:41:31.153 INFO:tasks.rados.rados.0.trial023.stdout:50: oid 50 version 21 is already newer than 19 2026-02-25T08:41:31.153 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 50 v 21 (ObjNum 49 snap 0 seq_num 49) dirty exists 2026-02-25T08:41:31.153 INFO:tasks.rados.rados.0.trial023.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2026-02-25T08:41:31.153 INFO:tasks.rados.rados.0.trial023.stdout:53: expect (ObjNum 19 snap 0 seq_num 19) 2026-02-25T08:41:31.153 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:41:31.171 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 57176 2026-02-25T08:41:31.190 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:41:31.210 INFO:tasks.rados.rados.0.trial023.stdout:24: finishing write tid 3 to trial02342524-24 2026-02-25T08:41:31.211 INFO:tasks.rados.rados.0.trial023.stdout:56: expect (ObjNum 2 snap 0 seq_num 2) 2026-02-25T08:41:31.211 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 49191 2026-02-25T08:41:31.227 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 363198 2026-02-25T08:41:31.260 INFO:tasks.rados.rados.0.trial023.stdout:24: finishing write tid 4 to trial02342524-24 2026-02-25T08:41:31.260 INFO:tasks.rados.rados.0.trial023.stdout:24: oid 24 updating version 0 to 6 2026-02-25T08:41:31.260 INFO:tasks.rados.rados.0.trial023.stdout:24: oid 24 version 6 is already newer than 6 2026-02-25T08:41:31.260 INFO:tasks.rados.rados.0.trial023.stdout:24: oid 24 version 6 is already newer than 4 2026-02-25T08:41:31.260 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 24 v 6 (ObjNum 23 snap 0 seq_num 23) dirty exists 2026-02-25T08:41:31.260 INFO:tasks.rados.rados.0.trial023.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2026-02-25T08:41:31.260 INFO:tasks.rados.rados.0.trial023.stdout:49: finishing write tid 2 to trial02342524-49 2026-02-25T08:41:31.260 INFO:tasks.rados.rados.0.trial023.stdout:47: finishing write tid 1 to trial02342524-47 2026-02-25T08:41:31.260 INFO:tasks.rados.rados.0.trial023.stdout:30: finishing write tid 1 to trial02342524-30 2026-02-25T08:41:31.260 INFO:tasks.rados.rados.0.trial023.stdout:61: expect (ObjNum 37 snap 0 seq_num 37) 2026-02-25T08:41:31.308 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1042256 2026-02-25T08:41:31.323 INFO:tasks.rados.rados.0.trial023.stdout:60: finishing write tid 1 to trial02342524-29 2026-02-25T08:41:31.323 INFO:tasks.rados.rados.0.trial023.stdout:60: finishing write tid 2 to trial02342524-29 2026-02-25T08:41:31.323 INFO:tasks.rados.rados.0.trial023.stdout:47: finishing write tid 2 to trial02342524-47 2026-02-25T08:41:31.323 INFO:tasks.rados.rados.0.trial023.stdout:60: finishing write tid 3 to trial02342524-29 2026-02-25T08:41:31.323 INFO:tasks.rados.rados.0.trial023.stdout:60: finishing write tid 4 to trial02342524-29 2026-02-25T08:41:31.323 INFO:tasks.rados.rados.0.trial023.stdout:60: finishing write tid 5 to trial02342524-29 2026-02-25T08:41:31.323 INFO:tasks.rados.rados.0.trial023.stdout:60: oid 29 updating version 0 to 22 2026-02-25T08:41:31.323 INFO:tasks.rados.rados.0.trial023.stdout:60: oid 29 version 22 is already newer than 22 2026-02-25T08:41:31.323 INFO:tasks.rados.rados.0.trial023.stdout:60: oid 29 version 22 is already newer than 22 2026-02-25T08:41:31.324 INFO:tasks.rados.rados.0.trial023.stdout:60: oid 29 version 22 is already newer than 22 2026-02-25T08:41:31.324 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 29 v 22 (ObjNum 54 snap 0 seq_num 54) dirty exists 2026-02-25T08:41:31.324 INFO:tasks.rados.rados.0.trial023.stdout:60: left oid 29 (ObjNum 54 snap 0 seq_num 54) 2026-02-25T08:41:31.324 INFO:tasks.rados.rados.0.trial023.stdout:49: finishing write tid 3 to trial02342524-49 2026-02-25T08:41:31.324 INFO:tasks.rados.rados.0.trial023.stdout:49: finishing write tid 4 to trial02342524-49 2026-02-25T08:41:31.324 INFO:tasks.rados.rados.0.trial023.stdout:49: oid 49 updating version 0 to 7 2026-02-25T08:41:31.324 INFO:tasks.rados.rados.0.trial023.stdout:49: oid 49 updating version 7 to 9 2026-02-25T08:41:31.324 INFO:tasks.rados.rados.0.trial023.stdout:49: oid 49 version 9 is already newer than 9 2026-02-25T08:41:31.324 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 49 v 9 (ObjNum 48 snap 0 seq_num 48) dirty exists 2026-02-25T08:41:31.324 INFO:tasks.rados.rados.0.trial023.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2026-02-25T08:41:31.324 INFO:tasks.rados.rados.0.trial023.stdout:30: finishing write tid 2 to trial02342524-30 2026-02-25T08:41:31.324 INFO:tasks.rados.rados.0.trial023.stdout:30: finishing write tid 3 to trial02342524-30 2026-02-25T08:41:31.325 INFO:tasks.rados.rados.0.trial023.stdout:30: finishing write tid 4 to trial02342524-30 2026-02-25T08:41:31.325 INFO:tasks.rados.rados.0.trial023.stdout:30: oid 30 updating version 0 to 7 2026-02-25T08:41:31.325 INFO:tasks.rados.rados.0.trial023.stdout:30: oid 30 updating version 7 to 9 2026-02-25T08:41:31.325 INFO:tasks.rados.rados.0.trial023.stdout:30: oid 30 version 9 is already newer than 9 2026-02-25T08:41:31.325 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 30 v 9 (ObjNum 29 snap 0 seq_num 29) dirty exists 2026-02-25T08:41:31.325 INFO:tasks.rados.rados.0.trial023.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2026-02-25T08:41:31.325 INFO:tasks.rados.rados.0.trial023.stdout:59: finishing write tid 1 to trial02342524-14 2026-02-25T08:41:31.325 INFO:tasks.rados.rados.0.trial023.stdout:47: finishing write tid 3 to trial02342524-47 2026-02-25T08:41:31.325 INFO:tasks.rados.rados.0.trial023.stdout:47: finishing write tid 4 to trial02342524-47 2026-02-25T08:41:31.325 INFO:tasks.rados.rados.0.trial023.stdout:47: oid 47 updating version 0 to 7 2026-02-25T08:41:31.325 INFO:tasks.rados.rados.0.trial023.stdout:47: oid 47 updating version 7 to 9 2026-02-25T08:41:31.325 INFO:tasks.rados.rados.0.trial023.stdout:47: oid 47 version 9 is already newer than 9 2026-02-25T08:41:31.326 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 47 v 9 (ObjNum 46 snap 0 seq_num 46) dirty exists 2026-02-25T08:41:31.326 INFO:tasks.rados.rados.0.trial023.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2026-02-25T08:41:31.326 INFO:tasks.rados.rados.0.trial023.stdout:59: finishing write tid 2 to trial02342524-14 2026-02-25T08:41:31.326 INFO:tasks.rados.rados.0.trial023.stdout:59: finishing write tid 3 to trial02342524-14 2026-02-25T08:41:31.326 INFO:tasks.rados.rados.0.trial023.stdout:59: finishing write tid 4 to trial02342524-14 2026-02-25T08:41:31.326 INFO:tasks.rados.rados.0.trial023.stdout:59: oid 14 updating version 0 to 9 2026-02-25T08:41:31.326 INFO:tasks.rados.rados.0.trial023.stdout:59: oid 14 version 9 is already newer than 9 2026-02-25T08:41:31.326 INFO:tasks.rados.rados.0.trial023.stdout:59: oid 14 version 9 is already newer than 9 2026-02-25T08:41:31.326 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 14 v 9 (ObjNum 53 snap 0 seq_num 53) dirty exists 2026-02-25T08:41:31.326 INFO:tasks.rados.rados.0.trial023.stdout:59: left oid 14 (ObjNum 53 snap 0 seq_num 53) 2026-02-25T08:41:31.326 INFO:tasks.rados.rados.0.trial023.stdout:32: finishing write tid 1 to trial02342524-32 2026-02-25T08:41:31.326 INFO:tasks.rados.rados.0.trial023.stdout:32: finishing write tid 2 to trial02342524-32 2026-02-25T08:41:31.327 INFO:tasks.rados.rados.0.trial023.stdout:32: finishing write tid 3 to trial02342524-32 2026-02-25T08:41:31.327 INFO:tasks.rados.rados.0.trial023.stdout:32: finishing write tid 4 to trial02342524-32 2026-02-25T08:41:31.327 INFO:tasks.rados.rados.0.trial023.stdout:32: oid 32 updating version 0 to 12 2026-02-25T08:41:31.327 INFO:tasks.rados.rados.0.trial023.stdout:32: oid 32 version 12 is already newer than 10 2026-02-25T08:41:31.327 INFO:tasks.rados.rados.0.trial023.stdout:32: oid 32 version 12 is already newer than 12 2026-02-25T08:41:31.327 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 32 v 12 (ObjNum 31 snap 0 seq_num 31) dirty exists 2026-02-25T08:41:31.327 INFO:tasks.rados.rados.0.trial023.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2026-02-25T08:41:31.327 INFO:tasks.rados.rados.0.trial023.stdout:41: finishing write tid 1 to trial02342524-41 2026-02-25T08:41:31.327 INFO:tasks.rados.rados.0.trial023.stdout:41: finishing write tid 2 to trial02342524-41 2026-02-25T08:41:31.327 INFO:tasks.rados.rados.0.trial023.stdout:41: finishing write tid 3 to trial02342524-41 2026-02-25T08:41:31.327 INFO:tasks.rados.rados.0.trial023.stdout:41: finishing write tid 4 to trial02342524-41 2026-02-25T08:41:31.327 INFO:tasks.rados.rados.0.trial023.stdout:41: oid 41 updating version 0 to 15 2026-02-25T08:41:31.328 INFO:tasks.rados.rados.0.trial023.stdout:41: oid 41 version 15 is already newer than 15 2026-02-25T08:41:31.328 INFO:tasks.rados.rados.0.trial023.stdout:41: oid 41 version 15 is already newer than 13 2026-02-25T08:41:31.328 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 41 v 15 (ObjNum 40 snap 0 seq_num 40) dirty exists 2026-02-25T08:41:31.328 INFO:tasks.rados.rados.0.trial023.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2026-02-25T08:41:31.394 INFO:tasks.rados.rados.0.trial023.stdout:24: done (12 left) 2026-02-25T08:41:31.394 INFO:tasks.rados.rados.0.trial023.stdout:30: done (11 left) 2026-02-25T08:41:31.394 INFO:tasks.rados.rados.0.trial023.stdout:32: done (10 left) 2026-02-25T08:41:31.394 INFO:tasks.rados.rados.0.trial023.stdout:41: done (9 left) 2026-02-25T08:41:31.394 INFO:tasks.rados.rados.0.trial023.stdout:47: done (8 left) 2026-02-25T08:41:31.394 INFO:tasks.rados.rados.0.trial023.stdout:49: done (7 left) 2026-02-25T08:41:31.395 INFO:tasks.rados.rados.0.trial023.stdout:50: done (6 left) 2026-02-25T08:41:31.395 INFO:tasks.rados.rados.0.trial023.stdout:53: done (5 left) 2026-02-25T08:41:31.395 INFO:tasks.rados.rados.0.trial023.stdout:56: done (4 left) 2026-02-25T08:41:31.395 INFO:tasks.rados.rados.0.trial023.stdout:59: done (3 left) 2026-02-25T08:41:31.395 INFO:tasks.rados.rados.0.trial023.stdout:60: done (2 left) 2026-02-25T08:41:31.395 INFO:tasks.rados.rados.0.trial023.stdout:61: done (1 left) 2026-02-25T08:41:31.395 INFO:tasks.rados.rados.0.trial023.stdout:62: done (0 left) 2026-02-25T08:41:31.395 INFO:tasks.rados.rados.0.trial023.stdout:63: read oid 49 snap -1 2026-02-25T08:41:31.395 INFO:tasks.rados.rados.0.trial023.stdout:63: expect (ObjNum 48 snap 0 seq_num 48) 2026-02-25T08:41:31.395 INFO:tasks.rados.rados.0.trial023.stdout:64: delete oid 24 current snap is 1 2026-02-25T08:41:31.438 INFO:tasks.rados.rados.0.trial023.stdout:63: expect (ObjNum 48 snap 0 seq_num 48) 2026-02-25T08:41:31.496 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2026-02-25T08:41:31.496 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-25T08:41:31.497 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [0, 1, 2, 3] 2026-02-25T08:41:31.497 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-25T08:41:31.497 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T08:41:31.510 INFO:tasks.rados.rados.0.trial023.stdout:63: done (1 left) 2026-02-25T08:41:31.510 INFO:tasks.rados.rados.0.trial023.stdout:64: done (0 left) 2026-02-25T08:41:31.510 INFO:tasks.rados.rados.0.trial023.stdout:65: read oid 39 snap 0 2026-02-25T08:41:31.510 INFO:tasks.rados.rados.0.trial023.stdout:65: expect (ObjNum 38 snap 0 seq_num 38) 2026-02-25T08:41:31.510 INFO:tasks.rados.rados.0.trial023.stdout:66: read oid 16 snap -1 2026-02-25T08:41:31.510 INFO:tasks.rados.rados.0.trial023.stdout:66: expect (ObjNum 51 snap 0 seq_num 51) 2026-02-25T08:41:31.510 INFO:tasks.rados.rados.0.trial023.stdout:67: write oid 45 current snap is 1 2026-02-25T08:41:31.510 INFO:tasks.rados.rados.0.trial023.stdout:67: seq_num 56 ranges {541192=628415,1660521=553673,2888511=52681} 2026-02-25T08:41:31.514 INFO:tasks.rados.rados.0.trial023.stdout:67: writing trial02342524-45 from 541192 to 1169607 tid 1 2026-02-25T08:41:31.518 INFO:tasks.rados.rados.0.trial023.stdout:67: writing trial02342524-45 from 1660521 to 2214194 tid 2 2026-02-25T08:41:31.521 INFO:tasks.rados.rados.0.trial023.stdout:67: writing trial02342524-45 from 2888511 to 2941192 tid 3 2026-02-25T08:41:31.521 INFO:tasks.rados.rados.0.trial023.stdout:68: read oid 48 snap -1 2026-02-25T08:41:31.521 INFO:tasks.rados.rados.0.trial023.stdout:68: expect (ObjNum 47 snap 0 seq_num 47) 2026-02-25T08:41:31.521 INFO:tasks.rados.rados.0.trial023.stdout:69: snap_create 2026-02-25T08:41:31.529 INFO:tasks.rados.rados.0.trial023.stdout:65: expect (ObjNum 38 snap 0 seq_num 38) 2026-02-25T08:41:31.529 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:41:31.556 INFO:tasks.ceph.osd.1:Stopped 2026-02-25T08:41:31.571 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:41:31.594 INFO:tasks.rados.rados.0.trial023.stdout:68: expect (ObjNum 47 snap 0 seq_num 47) 2026-02-25T08:41:31.594 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:41:31.632 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 995082 2026-02-25T08:41:32.396 INFO:tasks.rados.rados.0.trial023.stdout:65: done (4 left) 2026-02-25T08:41:32.396 INFO:tasks.rados.rados.0.trial023.stdout:68: done (3 left) 2026-02-25T08:41:32.396 INFO:tasks.rados.rados.0.trial023.stdout:69: done (2 left) 2026-02-25T08:41:32.396 INFO:tasks.rados.rados.0.trial023.stdout:70: delete oid 40 current snap is 2 2026-02-25T08:41:32.430 INFO:tasks.rados.rados.0.trial023.stdout:70: done (2 left) 2026-02-25T08:41:32.430 INFO:tasks.rados.rados.0.trial023.stdout:71: snap_remove snap 1 2026-02-25T08:41:32.853 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:41:32.930 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:32.928+0000 7f520b1dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:32.933 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:41:32.932+0000 7f520b1dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:33.400 INFO:tasks.rados.rados.0.trial023.stdout:71: done (2 left) 2026-02-25T08:41:33.401 INFO:tasks.rados.rados.0.trial023.stdout:72: read oid 25 snap -1 2026-02-25T08:41:33.401 INFO:tasks.rados.rados.0.trial023.stdout:72: expect deleted 2026-02-25T08:41:33.401 INFO:tasks.rados.rados.0.trial023.stdout:73: write oid 4 current snap is 2 2026-02-25T08:41:33.401 INFO:tasks.rados.rados.0.trial023.stdout:73: seq_num 58 ranges {574806=778913,2061132=557342} 2026-02-25T08:41:33.405 INFO:tasks.rados.rados.0.trial023.stdout:73: writing trial02342524-4 from 574806 to 1353719 tid 1 2026-02-25T08:41:33.410 INFO:tasks.rados.rados.0.trial023.stdout:73: writing trial02342524-4 from 2061132 to 2618474 tid 2 2026-02-25T08:41:33.410 INFO:tasks.rados.rados.0.trial023.stdout:74: write (excl) oid 33 current snap is 2 2026-02-25T08:41:33.410 INFO:tasks.rados.rados.0.trial023.stdout:74: seq_num 59 ranges {591613=454162,1719614=417353,2570780=420833} 2026-02-25T08:41:33.414 INFO:tasks.rados.rados.0.trial023.stdout:74: writing trial02342524-33 from 591613 to 1045775 tid 1 2026-02-25T08:41:33.418 INFO:tasks.rados.rados.0.trial023.stdout:74: writing trial02342524-33 from 1719614 to 2136967 tid 2 2026-02-25T08:41:33.422 INFO:tasks.rados.rados.0.trial023.stdout:74: writing trial02342524-33 from 2570780 to 2991613 tid 3 2026-02-25T08:41:33.422 INFO:tasks.rados.rados.0.trial023.stdout:75: write (excl) oid 12 current snap is 2 2026-02-25T08:41:33.422 INFO:tasks.rados.rados.0.trial023.stdout:75: seq_num 60 ranges {608420=529411,1778096=561011,2881850=126570} 2026-02-25T08:41:33.426 INFO:tasks.rados.rados.0.trial023.stdout:75: writing trial02342524-12 from 608420 to 1137831 tid 1 2026-02-25T08:41:33.430 INFO:tasks.rados.rados.0.trial023.stdout:75: writing trial02342524-12 from 1778096 to 2339107 tid 2 2026-02-25T08:41:33.433 INFO:tasks.rados.rados.0.trial023.stdout:75: writing trial02342524-12 from 2881850 to 3008420 tid 3 2026-02-25T08:41:33.433 INFO:tasks.rados.rados.0.trial023.stdout:76: snap_remove snap 0 2026-02-25T08:41:34.406 INFO:tasks.rados.rados.0.trial023.stdout:76: done (6 left) 2026-02-25T08:41:34.406 INFO:tasks.rados.rados.0.trial023.stdout:77: write oid 18 current snap is 2 2026-02-25T08:41:34.406 INFO:tasks.rados.rados.0.trial023.stdout:75: finishing write tid 1 to trial02342524-12 2026-02-25T08:41:34.406 INFO:tasks.rados.rados.0.trial023.stdout:75: finishing write tid 2 to trial02342524-12 2026-02-25T08:41:34.406 INFO:tasks.rados.rados.0.trial023.stdout:77: seq_num 61 ranges {625227=721013,1952931=704669} 2026-02-25T08:41:34.411 INFO:tasks.rados.rados.0.trial023.stdout:77: writing trial02342524-18 from 625227 to 1346240 tid 1 2026-02-25T08:41:34.417 INFO:tasks.rados.rados.0.trial023.stdout:77: writing trial02342524-18 from 1952931 to 2657600 tid 2 2026-02-25T08:41:34.417 INFO:tasks.rados.rados.0.trial023.stdout:72: done (6 left) 2026-02-25T08:41:34.417 INFO:tasks.rados.rados.0.trial023.stdout:78: snap_create 2026-02-25T08:41:34.418 INFO:tasks.rados.rados.0.trial023.stdout:75: finishing write tid 3 to trial02342524-12 2026-02-25T08:41:34.418 INFO:tasks.rados.rados.0.trial023.stdout:75: finishing write tid 4 to trial02342524-12 2026-02-25T08:41:34.418 INFO:tasks.rados.rados.0.trial023.stdout:75: finishing write tid 5 to trial02342524-12 2026-02-25T08:41:34.418 INFO:tasks.rados.rados.0.trial023.stdout:75: oid 12 updating version 0 to 14 2026-02-25T08:41:34.418 INFO:tasks.rados.rados.0.trial023.stdout:75: oid 12 version 14 is already newer than 11 2026-02-25T08:41:34.418 INFO:tasks.rados.rados.0.trial023.stdout:75: oid 12 version 14 is already newer than 14 2026-02-25T08:41:34.418 INFO:tasks.rados.rados.0.trial023.stdout:75: oid 12 version 14 is already newer than 14 2026-02-25T08:41:34.418 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 12 v 14 (ObjNum 60 snap 2 seq_num 60) dirty exists 2026-02-25T08:41:34.418 INFO:tasks.rados.rados.0.trial023.stdout:75: left oid 12 (ObjNum 60 snap 2 seq_num 60) 2026-02-25T08:41:35.406 INFO:tasks.rados.rados.0.trial023.stdout:75: done (6 left) 2026-02-25T08:41:35.406 INFO:tasks.rados.rados.0.trial023.stdout:78: done (5 left) 2026-02-25T08:41:35.407 INFO:tasks.rados.rados.0.trial023.stdout:79: write oid 15 current snap is 3 2026-02-25T08:41:35.407 INFO:tasks.rados.rados.0.trial023.stdout:79: seq_num 62 ranges {642034=796262,1895060=564680} 2026-02-25T08:41:35.412 INFO:tasks.rados.rados.0.trial023.stdout:79: writing trial02342524-15 from 642034 to 1438296 tid 1 2026-02-25T08:41:35.416 INFO:tasks.rados.rados.0.trial023.stdout:79: writing trial02342524-15 from 1895060 to 2459740 tid 2 2026-02-25T08:41:35.416 INFO:tasks.rados.rados.0.trial023.stdout:80: write oid 8 current snap is 3 2026-02-25T08:41:35.416 INFO:tasks.rados.rados.0.trial023.stdout:80: seq_num 63 ranges {658841=471511,1553542=708338,2964119=94722} 2026-02-25T08:41:35.420 INFO:tasks.rados.rados.0.trial023.stdout:80: writing trial02342524-8 from 658841 to 1130352 tid 1 2026-02-25T08:41:35.424 INFO:tasks.rados.rados.0.trial023.stdout:80: writing trial02342524-8 from 1553542 to 2261880 tid 2 2026-02-25T08:41:35.427 INFO:tasks.rados.rados.0.trial023.stdout:80: writing trial02342524-8 from 2964119 to 3058841 tid 3 2026-02-25T08:41:35.427 INFO:tasks.rados.rados.0.trial023.stdout:81: write oid 9 current snap is 3 2026-02-25T08:41:35.427 INFO:tasks.rados.rados.0.trial023.stdout:81: seq_num 64 ranges {675648=546760,2012024=568349} 2026-02-25T08:41:35.432 INFO:tasks.rados.rados.0.trial023.stdout:81: writing trial02342524-9 from 675648 to 1222408 tid 1 2026-02-25T08:41:35.437 INFO:tasks.rados.rados.0.trial023.stdout:81: writing trial02342524-9 from 2012024 to 2580373 tid 2 2026-02-25T08:41:35.437 INFO:tasks.rados.rados.0.trial023.stdout:82: delete oid 36 current snap is 3 2026-02-25T08:41:35.493 INFO:tasks.rados.rados.0.trial023.stdout:81: finishing write tid 1 to trial02342524-9 2026-02-25T08:41:35.498 INFO:tasks.rados.rados.0.trial023.stdout:79: finishing write tid 1 to trial02342524-15 2026-02-25T08:41:35.499 INFO:tasks.rados.rados.0.trial023.stdout:81: finishing write tid 2 to trial02342524-9 2026-02-25T08:41:35.503 INFO:tasks.rados.rados.0.trial023.stdout:79: finishing write tid 2 to trial02342524-15 2026-02-25T08:41:35.504 INFO:tasks.rados.rados.0.trial023.stdout:81: finishing write tid 3 to trial02342524-9 2026-02-25T08:41:35.506 INFO:tasks.rados.rados.0.trial023.stdout:81: finishing write tid 4 to trial02342524-9 2026-02-25T08:41:35.506 INFO:tasks.rados.rados.0.trial023.stdout:81: oid 9 updating version 0 to 16 2026-02-25T08:41:35.506 INFO:tasks.rados.rados.0.trial023.stdout:81: oid 9 version 16 is already newer than 16 2026-02-25T08:41:35.506 INFO:tasks.rados.rados.0.trial023.stdout:81: oid 9 version 16 is already newer than 16 2026-02-25T08:41:35.506 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 9 v 16 (ObjNum 64 snap 3 seq_num 64) dirty exists 2026-02-25T08:41:35.506 INFO:tasks.rados.rados.0.trial023.stdout:81: left oid 9 (ObjNum 64 snap 3 seq_num 64) 2026-02-25T08:41:35.509 INFO:tasks.rados.rados.0.trial023.stdout:79: finishing write tid 3 to trial02342524-15 2026-02-25T08:41:35.510 INFO:tasks.rados.rados.0.trial023.stdout:79: finishing write tid 4 to trial02342524-15 2026-02-25T08:41:35.510 INFO:tasks.rados.rados.0.trial023.stdout:79: oid 15 updating version 0 to 17 2026-02-25T08:41:35.510 INFO:tasks.rados.rados.0.trial023.stdout:79: oid 15 version 17 is already newer than 17 2026-02-25T08:41:35.511 INFO:tasks.rados.rados.0.trial023.stdout:79: oid 15 version 17 is already newer than 17 2026-02-25T08:41:35.511 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 15 v 17 (ObjNum 62 snap 3 seq_num 62) dirty exists 2026-02-25T08:41:35.511 INFO:tasks.rados.rados.0.trial023.stdout:79: left oid 15 (ObjNum 62 snap 3 seq_num 62) 2026-02-25T08:41:35.517 INFO:tasks.rados.rados.0.trial023.stdout:79: done (8 left) 2026-02-25T08:41:35.518 INFO:tasks.rados.rados.0.trial023.stdout:81: done (7 left) 2026-02-25T08:41:35.518 INFO:tasks.rados.rados.0.trial023.stdout:82: done (6 left) 2026-02-25T08:41:35.518 INFO:tasks.rados.rados.0.trial023.stdout:83: write (excl) oid 38 current snap is 3 2026-02-25T08:41:35.518 INFO:tasks.rados.rados.0.trial023.stdout:83: seq_num 66 ranges {709262=697258,2012635=572018,3046388=62874} 2026-02-25T08:41:35.523 INFO:tasks.rados.rados.0.trial023.stdout:83: writing trial02342524-38 from 709262 to 1406520 tid 1 2026-02-25T08:41:35.528 INFO:tasks.rados.rados.0.trial023.stdout:83: writing trial02342524-38 from 2012635 to 2584653 tid 2 2026-02-25T08:41:35.530 INFO:tasks.rados.rados.0.trial023.stdout:83: writing trial02342524-38 from 3046388 to 3109262 tid 3 2026-02-25T08:41:35.530 INFO:tasks.rados.rados.0.trial023.stdout:84: write oid 35 current snap is 3 2026-02-25T08:41:35.530 INFO:tasks.rados.rados.0.trial023.stdout:84: seq_num 67 ranges {726069=772507,2071117=715676} 2026-02-25T08:41:35.535 INFO:tasks.rados.rados.0.trial023.stdout:84: writing trial02342524-35 from 726069 to 1498576 tid 1 2026-02-25T08:41:35.541 INFO:tasks.rados.rados.0.trial023.stdout:84: writing trial02342524-35 from 2071117 to 2786793 tid 2 2026-02-25T08:41:35.541 INFO:tasks.rados.rados.0.trial023.stdout:85: read oid 50 snap -1 2026-02-25T08:41:35.541 INFO:tasks.rados.rados.0.trial023.stdout:85: expect (ObjNum 49 snap 0 seq_num 49) 2026-02-25T08:41:35.541 INFO:tasks.rados.rados.0.trial023.stdout:86: read oid 32 snap -1 2026-02-25T08:41:35.541 INFO:tasks.rados.rados.0.trial023.stdout:86: expect (ObjNum 31 snap 0 seq_num 31) 2026-02-25T08:41:35.541 INFO:tasks.rados.rados.0.trial023.stdout:87: write oid 10 current snap is 3 2026-02-25T08:41:35.541 INFO:tasks.rados.rados.0.trial023.stdout:87: seq_num 68 ranges {742876=447756,1729599=575687,3101234=41642} 2026-02-25T08:41:35.545 INFO:tasks.rados.rados.0.trial023.stdout:87: writing trial02342524-10 from 742876 to 1190632 tid 1 2026-02-25T08:41:35.549 INFO:tasks.rados.rados.0.trial023.stdout:87: writing trial02342524-10 from 1729599 to 2305286 tid 2 2026-02-25T08:41:35.552 INFO:tasks.rados.rados.0.trial023.stdout:87: writing trial02342524-10 from 3101234 to 3142876 tid 3 2026-02-25T08:41:35.552 INFO:tasks.rados.rados.0.trial023.stdout:88: snap_remove snap 2 2026-02-25T08:41:36.411 INFO:tasks.rados.rados.0.trial023.stdout:88: done (11 left) 2026-02-25T08:41:36.411 INFO:tasks.rados.rados.0.trial023.stdout:89: write oid 20 current snap is 3 2026-02-25T08:41:36.411 INFO:tasks.rados.rados.0.trial023.stdout:89: seq_num 69 ranges {759683=639358,1904434=719345,3128657=31026} 2026-02-25T08:41:36.416 INFO:tasks.rados.rados.0.trial023.stdout:89: writing trial02342524-20 from 759683 to 1399041 tid 1 2026-02-25T08:41:36.420 INFO:tasks.rados.rados.0.trial023.stdout:89: writing trial02342524-20 from 1904434 to 2623779 tid 2 2026-02-25T08:41:36.423 INFO:tasks.rados.rados.0.trial023.stdout:89: writing trial02342524-20 from 3128657 to 3159683 tid 3 2026-02-25T08:41:36.423 INFO:tasks.rados.rados.0.trial023.stdout:90: read oid 12 snap -1 2026-02-25T08:41:36.423 INFO:tasks.rados.rados.0.trial023.stdout:90: expect (ObjNum 60 snap 2 seq_num 60) 2026-02-25T08:41:36.423 INFO:tasks.rados.rados.0.trial023.stdout:91: read oid 40 snap -1 2026-02-25T08:41:36.423 INFO:tasks.rados.rados.0.trial023.stdout:91: expect deleted 2026-02-25T08:41:36.423 INFO:tasks.rados.rados.0.trial023.stdout:92: write oid 34 current snap is 3 2026-02-25T08:41:36.423 INFO:tasks.rados.rados.0.trial023.stdout:86: expect (ObjNum 31 snap 0 seq_num 31) 2026-02-25T08:41:36.446 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1574561 2026-02-25T08:41:36.460 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1358330 2026-02-25T08:41:36.473 INFO:tasks.rados.rados.0.trial023.stdout:85: expect (ObjNum 49 snap 0 seq_num 49) 2026-02-25T08:41:36.485 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 30049 2026-02-25T08:41:36.508 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:41:36.531 INFO:tasks.rados.rados.0.trial023.stdout:84: finishing write tid 1 to trial02342524-35 2026-02-25T08:41:36.532 INFO:tasks.rados.rados.0.trial023.stdout:84: finishing write tid 2 to trial02342524-35 2026-02-25T08:41:36.532 INFO:tasks.rados.rados.0.trial023.stdout:92: seq_num 70 ranges {776490=714607,2246563=579356} 2026-02-25T08:41:36.537 INFO:tasks.rados.rados.0.trial023.stdout:92: writing trial02342524-34 from 776490 to 1491097 tid 1 2026-02-25T08:41:36.543 INFO:tasks.rados.rados.0.trial023.stdout:92: writing trial02342524-34 from 2246563 to 2825919 tid 2 2026-02-25T08:41:36.543 INFO:tasks.rados.rados.0.trial023.stdout:85: done (14 left) 2026-02-25T08:41:36.543 INFO:tasks.rados.rados.0.trial023.stdout:86: done (13 left) 2026-02-25T08:41:36.543 INFO:tasks.rados.rados.0.trial023.stdout:93: read oid 31 snap -1 2026-02-25T08:41:36.543 INFO:tasks.rados.rados.0.trial023.stdout:93: expect (ObjNum 30 snap 0 seq_num 30) 2026-02-25T08:41:36.543 INFO:tasks.rados.rados.0.trial023.stdout:94: read oid 19 snap -1 2026-02-25T08:41:36.543 INFO:tasks.rados.rados.0.trial023.stdout:94: expect (ObjNum 18 snap 0 seq_num 18) 2026-02-25T08:41:36.543 INFO:tasks.rados.rados.0.trial023.stdout:95: write oid 9 current snap is 3 2026-02-25T08:41:36.543 INFO:tasks.rados.rados.0.trial023.stdout:84: finishing write tid 3 to trial02342524-35 2026-02-25T08:41:36.544 INFO:tasks.rados.rados.0.trial023.stdout:84: finishing write tid 4 to trial02342524-35 2026-02-25T08:41:36.544 INFO:tasks.rados.rados.0.trial023.stdout:84: oid 35 updating version 0 to 20 2026-02-25T08:41:36.544 INFO:tasks.rados.rados.0.trial023.stdout:84: oid 35 version 20 is already newer than 20 2026-02-25T08:41:36.544 INFO:tasks.rados.rados.0.trial023.stdout:84: oid 35 version 20 is already newer than 20 2026-02-25T08:41:36.544 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 35 v 20 (ObjNum 67 snap 3 seq_num 67) dirty exists 2026-02-25T08:41:36.544 INFO:tasks.rados.rados.0.trial023.stdout:84: left oid 35 (ObjNum 67 snap 3 seq_num 67) 2026-02-25T08:41:36.544 INFO:tasks.rados.rados.0.trial023.stdout:95: seq_num 71 ranges {793297=789856,2305045=723014} 2026-02-25T08:41:36.548 INFO:tasks.rados.rados.0.trial023.stdout:95: writing trial02342524-9 from 793297 to 1583153 tid 1 2026-02-25T08:41:36.554 INFO:tasks.rados.rados.0.trial023.stdout:95: writing trial02342524-9 from 2305045 to 3028059 tid 2 2026-02-25T08:41:36.554 INFO:tasks.rados.rados.0.trial023.stdout:84: done (15 left) 2026-02-25T08:41:36.554 INFO:tasks.rados.rados.0.trial023.stdout:96: delete oid 11 current snap is 3 2026-02-25T08:41:36.554 INFO:tasks.rados.rados.0.trial023.stdout:90: expect (ObjNum 60 snap 2 seq_num 60) 2026-02-25T08:41:36.557 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3] 2026-02-25T08:41:36.557 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-25T08:41:36.557 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2026-02-25T08:41:36.558 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-25T08:41:36.558 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2026-02-25T08:41:36.561 INFO:tasks.ceph.osd.1:Started 2026-02-25T08:41:36.561 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:41:36.615 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 344923 2026-02-25T08:41:36.631 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:41:36.634 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:41:36.634 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:41:36.647 INFO:tasks.rados.rados.0.trial023.stdout:93: expect (ObjNum 30 snap 0 seq_num 30) 2026-02-25T08:41:36.705 INFO:tasks.rados.rados.0.trial023.stdout:95: finishing write tid 1 to trial02342524-9 2026-02-25T08:41:36.706 INFO:tasks.rados.rados.0.trial023.stdout:95: finishing write tid 2 to trial02342524-9 2026-02-25T08:41:36.706 INFO:tasks.rados.rados.0.trial023.stdout:95: finishing write tid 3 to trial02342524-9 2026-02-25T08:41:36.706 INFO:tasks.rados.rados.0.trial023.stdout:95: finishing write tid 4 to trial02342524-9 2026-02-25T08:41:36.706 INFO:tasks.rados.rados.0.trial023.stdout:95: oid 9 updating version 0 to 27 2026-02-25T08:41:36.706 INFO:tasks.rados.rados.0.trial023.stdout:95: oid 9 version 27 is already newer than 27 2026-02-25T08:41:36.706 INFO:tasks.rados.rados.0.trial023.stdout:95: oid 9 version 27 is already newer than 27 2026-02-25T08:41:36.706 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 9 v 27 (ObjNum 71 snap 3 seq_num 71) dirty exists 2026-02-25T08:41:36.706 INFO:tasks.rados.rados.0.trial023.stdout:95: left oid 9 (ObjNum 71 snap 3 seq_num 71) 2026-02-25T08:41:40.423 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2026-02-25T08:41:41.636 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:41:41.706 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:41:41.709 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:41:41.709 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:41:45.927 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~6s 2026-02-25T08:41:46.710 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:41:46.781 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:41:46.784 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:41:46.785 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:41:51.431 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~11s 2026-02-25T08:41:51.786 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:41:51.858 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:41:51.860 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:41:51.861 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:41:56.862 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:41:56.934 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:41:56.934 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~17s 2026-02-25T08:41:56.936 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:41:56.937 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:42:00.797 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:42:00.865 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:00.866 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:01.855 INFO:tasks.rados.rados.0.trial023.stdout:67: finishing write tid 1 to trial02342524-45 2026-02-25T08:42:01.855 INFO:tasks.rados.rados.0.trial023.stdout:67: finishing write tid 2 to trial02342524-45 2026-02-25T08:42:01.856 INFO:tasks.rados.rados.0.trial023.stdout:67: finishing write tid 3 to trial02342524-45 2026-02-25T08:42:01.856 INFO:tasks.rados.rados.0.trial023.stdout:67: finishing write tid 4 to trial02342524-45 2026-02-25T08:42:01.856 INFO:tasks.rados.rados.0.trial023.stdout:80: finishing write tid 1 to trial02342524-8 2026-02-25T08:42:01.857 INFO:tasks.rados.rados.0.trial023.stdout:80: finishing write tid 2 to trial02342524-8 2026-02-25T08:42:01.858 INFO:tasks.rados.rados.0.trial023.stdout:80: finishing write tid 3 to trial02342524-8 2026-02-25T08:42:01.858 INFO:tasks.rados.rados.0.trial023.stdout:80: finishing write tid 4 to trial02342524-8 2026-02-25T08:42:01.862 INFO:tasks.rados.rados.0.trial023.stdout:73: finishing write tid 1 to trial02342524-4 2026-02-25T08:42:01.862 INFO:tasks.rados.rados.0.trial023.stdout:73: finishing write tid 2 to trial02342524-4 2026-02-25T08:42:01.862 INFO:tasks.rados.rados.0.trial023.stdout:73: finishing write tid 3 to trial02342524-4 2026-02-25T08:42:01.864 INFO:tasks.rados.rados.0.trial023.stdout:80: finishing write tid 5 to trial02342524-8 2026-02-25T08:42:01.864 INFO:tasks.rados.rados.0.trial023.stdout:80: oid 8 updating version 0 to 9 2026-02-25T08:42:01.864 INFO:tasks.rados.rados.0.trial023.stdout:80: oid 8 updating version 9 to 10 2026-02-25T08:42:01.864 INFO:tasks.rados.rados.0.trial023.stdout:80: oid 8 version 10 is already newer than 8 2026-02-25T08:42:01.864 INFO:tasks.rados.rados.0.trial023.stdout:80: oid 8 updating version 10 to 11 2026-02-25T08:42:01.864 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 8 v 11 (ObjNum 63 snap 3 seq_num 63) dirty exists 2026-02-25T08:42:01.865 INFO:tasks.rados.rados.0.trial023.stdout:80: left oid 8 (ObjNum 63 snap 3 seq_num 63) 2026-02-25T08:42:01.866 INFO:tasks.rados.rados.0.trial023.stdout:89: finishing write tid 1 to trial02342524-20 2026-02-25T08:42:01.867 INFO:tasks.rados.rados.0.trial023.stdout:89: finishing write tid 2 to trial02342524-20 2026-02-25T08:42:01.867 INFO:tasks.rados.rados.0.trial023.stdout:89: finishing write tid 3 to trial02342524-20 2026-02-25T08:42:01.867 INFO:tasks.rados.rados.0.trial023.stdout:89: finishing write tid 4 to trial02342524-20 2026-02-25T08:42:01.867 INFO:tasks.rados.rados.0.trial023.stdout:80: done (15 left) 2026-02-25T08:42:01.868 INFO:tasks.rados.rados.0.trial023.stdout:90: done (14 left) 2026-02-25T08:42:01.868 INFO:tasks.rados.rados.0.trial023.stdout:91: done (13 left) 2026-02-25T08:42:01.868 INFO:tasks.rados.rados.0.trial023.stdout:93: done (12 left) 2026-02-25T08:42:01.868 INFO:tasks.rados.rados.0.trial023.stdout:95: done (11 left) 2026-02-25T08:42:01.868 INFO:tasks.rados.rados.0.trial023.stdout:96: done (10 left) 2026-02-25T08:42:01.868 INFO:tasks.rados.rados.0.trial023.stdout:97: read oid 36 snap -1 2026-02-25T08:42:01.868 INFO:tasks.rados.rados.0.trial023.stdout:97: expect deleted 2026-02-25T08:42:01.868 INFO:tasks.rados.rados.0.trial023.stdout:98: write oid 22 current snap is 3 2026-02-25T08:42:01.868 INFO:tasks.rados.rados.0.trial023.stdout:98: seq_num 73 ranges {426911=540354,1622009=726683,3121996=104915} 2026-02-25T08:42:01.871 INFO:tasks.rados.rados.0.trial023.stdout:98: writing trial02342524-22 from 426911 to 967265 tid 1 2026-02-25T08:42:01.876 INFO:tasks.rados.rados.0.trial023.stdout:98: writing trial02342524-22 from 1622009 to 2348692 tid 2 2026-02-25T08:42:01.880 INFO:tasks.rados.rados.0.trial023.stdout:98: writing trial02342524-22 from 3121996 to 3226911 tid 3 2026-02-25T08:42:01.880 INFO:tasks.rados.rados.0.trial023.stdout:99: write oid 6 current snap is 3 2026-02-25T08:42:01.880 INFO:tasks.rados.rados.0.trial023.stdout:99: seq_num 74 ranges {443718=615603,1564138=470341,2633066=610652} 2026-02-25T08:42:01.884 INFO:tasks.rados.rados.0.trial023.stdout:99: writing trial02342524-6 from 443718 to 1059321 tid 1 2026-02-25T08:42:01.887 INFO:tasks.rados.rados.0.trial023.stdout:99: writing trial02342524-6 from 1564138 to 2034479 tid 2 2026-02-25T08:42:01.893 INFO:tasks.rados.rados.0.trial023.stdout:99: writing trial02342524-6 from 2633066 to 3243718 tid 3 2026-02-25T08:42:01.893 INFO:tasks.rados.rados.0.trial023.stdout:100: write oid 12 current snap is 3 2026-02-25T08:42:01.893 INFO:tasks.rados.rados.0.trial023.stdout:73: finishing write tid 4 to trial02342524-4 2026-02-25T08:42:01.893 INFO:tasks.rados.rados.0.trial023.stdout:73: oid 4 updating version 0 to 31 2026-02-25T08:42:01.893 INFO:tasks.rados.rados.0.trial023.stdout:73: oid 4 version 31 is already newer than 29 2026-02-25T08:42:01.893 INFO:tasks.rados.rados.0.trial023.stdout:73: oid 4 version 31 is already newer than 30 2026-02-25T08:42:01.894 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 4 v 31 (ObjNum 58 snap 2 seq_num 58) dirty exists 2026-02-25T08:42:01.894 INFO:tasks.rados.rados.0.trial023.stdout:73: left oid 4 (ObjNum 58 snap 2 seq_num 58) 2026-02-25T08:42:01.894 INFO:tasks.rados.rados.0.trial023.stdout:67: finishing write tid 5 to trial02342524-45 2026-02-25T08:42:01.894 INFO:tasks.rados.rados.0.trial023.stdout:67: oid 45 updating version 0 to 25 2026-02-25T08:42:01.894 INFO:tasks.rados.rados.0.trial023.stdout:67: oid 45 updating version 25 to 26 2026-02-25T08:42:01.894 INFO:tasks.rados.rados.0.trial023.stdout:67: oid 45 updating version 26 to 27 2026-02-25T08:42:01.894 INFO:tasks.rados.rados.0.trial023.stdout:67: oid 45 version 27 is already newer than 24 2026-02-25T08:42:01.894 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 45 v 27 (ObjNum 56 snap 1 seq_num 56) dirty exists 2026-02-25T08:42:01.894 INFO:tasks.rados.rados.0.trial023.stdout:67: left oid 45 (ObjNum 56 snap 1 seq_num 56) 2026-02-25T08:42:01.894 INFO:tasks.rados.rados.0.trial023.stdout:89: finishing write tid 5 to trial02342524-20 2026-02-25T08:42:01.894 INFO:tasks.rados.rados.0.trial023.stdout:89: oid 20 updating version 0 to 33 2026-02-25T08:42:01.894 INFO:tasks.rados.rados.0.trial023.stdout:89: oid 20 updating version 33 to 34 2026-02-25T08:42:01.894 INFO:tasks.rados.rados.0.trial023.stdout:89: oid 20 version 34 is already newer than 32 2026-02-25T08:42:01.895 INFO:tasks.rados.rados.0.trial023.stdout:89: oid 20 updating version 34 to 35 2026-02-25T08:42:01.895 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 20 v 35 (ObjNum 69 snap 3 seq_num 69) dirty exists 2026-02-25T08:42:01.895 INFO:tasks.rados.rados.0.trial023.stdout:89: left oid 20 (ObjNum 69 snap 3 seq_num 69) 2026-02-25T08:42:01.895 INFO:tasks.rados.rados.0.trial023.stdout:100: seq_num 75 ranges {460525=690852,1622620=730352,3060489=200036} 2026-02-25T08:42:01.896 INFO:tasks.rados.rados.0.trial023.stdout:100: writing trial02342524-12 from 460525 to 1151377 tid 1 2026-02-25T08:42:01.901 INFO:tasks.rados.rados.0.trial023.stdout:100: writing trial02342524-12 from 1622620 to 2352972 tid 2 2026-02-25T08:42:01.905 INFO:tasks.rados.rados.0.trial023.stdout:100: writing trial02342524-12 from 3060489 to 3260525 tid 3 2026-02-25T08:42:01.905 INFO:tasks.rados.rados.0.trial023.stdout:67: done (13 left) 2026-02-25T08:42:01.905 INFO:tasks.rados.rados.0.trial023.stdout:73: done (12 left) 2026-02-25T08:42:01.905 INFO:tasks.rados.rados.0.trial023.stdout:89: done (11 left) 2026-02-25T08:42:01.906 INFO:tasks.rados.rados.0.trial023.stdout:97: done (10 left) 2026-02-25T08:42:01.906 INFO:tasks.rados.rados.0.trial023.stdout:101: write (excl) oid 43 current snap is 3 2026-02-25T08:42:01.906 INFO:tasks.rados.rados.0.trial023.stdout:87: finishing write tid 1 to trial02342524-10 2026-02-25T08:42:01.906 INFO:tasks.rados.rados.0.trial023.stdout:101: seq_num 76 ranges {477332=766101,1681102=474010,2687912=589420} 2026-02-25T08:42:01.909 INFO:tasks.rados.rados.0.trial023.stdout:101: writing trial02342524-43 from 477332 to 1243433 tid 1 2026-02-25T08:42:01.913 INFO:tasks.rados.rados.0.trial023.stdout:101: writing trial02342524-43 from 1681102 to 2155112 tid 2 2026-02-25T08:42:01.917 INFO:tasks.rados.rados.0.trial023.stdout:101: writing trial02342524-43 from 2687912 to 3277332 tid 3 2026-02-25T08:42:01.918 INFO:tasks.rados.rados.0.trial023.stdout:102: write (excl) oid 14 current snap is 3 2026-02-25T08:42:01.918 INFO:tasks.rados.rados.0.trial023.stdout:92: finishing write tid 1 to trial02342524-34 2026-02-25T08:42:01.918 INFO:tasks.rados.rados.0.trial023.stdout:102: seq_num 77 ranges {494139=557703,1455937=734021,2948041=346098} 2026-02-25T08:42:01.921 INFO:tasks.rados.rados.0.trial023.stdout:102: writing trial02342524-14 from 494139 to 1051842 tid 1 2026-02-25T08:42:01.926 INFO:tasks.rados.rados.0.trial023.stdout:102: writing trial02342524-14 from 1455937 to 2189958 tid 2 2026-02-25T08:42:01.930 INFO:tasks.rados.rados.0.trial023.stdout:102: writing trial02342524-14 from 2948041 to 3294139 tid 3 2026-02-25T08:42:01.931 INFO:tasks.rados.rados.0.trial023.stdout:103: read oid 45 snap -1 2026-02-25T08:42:01.931 INFO:tasks.rados.rados.0.trial023.stdout:103: expect (ObjNum 56 snap 1 seq_num 56) 2026-02-25T08:42:01.931 INFO:tasks.rados.rados.0.trial023.stdout:104: write (excl) oid 24 current snap is 3 2026-02-25T08:42:01.931 INFO:tasks.rados.rados.0.trial023.stdout:92: finishing write tid 2 to trial02342524-34 2026-02-25T08:42:01.931 INFO:tasks.rados.rados.0.trial023.stdout:104: seq_num 78 ranges {510946=632952,1798066=477679,2742758=457217} 2026-02-25T08:42:01.934 INFO:tasks.rados.rados.0.trial023.stdout:104: writing trial02342524-24 from 510946 to 1143898 tid 1 2026-02-25T08:42:01.938 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:42:01.939 INFO:tasks.rados.rados.0.trial023.stdout:104: writing trial02342524-24 from 1798066 to 2275745 tid 2 2026-02-25T08:42:01.943 INFO:tasks.rados.rados.0.trial023.stdout:104: writing trial02342524-24 from 2742758 to 3199975 tid 3 2026-02-25T08:42:01.943 INFO:tasks.rados.rados.0.trial023.stdout:66: expect (ObjNum 51 snap 0 seq_num 51) 2026-02-25T08:42:01.943 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:42:02.013 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:42:02.016 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:42:02.016 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:42:02.028 INFO:tasks.rados.rados.0.trial023.stdout:77: finishing write tid 1 to trial02342524-18 2026-02-25T08:42:02.028 INFO:tasks.rados.rados.0.trial023.stdout:87: finishing write tid 2 to trial02342524-10 2026-02-25T08:42:02.028 INFO:tasks.rados.rados.0.trial023.stdout:66: done (13 left) 2026-02-25T08:42:02.028 INFO:tasks.rados.rados.0.trial023.stdout:105: read oid 44 snap -1 2026-02-25T08:42:02.028 INFO:tasks.rados.rados.0.trial023.stdout:105: expect (ObjNum 43 snap 0 seq_num 43) 2026-02-25T08:42:02.028 INFO:tasks.rados.rados.0.trial023.stdout:106: write oid 26 current snap is 3 2026-02-25T08:42:02.028 INFO:tasks.rados.rados.0.trial023.stdout:92: finishing write tid 3 to trial02342524-34 2026-02-25T08:42:02.028 INFO:tasks.rados.rados.0.trial023.stdout:92: finishing write tid 4 to trial02342524-34 2026-02-25T08:42:02.028 INFO:tasks.rados.rados.0.trial023.stdout:92: oid 34 updating version 0 to 9 2026-02-25T08:42:02.029 INFO:tasks.rados.rados.0.trial023.stdout:92: oid 34 version 9 is already newer than 9 2026-02-25T08:42:02.029 INFO:tasks.rados.rados.0.trial023.stdout:92: oid 34 version 9 is already newer than 9 2026-02-25T08:42:02.029 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 34 v 9 (ObjNum 70 snap 3 seq_num 70) dirty exists 2026-02-25T08:42:02.029 INFO:tasks.rados.rados.0.trial023.stdout:92: left oid 34 (ObjNum 70 snap 3 seq_num 70) 2026-02-25T08:42:02.029 INFO:tasks.rados.rados.0.trial023.stdout:98: finishing write tid 1 to trial02342524-22 2026-02-25T08:42:02.029 INFO:tasks.rados.rados.0.trial023.stdout:98: finishing write tid 2 to trial02342524-22 2026-02-25T08:42:02.029 INFO:tasks.rados.rados.0.trial023.stdout:106: seq_num 79 ranges {527753=708201,1856548=737690,3286534=41219} 2026-02-25T08:42:02.032 INFO:tasks.rados.rados.0.trial023.stdout:106: writing trial02342524-26 from 527753 to 1235954 tid 1 2026-02-25T08:42:02.037 INFO:tasks.rados.rados.0.trial023.stdout:106: writing trial02342524-26 from 1856548 to 2594238 tid 2 2026-02-25T08:42:02.042 INFO:tasks.rados.rados.0.trial023.stdout:106: writing trial02342524-26 from 3286534 to 3327753 tid 3 2026-02-25T08:42:02.042 INFO:tasks.rados.rados.0.trial023.stdout:92: done (14 left) 2026-02-25T08:42:02.042 INFO:tasks.rados.rados.0.trial023.stdout:107: read oid 15 snap -1 2026-02-25T08:42:02.042 INFO:tasks.rados.rados.0.trial023.stdout:107: expect (ObjNum 62 snap 3 seq_num 62) 2026-02-25T08:42:02.042 INFO:tasks.rados.rados.0.trial023.stdout:108: snap_create 2026-02-25T08:42:02.042 INFO:tasks.rados.rados.0.trial023.stdout:77: finishing write tid 2 to trial02342524-18 2026-02-25T08:42:02.042 INFO:tasks.rados.rados.0.trial023.stdout:98: finishing write tid 3 to trial02342524-22 2026-02-25T08:42:02.042 INFO:tasks.rados.rados.0.trial023.stdout:77: finishing write tid 3 to trial02342524-18 2026-02-25T08:42:02.042 INFO:tasks.rados.rados.0.trial023.stdout:98: finishing write tid 4 to trial02342524-22 2026-02-25T08:42:02.042 INFO:tasks.rados.rados.0.trial023.stdout:98: finishing write tid 5 to trial02342524-22 2026-02-25T08:42:02.042 INFO:tasks.rados.rados.0.trial023.stdout:98: oid 22 updating version 0 to 13 2026-02-25T08:42:02.043 INFO:tasks.rados.rados.0.trial023.stdout:98: oid 22 version 13 is already newer than 13 2026-02-25T08:42:02.043 INFO:tasks.rados.rados.0.trial023.stdout:98: oid 22 version 13 is already newer than 13 2026-02-25T08:42:02.043 INFO:tasks.rados.rados.0.trial023.stdout:98: oid 22 version 13 is already newer than 13 2026-02-25T08:42:02.043 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 22 v 13 (ObjNum 73 snap 3 seq_num 73) dirty exists 2026-02-25T08:42:02.043 INFO:tasks.rados.rados.0.trial023.stdout:98: left oid 22 (ObjNum 73 snap 3 seq_num 73) 2026-02-25T08:42:02.043 INFO:tasks.rados.rados.0.trial023.stdout:77: finishing write tid 4 to trial02342524-18 2026-02-25T08:42:02.043 INFO:tasks.rados.rados.0.trial023.stdout:77: oid 18 updating version 0 to 25 2026-02-25T08:42:02.043 INFO:tasks.rados.rados.0.trial023.stdout:77: oid 18 version 25 is already newer than 25 2026-02-25T08:42:02.043 INFO:tasks.rados.rados.0.trial023.stdout:77: oid 18 version 25 is already newer than 25 2026-02-25T08:42:02.043 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 18 v 25 (ObjNum 61 snap 2 seq_num 61) dirty exists 2026-02-25T08:42:02.043 INFO:tasks.rados.rados.0.trial023.stdout:77: left oid 18 (ObjNum 61 snap 2 seq_num 61) 2026-02-25T08:42:02.043 INFO:tasks.rados.rados.0.trial023.stdout:87: finishing write tid 3 to trial02342524-10 2026-02-25T08:42:02.044 INFO:tasks.rados.rados.0.trial023.stdout:74: finishing write tid 1 to trial02342524-33 2026-02-25T08:42:02.044 INFO:tasks.rados.rados.0.trial023.stdout:99: finishing write tid 1 to trial02342524-6 2026-02-25T08:42:02.044 INFO:tasks.rados.rados.0.trial023.stdout:87: finishing write tid 4 to trial02342524-10 2026-02-25T08:42:02.044 INFO:tasks.rados.rados.0.trial023.stdout:87: finishing write tid 5 to trial02342524-10 2026-02-25T08:42:02.044 INFO:tasks.rados.rados.0.trial023.stdout:87: oid 10 updating version 0 to 27 2026-02-25T08:42:02.044 INFO:tasks.rados.rados.0.trial023.stdout:87: oid 10 version 27 is already newer than 27 2026-02-25T08:42:02.044 INFO:tasks.rados.rados.0.trial023.stdout:87: oid 10 version 27 is already newer than 27 2026-02-25T08:42:02.044 INFO:tasks.rados.rados.0.trial023.stdout:87: oid 10 version 27 is already newer than 27 2026-02-25T08:42:02.044 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 10 v 27 (ObjNum 68 snap 3 seq_num 68) dirty exists 2026-02-25T08:42:02.044 INFO:tasks.rados.rados.0.trial023.stdout:87: left oid 10 (ObjNum 68 snap 3 seq_num 68) 2026-02-25T08:42:02.044 INFO:tasks.rados.rados.0.trial023.stdout:101: finishing write tid 1 to trial02342524-43 2026-02-25T08:42:02.044 INFO:tasks.rados.rados.0.trial023.stdout:83: finishing write tid 1 to trial02342524-38 2026-02-25T08:42:02.044 INFO:tasks.rados.rados.0.trial023.stdout:83: finishing write tid 2 to trial02342524-38 2026-02-25T08:42:02.044 INFO:tasks.rados.rados.0.trial023.stdout:101: finishing write tid 2 to trial02342524-43 2026-02-25T08:42:02.045 INFO:tasks.rados.rados.0.trial023.stdout:99: finishing write tid 2 to trial02342524-6 2026-02-25T08:42:02.045 INFO:tasks.rados.rados.0.trial023.stdout:101: finishing write tid 3 to trial02342524-43 2026-02-25T08:42:02.045 INFO:tasks.rados.rados.0.trial023.stdout:83: finishing write tid 3 to trial02342524-38 2026-02-25T08:42:02.045 INFO:tasks.rados.rados.0.trial023.stdout:101: finishing write tid 4 to trial02342524-43 2026-02-25T08:42:02.045 INFO:tasks.rados.rados.0.trial023.stdout:101: finishing write tid 5 to trial02342524-43 2026-02-25T08:42:02.045 INFO:tasks.rados.rados.0.trial023.stdout:101: oid 43 updating version 0 to 7 2026-02-25T08:42:02.045 INFO:tasks.rados.rados.0.trial023.stdout:101: oid 43 version 7 is already newer than 7 2026-02-25T08:42:02.045 INFO:tasks.rados.rados.0.trial023.stdout:101: oid 43 version 7 is already newer than 7 2026-02-25T08:42:02.045 INFO:tasks.rados.rados.0.trial023.stdout:101: oid 43 version 7 is already newer than 4 2026-02-25T08:42:02.045 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 43 v 7 (ObjNum 76 snap 3 seq_num 76) dirty exists 2026-02-25T08:42:02.045 INFO:tasks.rados.rados.0.trial023.stdout:101: left oid 43 (ObjNum 76 snap 3 seq_num 76) 2026-02-25T08:42:02.045 INFO:tasks.rados.rados.0.trial023.stdout:103: expect (ObjNum 56 snap 1 seq_num 56) 2026-02-25T08:42:02.109 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:42:02.144 INFO:tasks.rados.rados.0.trial023.stdout:83: finishing write tid 4 to trial02342524-38 2026-02-25T08:42:02.144 INFO:tasks.rados.rados.0.trial023.stdout:83: finishing write tid 5 to trial02342524-38 2026-02-25T08:42:02.144 INFO:tasks.rados.rados.0.trial023.stdout:83: oid 38 updating version 0 to 30 2026-02-25T08:42:02.144 INFO:tasks.rados.rados.0.trial023.stdout:83: oid 38 version 30 is already newer than 30 2026-02-25T08:42:02.144 INFO:tasks.rados.rados.0.trial023.stdout:83: oid 38 version 30 is already newer than 27 2026-02-25T08:42:02.144 INFO:tasks.rados.rados.0.trial023.stdout:83: oid 38 version 30 is already newer than 30 2026-02-25T08:42:02.144 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 38 v 30 (ObjNum 66 snap 3 seq_num 66) dirty exists 2026-02-25T08:42:02.144 INFO:tasks.rados.rados.0.trial023.stdout:83: left oid 38 (ObjNum 66 snap 3 seq_num 66) 2026-02-25T08:42:02.144 INFO:tasks.rados.rados.0.trial023.stdout:94: expect (ObjNum 18 snap 0 seq_num 18) 2026-02-25T08:42:02.144 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:42:02.200 INFO:tasks.rados.rados.0.trial023.stdout:104: finishing write tid 1 to trial02342524-24 2026-02-25T08:42:02.200 INFO:tasks.rados.rados.0.trial023.stdout:74: finishing write tid 2 to trial02342524-33 2026-02-25T08:42:02.200 INFO:tasks.rados.rados.0.trial023.stdout:74: finishing write tid 3 to trial02342524-33 2026-02-25T08:42:02.200 INFO:tasks.rados.rados.0.trial023.stdout:74: finishing write tid 4 to trial02342524-33 2026-02-25T08:42:02.200 INFO:tasks.rados.rados.0.trial023.stdout:74: finishing write tid 5 to trial02342524-33 2026-02-25T08:42:02.200 INFO:tasks.rados.rados.0.trial023.stdout:74: oid 33 updating version 0 to 8 2026-02-25T08:42:02.200 INFO:tasks.rados.rados.0.trial023.stdout:74: oid 33 version 8 is already newer than 5 2026-02-25T08:42:02.200 INFO:tasks.rados.rados.0.trial023.stdout:74: oid 33 version 8 is already newer than 8 2026-02-25T08:42:02.200 INFO:tasks.rados.rados.0.trial023.stdout:74: oid 33 version 8 is already newer than 8 2026-02-25T08:42:02.200 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 33 v 8 (ObjNum 59 snap 2 seq_num 59) dirty exists 2026-02-25T08:42:02.201 INFO:tasks.rados.rados.0.trial023.stdout:74: left oid 33 (ObjNum 59 snap 2 seq_num 59) 2026-02-25T08:42:02.201 INFO:tasks.rados.rados.0.trial023.stdout:102: finishing write tid 1 to trial02342524-14 2026-02-25T08:42:02.201 INFO:tasks.rados.rados.0.trial023.stdout:106: finishing write tid 1 to trial02342524-26 2026-02-25T08:42:02.201 INFO:tasks.rados.rados.0.trial023.stdout:105: expect (ObjNum 43 snap 0 seq_num 43) 2026-02-25T08:42:02.239 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:42:02.262 INFO:tasks.rados.rados.0.trial023.stdout:106: finishing write tid 2 to trial02342524-26 2026-02-25T08:42:02.262 INFO:tasks.rados.rados.0.trial023.stdout:99: finishing write tid 3 to trial02342524-6 2026-02-25T08:42:02.262 INFO:tasks.rados.rados.0.trial023.stdout:106: finishing write tid 3 to trial02342524-26 2026-02-25T08:42:02.262 INFO:tasks.rados.rados.0.trial023.stdout:107: expect (ObjNum 62 snap 3 seq_num 62) 2026-02-25T08:42:02.359 INFO:tasks.rados.rados.0.trial023.stdout:106: finishing write tid 4 to trial02342524-26 2026-02-25T08:42:02.359 INFO:tasks.rados.rados.0.trial023.stdout:106: finishing write tid 5 to trial02342524-26 2026-02-25T08:42:02.359 INFO:tasks.rados.rados.0.trial023.stdout:106: oid 26 updating version 0 to 31 2026-02-25T08:42:02.359 INFO:tasks.rados.rados.0.trial023.stdout:106: oid 26 version 31 is already newer than 31 2026-02-25T08:42:02.359 INFO:tasks.rados.rados.0.trial023.stdout:106: oid 26 version 31 is already newer than 31 2026-02-25T08:42:02.360 INFO:tasks.rados.rados.0.trial023.stdout:106: oid 26 version 31 is already newer than 31 2026-02-25T08:42:02.360 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 26 v 31 (ObjNum 79 snap 3 seq_num 79) dirty exists 2026-02-25T08:42:02.360 INFO:tasks.rados.rados.0.trial023.stdout:106: left oid 26 (ObjNum 79 snap 3 seq_num 79) 2026-02-25T08:42:02.360 INFO:tasks.rados.rados.0.trial023.stdout:99: finishing write tid 4 to trial02342524-6 2026-02-25T08:42:02.360 INFO:tasks.rados.rados.0.trial023.stdout:99: finishing write tid 5 to trial02342524-6 2026-02-25T08:42:02.360 INFO:tasks.rados.rados.0.trial023.stdout:99: oid 6 updating version 0 to 13 2026-02-25T08:42:02.360 INFO:tasks.rados.rados.0.trial023.stdout:99: oid 6 version 13 is already newer than 13 2026-02-25T08:42:02.360 INFO:tasks.rados.rados.0.trial023.stdout:99: oid 6 version 13 is already newer than 13 2026-02-25T08:42:02.360 INFO:tasks.rados.rados.0.trial023.stdout:99: oid 6 version 13 is already newer than 13 2026-02-25T08:42:02.360 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 6 v 13 (ObjNum 74 snap 3 seq_num 74) dirty exists 2026-02-25T08:42:02.360 INFO:tasks.rados.rados.0.trial023.stdout:99: left oid 6 (ObjNum 74 snap 3 seq_num 74) 2026-02-25T08:42:02.360 INFO:tasks.rados.rados.0.trial023.stdout:102: finishing write tid 2 to trial02342524-14 2026-02-25T08:42:02.361 INFO:tasks.rados.rados.0.trial023.stdout:104: finishing write tid 2 to trial02342524-24 2026-02-25T08:42:02.361 INFO:tasks.rados.rados.0.trial023.stdout:102: finishing write tid 3 to trial02342524-14 2026-02-25T08:42:02.361 INFO:tasks.rados.rados.0.trial023.stdout:102: finishing write tid 4 to trial02342524-14 2026-02-25T08:42:02.361 INFO:tasks.rados.rados.0.trial023.stdout:102: finishing write tid 5 to trial02342524-14 2026-02-25T08:42:02.361 INFO:tasks.rados.rados.0.trial023.stdout:102: oid 14 updating version 0 to 10 2026-02-25T08:42:02.361 INFO:tasks.rados.rados.0.trial023.stdout:102: oid 14 updating version 10 to 13 2026-02-25T08:42:02.361 INFO:tasks.rados.rados.0.trial023.stdout:102: oid 14 version 13 is already newer than 13 2026-02-25T08:42:02.361 INFO:tasks.rados.rados.0.trial023.stdout:102: oid 14 version 13 is already newer than 13 2026-02-25T08:42:02.361 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 14 v 13 (ObjNum 77 snap 3 seq_num 77) dirty exists 2026-02-25T08:42:02.361 INFO:tasks.rados.rados.0.trial023.stdout:102: left oid 14 (ObjNum 77 snap 3 seq_num 77) 2026-02-25T08:42:02.361 INFO:tasks.rados.rados.0.trial023.stdout:104: finishing write tid 3 to trial02342524-24 2026-02-25T08:42:02.361 INFO:tasks.rados.rados.0.trial023.stdout:104: finishing write tid 4 to trial02342524-24 2026-02-25T08:42:02.362 INFO:tasks.rados.rados.0.trial023.stdout:104: finishing write tid 5 to trial02342524-24 2026-02-25T08:42:02.362 INFO:tasks.rados.rados.0.trial023.stdout:104: oid 24 updating version 0 to 23 2026-02-25T08:42:02.362 INFO:tasks.rados.rados.0.trial023.stdout:104: oid 24 version 23 is already newer than 20 2026-02-25T08:42:02.362 INFO:tasks.rados.rados.0.trial023.stdout:104: oid 24 version 23 is already newer than 23 2026-02-25T08:42:02.362 INFO:tasks.rados.rados.0.trial023.stdout:104: oid 24 version 23 is already newer than 23 2026-02-25T08:42:02.362 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 24 v 23 (ObjNum 78 snap 3 seq_num 78) dirty exists 2026-02-25T08:42:02.362 INFO:tasks.rados.rados.0.trial023.stdout:104: left oid 24 (ObjNum 78 snap 3 seq_num 78) 2026-02-25T08:42:02.362 INFO:tasks.rados.rados.0.trial023.stdout:100: finishing write tid 1 to trial02342524-12 2026-02-25T08:42:02.362 INFO:tasks.rados.rados.0.trial023.stdout:100: finishing write tid 2 to trial02342524-12 2026-02-25T08:42:02.362 INFO:tasks.rados.rados.0.trial023.stdout:100: finishing write tid 3 to trial02342524-12 2026-02-25T08:42:02.362 INFO:tasks.rados.rados.0.trial023.stdout:100: finishing write tid 4 to trial02342524-12 2026-02-25T08:42:02.362 INFO:tasks.rados.rados.0.trial023.stdout:100: finishing write tid 5 to trial02342524-12 2026-02-25T08:42:02.362 INFO:tasks.rados.rados.0.trial023.stdout:100: oid 12 updating version 0 to 20 2026-02-25T08:42:02.362 INFO:tasks.rados.rados.0.trial023.stdout:100: oid 12 version 20 is already newer than 20 2026-02-25T08:42:02.363 INFO:tasks.rados.rados.0.trial023.stdout:100: oid 12 version 20 is already newer than 20 2026-02-25T08:42:02.363 INFO:tasks.rados.rados.0.trial023.stdout:100: oid 12 version 20 is already newer than 20 2026-02-25T08:42:02.363 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 12 v 20 (ObjNum 75 snap 3 seq_num 75) dirty exists 2026-02-25T08:42:02.363 INFO:tasks.rados.rados.0.trial023.stdout:100: left oid 12 (ObjNum 75 snap 3 seq_num 75) 2026-02-25T08:42:02.437 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~22s 2026-02-25T08:42:02.805 INFO:tasks.rados.rados.0.trial023.stdout:74: done (15 left) 2026-02-25T08:42:02.805 INFO:tasks.rados.rados.0.trial023.stdout:77: done (14 left) 2026-02-25T08:42:02.805 INFO:tasks.rados.rados.0.trial023.stdout:83: done (13 left) 2026-02-25T08:42:02.805 INFO:tasks.rados.rados.0.trial023.stdout:87: done (12 left) 2026-02-25T08:42:02.805 INFO:tasks.rados.rados.0.trial023.stdout:94: done (11 left) 2026-02-25T08:42:02.805 INFO:tasks.rados.rados.0.trial023.stdout:98: done (10 left) 2026-02-25T08:42:02.805 INFO:tasks.rados.rados.0.trial023.stdout:99: done (9 left) 2026-02-25T08:42:02.805 INFO:tasks.rados.rados.0.trial023.stdout:100: done (8 left) 2026-02-25T08:42:02.805 INFO:tasks.rados.rados.0.trial023.stdout:101: done (7 left) 2026-02-25T08:42:02.805 INFO:tasks.rados.rados.0.trial023.stdout:102: done (6 left) 2026-02-25T08:42:02.806 INFO:tasks.rados.rados.0.trial023.stdout:103: done (5 left) 2026-02-25T08:42:02.806 INFO:tasks.rados.rados.0.trial023.stdout:104: done (4 left) 2026-02-25T08:42:02.806 INFO:tasks.rados.rados.0.trial023.stdout:105: done (3 left) 2026-02-25T08:42:02.806 INFO:tasks.rados.rados.0.trial023.stdout:106: done (2 left) 2026-02-25T08:42:02.806 INFO:tasks.rados.rados.0.trial023.stdout:107: done (1 left) 2026-02-25T08:42:02.806 INFO:tasks.rados.rados.0.trial023.stdout:108: done (0 left) 2026-02-25T08:42:02.806 INFO:tasks.rados.rados.0.trial023.stdout:109: read oid 40 snap 3 2026-02-25T08:42:02.806 INFO:tasks.rados.rados.0.trial023.stdout:109: expect deleted 2026-02-25T08:42:02.806 INFO:tasks.rados.rados.0.trial023.stdout:110: snap_create 2026-02-25T08:42:02.867 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:42:02.943 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:02.939+0000 7fc254ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:02.945 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:02.943+0000 7fc254ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:02.960 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:42:02.987 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:02.987 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:03.805 INFO:tasks.rados.rados.0.trial023.stdout:109: done (1 left) 2026-02-25T08:42:03.806 INFO:tasks.rados.rados.0.trial023.stdout:110: done (0 left) 2026-02-25T08:42:03.806 INFO:tasks.rados.rados.0.trial023.stdout:111: snap_remove snap 3 2026-02-25T08:42:04.811 INFO:tasks.rados.rados.0.trial023.stdout:111: done (0 left) 2026-02-25T08:42:04.812 INFO:tasks.rados.rados.0.trial023.stdout:112: read oid 18 snap -1 2026-02-25T08:42:04.812 INFO:tasks.rados.rados.0.trial023.stdout:112: expect (ObjNum 61 snap 2 seq_num 61) 2026-02-25T08:42:04.812 INFO:tasks.rados.rados.0.trial023.stdout:113: delete oid 25 current snap is 5 2026-02-25T08:42:04.834 INFO:tasks.rados.rados.0.trial023.stdout:113: done (1 left) 2026-02-25T08:42:04.834 INFO:tasks.rados.rados.0.trial023.stdout:114: snap_remove snap 4 2026-02-25T08:42:04.988 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:42:05.065 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:05.062+0000 7f308f4cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:05.068 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:05.066+0000 7f308e242640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:05.083 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:42:05.111 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:05.111 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:05.813 INFO:tasks.rados.rados.0.trial023.stdout:114: done (1 left) 2026-02-25T08:42:05.813 INFO:tasks.rados.rados.0.trial023.stdout:115: snap_create 2026-02-25T08:42:05.813 INFO:tasks.rados.rados.0.trial023.stdout:112: expect (ObjNum 61 snap 2 seq_num 61) 2026-02-25T08:42:06.818 INFO:tasks.rados.rados.0.trial023.stdout:112: done (1 left) 2026-02-25T08:42:06.818 INFO:tasks.rados.rados.0.trial023.stdout:115: done (0 left) 2026-02-25T08:42:06.819 INFO:tasks.rados.rados.0.trial023.stdout:116: snap_create 2026-02-25T08:42:07.018 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:42:07.088 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:42:07.090 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:42:07.091 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:42:07.112 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:42:07.188 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:07.185+0000 7f3d5d8cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:07.190 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:07.188+0000 7f3d5d8cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:07.205 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:42:07.232 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:07.233 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:07.822 INFO:tasks.rados.rados.0.trial023.stdout:116: done (0 left) 2026-02-25T08:42:07.822 INFO:tasks.rados.rados.0.trial023.stdout:117: snap_remove snap 5 2026-02-25T08:42:07.940 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~28s 2026-02-25T08:42:08.826 INFO:tasks.rados.rados.0.trial023.stdout:117: done (0 left) 2026-02-25T08:42:08.826 INFO:tasks.rados.rados.0.trial023.stdout:118: read oid 14 snap 6 2026-02-25T08:42:08.827 INFO:tasks.rados.rados.0.trial023.stdout:118: expect (ObjNum 77 snap 3 seq_num 77) 2026-02-25T08:42:08.827 INFO:tasks.rados.rados.0.trial023.stdout:119: read oid 44 snap -1 2026-02-25T08:42:08.827 INFO:tasks.rados.rados.0.trial023.stdout:119: expect (ObjNum 43 snap 0 seq_num 43) 2026-02-25T08:42:08.827 INFO:tasks.rados.rados.0.trial023.stdout:120: write oid 46 current snap is 7 2026-02-25T08:42:08.827 INFO:tasks.rados.rados.0.trial023.stdout:120: seq_num 81 ranges {561367=458699,1573512=741359,2941380=419987} 2026-02-25T08:42:08.829 INFO:tasks.rados.rados.0.trial023.stdout:120: writing trial02342524-46 from 561367 to 1020066 tid 1 2026-02-25T08:42:08.835 INFO:tasks.rados.rados.0.trial023.stdout:120: writing trial02342524-46 from 1573512 to 2314871 tid 2 2026-02-25T08:42:08.839 INFO:tasks.rados.rados.0.trial023.stdout:120: writing trial02342524-46 from 2941380 to 3361367 tid 3 2026-02-25T08:42:08.839 INFO:tasks.rados.rados.0.trial023.stdout:121: write oid 45 current snap is 7 2026-02-25T08:42:08.839 INFO:tasks.rados.rados.0.trial023.stdout:121: seq_num 82 ranges {578174=533948,1515641=485017,2736097=502305} 2026-02-25T08:42:08.843 INFO:tasks.rados.rados.0.trial023.stdout:121: writing trial02342524-45 from 578174 to 1112122 tid 1 2026-02-25T08:42:08.847 INFO:tasks.rados.rados.0.trial023.stdout:121: writing trial02342524-45 from 1515641 to 2000658 tid 2 2026-02-25T08:42:08.851 INFO:tasks.rados.rados.0.trial023.stdout:121: writing trial02342524-45 from 2736097 to 3238402 tid 3 2026-02-25T08:42:08.851 INFO:tasks.rados.rados.0.trial023.stdout:122: write oid 18 current snap is 7 2026-02-25T08:42:08.851 INFO:tasks.rados.rados.0.trial023.stdout:122: seq_num 83 ranges {594981=609197,1974123=745028,3279873=115108} 2026-02-25T08:42:08.856 INFO:tasks.rados.rados.0.trial023.stdout:122: writing trial02342524-18 from 594981 to 1204178 tid 1 2026-02-25T08:42:08.861 INFO:tasks.rados.rados.0.trial023.stdout:122: writing trial02342524-18 from 1974123 to 2719151 tid 2 2026-02-25T08:42:08.864 INFO:tasks.rados.rados.0.trial023.stdout:122: writing trial02342524-18 from 3279873 to 3394981 tid 3 2026-02-25T08:42:08.864 INFO:tasks.rados.rados.0.trial023.stdout:123: read oid 49 snap -1 2026-02-25T08:42:08.864 INFO:tasks.rados.rados.0.trial023.stdout:123: expect (ObjNum 48 snap 0 seq_num 48) 2026-02-25T08:42:08.864 INFO:tasks.rados.rados.0.trial023.stdout:124: write oid 22 current snap is 7 2026-02-25T08:42:08.864 INFO:tasks.rados.rados.0.trial023.stdout:124: seq_num 84 ranges {611788=400799,1748958=488686,2907296=504492} 2026-02-25T08:42:08.868 INFO:tasks.rados.rados.0.trial023.stdout:124: writing trial02342524-22 from 611788 to 1012587 tid 1 2026-02-25T08:42:08.872 INFO:tasks.rados.rados.0.trial023.stdout:124: writing trial02342524-22 from 1748958 to 2237644 tid 2 2026-02-25T08:42:08.877 INFO:tasks.rados.rados.0.trial023.stdout:124: writing trial02342524-22 from 2907296 to 3411788 tid 3 2026-02-25T08:42:08.877 INFO:tasks.rados.rados.0.trial023.stdout:125: read oid 6 snap -1 2026-02-25T08:42:08.877 INFO:tasks.rados.rados.0.trial023.stdout:125: expect (ObjNum 74 snap 3 seq_num 74) 2026-02-25T08:42:08.877 INFO:tasks.rados.rados.0.trial023.stdout:126: read oid 35 snap -1 2026-02-25T08:42:08.877 INFO:tasks.rados.rados.0.trial023.stdout:126: expect (ObjNum 67 snap 3 seq_num 67) 2026-02-25T08:42:08.877 INFO:tasks.rados.rados.0.trial023.stdout:127: delete oid 9 current snap is 7 2026-02-25T08:42:08.877 INFO:tasks.rados.rados.0.trial023.stdout:119: expect (ObjNum 43 snap 0 seq_num 43) 2026-02-25T08:42:08.925 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:42:08.949 INFO:tasks.rados.rados.0.trial023.stdout:120: finishing write tid 1 to trial02342524-46 2026-02-25T08:42:08.949 INFO:tasks.rados.rados.0.trial023.stdout:125: expect (ObjNum 74 snap 3 seq_num 74) 2026-02-25T08:42:09.051 INFO:tasks.rados.rados.0.trial023.stdout:120: finishing write tid 2 to trial02342524-46 2026-02-25T08:42:09.052 INFO:tasks.rados.rados.0.trial023.stdout:123: expect (ObjNum 48 snap 0 seq_num 48) 2026-02-25T08:42:09.059 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2115054 2026-02-25T08:42:09.065 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:42:09.090 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1465524 2026-02-25T08:42:09.105 INFO:tasks.rados.rados.0.trial023.stdout:118: expect (ObjNum 77 snap 3 seq_num 77) 2026-02-25T08:42:09.203 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1012692 2026-02-25T08:42:09.233 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:42:09.240 INFO:tasks.rados.rados.0.trial023.stdout:120: finishing write tid 3 to trial02342524-46 2026-02-25T08:42:09.240 INFO:tasks.rados.rados.0.trial023.stdout:120: finishing write tid 4 to trial02342524-46 2026-02-25T08:42:09.240 INFO:tasks.rados.rados.0.trial023.stdout:124: finishing write tid 1 to trial02342524-22 2026-02-25T08:42:09.240 INFO:tasks.rados.rados.0.trial023.stdout:120: finishing write tid 5 to trial02342524-46 2026-02-25T08:42:09.240 INFO:tasks.rados.rados.0.trial023.stdout:120: oid 46 updating version 0 to 32 2026-02-25T08:42:09.240 INFO:tasks.rados.rados.0.trial023.stdout:120: oid 46 version 32 is already newer than 32 2026-02-25T08:42:09.240 INFO:tasks.rados.rados.0.trial023.stdout:120: oid 46 version 32 is already newer than 32 2026-02-25T08:42:09.241 INFO:tasks.rados.rados.0.trial023.stdout:120: oid 46 version 32 is already newer than 32 2026-02-25T08:42:09.241 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 46 v 32 (ObjNum 81 snap 7 seq_num 81) dirty exists 2026-02-25T08:42:09.241 INFO:tasks.rados.rados.0.trial023.stdout:120: left oid 46 (ObjNum 81 snap 7 seq_num 81) 2026-02-25T08:42:09.241 INFO:tasks.rados.rados.0.trial023.stdout:124: finishing write tid 2 to trial02342524-22 2026-02-25T08:42:09.241 INFO:tasks.rados.rados.0.trial023.stdout:118: done (9 left) 2026-02-25T08:42:09.241 INFO:tasks.rados.rados.0.trial023.stdout:119: done (8 left) 2026-02-25T08:42:09.241 INFO:tasks.rados.rados.0.trial023.stdout:120: done (7 left) 2026-02-25T08:42:09.241 INFO:tasks.rados.rados.0.trial023.stdout:123: done (6 left) 2026-02-25T08:42:09.241 INFO:tasks.rados.rados.0.trial023.stdout:125: done (5 left) 2026-02-25T08:42:09.241 INFO:tasks.rados.rados.0.trial023.stdout:127: done (4 left) 2026-02-25T08:42:09.241 INFO:tasks.rados.rados.0.trial023.stdout:128: snap_create 2026-02-25T08:42:09.241 INFO:tasks.rados.rados.0.trial023.stdout:126: expect (ObjNum 67 snap 3 seq_num 67) 2026-02-25T08:42:09.307 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:09.305+0000 7fdde270b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:09.310 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:09.308+0000 7fdde270b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:09.325 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:42:09.336 INFO:tasks.rados.rados.0.trial023.stdout:121: finishing write tid 1 to trial02342524-45 2026-02-25T08:42:09.336 INFO:tasks.rados.rados.0.trial023.stdout:122: finishing write tid 1 to trial02342524-18 2026-02-25T08:42:09.336 INFO:tasks.rados.rados.0.trial023.stdout:124: finishing write tid 3 to trial02342524-22 2026-02-25T08:42:09.336 INFO:tasks.rados.rados.0.trial023.stdout:121: finishing write tid 2 to trial02342524-45 2026-02-25T08:42:09.336 INFO:tasks.rados.rados.0.trial023.stdout:124: finishing write tid 4 to trial02342524-22 2026-02-25T08:42:09.336 INFO:tasks.rados.rados.0.trial023.stdout:124: finishing write tid 5 to trial02342524-22 2026-02-25T08:42:09.336 INFO:tasks.rados.rados.0.trial023.stdout:124: oid 22 updating version 0 to 18 2026-02-25T08:42:09.336 INFO:tasks.rados.rados.0.trial023.stdout:124: oid 22 version 18 is already newer than 18 2026-02-25T08:42:09.336 INFO:tasks.rados.rados.0.trial023.stdout:124: oid 22 version 18 is already newer than 18 2026-02-25T08:42:09.336 INFO:tasks.rados.rados.0.trial023.stdout:124: oid 22 version 18 is already newer than 18 2026-02-25T08:42:09.337 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 22 v 18 (ObjNum 84 snap 7 seq_num 84) dirty exists 2026-02-25T08:42:09.337 INFO:tasks.rados.rados.0.trial023.stdout:124: left oid 22 (ObjNum 84 snap 7 seq_num 84) 2026-02-25T08:42:09.337 INFO:tasks.rados.rados.0.trial023.stdout:122: finishing write tid 2 to trial02342524-18 2026-02-25T08:42:09.337 INFO:tasks.rados.rados.0.trial023.stdout:121: finishing write tid 3 to trial02342524-45 2026-02-25T08:42:09.337 INFO:tasks.rados.rados.0.trial023.stdout:122: finishing write tid 3 to trial02342524-18 2026-02-25T08:42:09.337 INFO:tasks.rados.rados.0.trial023.stdout:121: finishing write tid 4 to trial02342524-45 2026-02-25T08:42:09.337 INFO:tasks.rados.rados.0.trial023.stdout:122: finishing write tid 4 to trial02342524-18 2026-02-25T08:42:09.337 INFO:tasks.rados.rados.0.trial023.stdout:121: finishing write tid 5 to trial02342524-45 2026-02-25T08:42:09.337 INFO:tasks.rados.rados.0.trial023.stdout:121: oid 45 updating version 0 to 42 2026-02-25T08:42:09.337 INFO:tasks.rados.rados.0.trial023.stdout:121: oid 45 version 42 is already newer than 42 2026-02-25T08:42:09.337 INFO:tasks.rados.rados.0.trial023.stdout:121: oid 45 version 42 is already newer than 42 2026-02-25T08:42:09.337 INFO:tasks.rados.rados.0.trial023.stdout:121: oid 45 version 42 is already newer than 42 2026-02-25T08:42:09.337 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 45 v 42 (ObjNum 82 snap 7 seq_num 82) dirty exists 2026-02-25T08:42:09.338 INFO:tasks.rados.rados.0.trial023.stdout:121: left oid 45 (ObjNum 82 snap 7 seq_num 82) 2026-02-25T08:42:09.338 INFO:tasks.rados.rados.0.trial023.stdout:122: finishing write tid 5 to trial02342524-18 2026-02-25T08:42:09.338 INFO:tasks.rados.rados.0.trial023.stdout:122: oid 18 updating version 0 to 35 2026-02-25T08:42:09.338 INFO:tasks.rados.rados.0.trial023.stdout:122: oid 18 version 35 is already newer than 35 2026-02-25T08:42:09.338 INFO:tasks.rados.rados.0.trial023.stdout:122: oid 18 version 35 is already newer than 35 2026-02-25T08:42:09.338 INFO:tasks.rados.rados.0.trial023.stdout:122: oid 18 version 35 is already newer than 35 2026-02-25T08:42:09.338 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 18 v 35 (ObjNum 83 snap 7 seq_num 83) dirty exists 2026-02-25T08:42:09.338 INFO:tasks.rados.rados.0.trial023.stdout:122: left oid 18 (ObjNum 83 snap 7 seq_num 83) 2026-02-25T08:42:09.353 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:09.353 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:09.828 INFO:tasks.rados.rados.0.trial023.stdout:121: done (4 left) 2026-02-25T08:42:09.828 INFO:tasks.rados.rados.0.trial023.stdout:122: done (3 left) 2026-02-25T08:42:09.828 INFO:tasks.rados.rados.0.trial023.stdout:124: done (2 left) 2026-02-25T08:42:09.829 INFO:tasks.rados.rados.0.trial023.stdout:126: done (1 left) 2026-02-25T08:42:09.829 INFO:tasks.rados.rados.0.trial023.stdout:128: done (0 left) 2026-02-25T08:42:09.829 INFO:tasks.rados.rados.0.trial023.stdout:129: delete oid 48 current snap is 8 2026-02-25T08:42:09.868 INFO:tasks.rados.rados.0.trial023.stdout:129: done (0 left) 2026-02-25T08:42:09.868 INFO:tasks.rados.rados.0.trial023.stdout:130: read oid 45 snap 6 2026-02-25T08:42:09.868 INFO:tasks.rados.rados.0.trial023.stdout:130: expect (ObjNum 56 snap 1 seq_num 56) 2026-02-25T08:42:09.868 INFO:tasks.rados.rados.0.trial023.stdout:131: write oid 2 current snap is 8 2026-02-25T08:42:09.868 INFO:tasks.rados.rados.0.trial023.stdout:131: seq_num 87 ranges {662209=626546,1808051=636013,2873212=588997} 2026-02-25T08:42:09.873 INFO:tasks.rados.rados.0.trial023.stdout:131: writing trial02342524-2 from 662209 to 1288755 tid 1 2026-02-25T08:42:09.877 INFO:tasks.rados.rados.0.trial023.stdout:131: writing trial02342524-2 from 1808051 to 2444064 tid 2 2026-02-25T08:42:09.881 INFO:tasks.rados.rados.0.trial023.stdout:131: writing trial02342524-2 from 2873212 to 3462209 tid 3 2026-02-25T08:42:09.881 INFO:tasks.rados.rados.0.trial023.stdout:132: snap_create 2026-02-25T08:42:09.883 INFO:tasks.rados.rados.0.trial023.stdout:130: expect (ObjNum 56 snap 1 seq_num 56) 2026-02-25T08:42:09.933 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2626392 2026-02-25T08:42:09.944 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 552863 2026-02-25T08:42:09.973 INFO:tasks.rados.rados.0.trial023.stdout:131: finishing write tid 1 to trial02342524-2 2026-02-25T08:42:09.973 INFO:tasks.rados.rados.0.trial023.stdout:131: finishing write tid 2 to trial02342524-2 2026-02-25T08:42:09.973 INFO:tasks.rados.rados.0.trial023.stdout:131: finishing write tid 3 to trial02342524-2 2026-02-25T08:42:09.973 INFO:tasks.rados.rados.0.trial023.stdout:131: finishing write tid 4 to trial02342524-2 2026-02-25T08:42:09.973 INFO:tasks.rados.rados.0.trial023.stdout:131: finishing write tid 5 to trial02342524-2 2026-02-25T08:42:09.974 INFO:tasks.rados.rados.0.trial023.stdout:131: oid 2 updating version 0 to 18 2026-02-25T08:42:09.974 INFO:tasks.rados.rados.0.trial023.stdout:131: oid 2 version 18 is already newer than 18 2026-02-25T08:42:09.974 INFO:tasks.rados.rados.0.trial023.stdout:131: oid 2 version 18 is already newer than 18 2026-02-25T08:42:09.974 INFO:tasks.rados.rados.0.trial023.stdout:131: oid 2 version 18 is already newer than 18 2026-02-25T08:42:09.974 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 2 v 18 (ObjNum 87 snap 8 seq_num 87) dirty exists 2026-02-25T08:42:09.974 INFO:tasks.rados.rados.0.trial023.stdout:131: left oid 2 (ObjNum 87 snap 8 seq_num 87) 2026-02-25T08:42:10.834 INFO:tasks.rados.rados.0.trial023.stdout:130: done (2 left) 2026-02-25T08:42:10.834 INFO:tasks.rados.rados.0.trial023.stdout:131: done (1 left) 2026-02-25T08:42:10.834 INFO:tasks.rados.rados.0.trial023.stdout:132: done (0 left) 2026-02-25T08:42:10.834 INFO:tasks.rados.rados.0.trial023.stdout:133: snap_remove snap 7 2026-02-25T08:42:11.354 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:42:11.433 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:11.430+0000 7f8f3a59c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:11.436 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:11.434+0000 7f8f3a59c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:11.450 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:42:11.477 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:11.478 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:11.836 INFO:tasks.rados.rados.0.trial023.stdout:133: done (0 left) 2026-02-25T08:42:11.837 INFO:tasks.rados.rados.0.trial023.stdout:134: write (excl) oid 26 current snap is 9 2026-02-25T08:42:11.837 INFO:tasks.rados.rados.0.trial023.stdout:134: seq_num 88 ranges {679016=701795,1866533=496024,2900635=402643} 2026-02-25T08:42:11.842 INFO:tasks.rados.rados.0.trial023.stdout:134: writing trial02342524-26 from 679016 to 1380811 tid 1 2026-02-25T08:42:11.845 INFO:tasks.rados.rados.0.trial023.stdout:134: writing trial02342524-26 from 1866533 to 2362557 tid 2 2026-02-25T08:42:11.849 INFO:tasks.rados.rados.0.trial023.stdout:134: writing trial02342524-26 from 2900635 to 3303278 tid 3 2026-02-25T08:42:11.849 INFO:tasks.rados.rados.0.trial023.stdout:135: snap_remove snap 6 2026-02-25T08:42:12.092 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:42:12.160 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:42:12.163 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:42:12.163 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:42:12.843 INFO:tasks.rados.rados.0.trial023.stdout:135: done (1 left) 2026-02-25T08:42:12.843 INFO:tasks.rados.rados.0.trial023.stdout:136: read oid 7 snap -1 2026-02-25T08:42:12.843 INFO:tasks.rados.rados.0.trial023.stdout:136: expect (ObjNum 6 snap 0 seq_num 6) 2026-02-25T08:42:12.843 INFO:tasks.rados.rados.0.trial023.stdout:137: delete oid 3 current snap is 9 2026-02-25T08:42:12.844 INFO:tasks.rados.rados.0.trial023.stdout:134: finishing write tid 1 to trial02342524-26 2026-02-25T08:42:12.844 INFO:tasks.rados.rados.0.trial023.stdout:134: finishing write tid 2 to trial02342524-26 2026-02-25T08:42:12.844 INFO:tasks.rados.rados.0.trial023.stdout:134: finishing write tid 3 to trial02342524-26 2026-02-25T08:42:12.844 INFO:tasks.rados.rados.0.trial023.stdout:134: finishing write tid 4 to trial02342524-26 2026-02-25T08:42:12.844 INFO:tasks.rados.rados.0.trial023.stdout:134: finishing write tid 5 to trial02342524-26 2026-02-25T08:42:12.844 INFO:tasks.rados.rados.0.trial023.stdout:134: oid 26 updating version 0 to 36 2026-02-25T08:42:12.844 INFO:tasks.rados.rados.0.trial023.stdout:134: oid 26 version 36 is already newer than 33 2026-02-25T08:42:12.844 INFO:tasks.rados.rados.0.trial023.stdout:134: oid 26 version 36 is already newer than 36 2026-02-25T08:42:12.844 INFO:tasks.rados.rados.0.trial023.stdout:134: oid 26 version 36 is already newer than 36 2026-02-25T08:42:12.844 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 26 v 36 (ObjNum 88 snap 9 seq_num 88) dirty exists 2026-02-25T08:42:12.844 INFO:tasks.rados.rados.0.trial023.stdout:134: left oid 26 (ObjNum 88 snap 9 seq_num 88) 2026-02-25T08:42:12.874 INFO:tasks.rados.rados.0.trial023.stdout:136: expect (ObjNum 6 snap 0 seq_num 6) 2026-02-25T08:42:12.876 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1631716 2026-02-25T08:42:12.886 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1638697 2026-02-25T08:42:12.912 INFO:tasks.rados.rados.0.trial023.stdout:134: done (2 left) 2026-02-25T08:42:12.912 INFO:tasks.rados.rados.0.trial023.stdout:136: done (1 left) 2026-02-25T08:42:12.912 INFO:tasks.rados.rados.0.trial023.stdout:137: done (0 left) 2026-02-25T08:42:12.912 INFO:tasks.rados.rados.0.trial023.stdout:138: snap_remove snap 8 2026-02-25T08:42:13.444 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~33s 2026-02-25T08:42:13.479 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:42:13.556 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:13.554+0000 7f3f19a5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:13.559 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:13.557+0000 7f3f19a5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:13.574 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:42:13.600 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:13.601 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:13.844 INFO:tasks.rados.rados.0.trial023.stdout:138: done (0 left) 2026-02-25T08:42:13.844 INFO:tasks.rados.rados.0.trial023.stdout:139: write (excl) oid 16 current snap is 9 2026-02-25T08:42:13.844 INFO:tasks.rados.rados.0.trial023.stdout:139: seq_num 90 ranges {712630=452293,1583497=499693,2555481=425187} 2026-02-25T08:42:13.848 INFO:tasks.rados.rados.0.trial023.stdout:139: writing trial02342524-16 from 712630 to 1164923 tid 1 2026-02-25T08:42:13.851 INFO:tasks.rados.rados.0.trial023.stdout:139: writing trial02342524-16 from 1583497 to 2083190 tid 2 2026-02-25T08:42:13.855 INFO:tasks.rados.rados.0.trial023.stdout:139: writing trial02342524-16 from 2555481 to 2980668 tid 3 2026-02-25T08:42:13.855 INFO:tasks.rados.rados.0.trial023.stdout:140: write (excl) oid 25 current snap is 9 2026-02-25T08:42:13.855 INFO:tasks.rados.rados.0.trial023.stdout:140: seq_num 91 ranges {729437=527542,1925626=643351,3266551=262886} 2026-02-25T08:42:13.859 INFO:tasks.rados.rados.0.trial023.stdout:140: writing trial02342524-25 from 729437 to 1256979 tid 1 2026-02-25T08:42:13.864 INFO:tasks.rados.rados.0.trial023.stdout:140: writing trial02342524-25 from 1925626 to 2568977 tid 2 2026-02-25T08:42:13.868 INFO:tasks.rados.rados.0.trial023.stdout:140: writing trial02342524-25 from 3266551 to 3529437 tid 3 2026-02-25T08:42:13.868 INFO:tasks.rados.rados.0.trial023.stdout:141: write (excl) oid 46 current snap is 9 2026-02-25T08:42:13.868 INFO:tasks.rados.rados.0.trial023.stdout:141: seq_num 92 ranges {746244=719144,2100461=503362,3126680=419564} 2026-02-25T08:42:13.874 INFO:tasks.rados.rados.0.trial023.stdout:141: writing trial02342524-46 from 746244 to 1465388 tid 1 2026-02-25T08:42:13.878 INFO:tasks.rados.rados.0.trial023.stdout:141: writing trial02342524-46 from 2100461 to 2603823 tid 2 2026-02-25T08:42:13.882 INFO:tasks.rados.rados.0.trial023.stdout:141: writing trial02342524-46 from 3126680 to 3546244 tid 3 2026-02-25T08:42:13.882 INFO:tasks.rados.rados.0.trial023.stdout:142: read oid 43 snap -1 2026-02-25T08:42:13.882 INFO:tasks.rados.rados.0.trial023.stdout:142: expect (ObjNum 76 snap 3 seq_num 76) 2026-02-25T08:42:13.882 INFO:tasks.rados.rados.0.trial023.stdout:143: delete oid 20 current snap is 9 2026-02-25T08:42:13.882 INFO:tasks.rados.rados.0.trial023.stdout:140: finishing write tid 1 to trial02342524-25 2026-02-25T08:42:13.883 INFO:tasks.rados.rados.0.trial023.stdout:139: finishing write tid 1 to trial02342524-16 2026-02-25T08:42:13.902 INFO:tasks.rados.rados.0.trial023.stdout:140: finishing write tid 2 to trial02342524-25 2026-02-25T08:42:13.905 INFO:tasks.rados.rados.0.trial023.stdout:139: finishing write tid 2 to trial02342524-16 2026-02-25T08:42:13.910 INFO:tasks.rados.rados.0.trial023.stdout:139: finishing write tid 3 to trial02342524-16 2026-02-25T08:42:13.916 INFO:tasks.rados.rados.0.trial023.stdout:139: finishing write tid 4 to trial02342524-16 2026-02-25T08:42:13.918 INFO:tasks.rados.rados.0.trial023.stdout:140: finishing write tid 3 to trial02342524-25 2026-02-25T08:42:13.919 INFO:tasks.rados.rados.0.trial023.stdout:139: finishing write tid 5 to trial02342524-16 2026-02-25T08:42:13.919 INFO:tasks.rados.rados.0.trial023.stdout:139: oid 16 updating version 0 to 39 2026-02-25T08:42:13.919 INFO:tasks.rados.rados.0.trial023.stdout:139: oid 16 version 39 is already newer than 39 2026-02-25T08:42:13.919 INFO:tasks.rados.rados.0.trial023.stdout:139: oid 16 version 39 is already newer than 39 2026-02-25T08:42:13.919 INFO:tasks.rados.rados.0.trial023.stdout:139: oid 16 version 39 is already newer than 36 2026-02-25T08:42:13.919 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 16 v 39 (ObjNum 90 snap 9 seq_num 90) dirty exists 2026-02-25T08:42:13.919 INFO:tasks.rados.rados.0.trial023.stdout:139: left oid 16 (ObjNum 90 snap 9 seq_num 90) 2026-02-25T08:42:13.924 INFO:tasks.rados.rados.0.trial023.stdout:142: expect (ObjNum 76 snap 3 seq_num 76) 2026-02-25T08:42:13.968 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:42:14.005 INFO:tasks.rados.rados.0.trial023.stdout:140: finishing write tid 4 to trial02342524-25 2026-02-25T08:42:14.005 INFO:tasks.rados.rados.0.trial023.stdout:140: finishing write tid 5 to trial02342524-25 2026-02-25T08:42:14.005 INFO:tasks.rados.rados.0.trial023.stdout:140: oid 25 updating version 0 to 48 2026-02-25T08:42:14.005 INFO:tasks.rados.rados.0.trial023.stdout:140: oid 25 version 48 is already newer than 45 2026-02-25T08:42:14.005 INFO:tasks.rados.rados.0.trial023.stdout:140: oid 25 version 48 is already newer than 48 2026-02-25T08:42:14.006 INFO:tasks.rados.rados.0.trial023.stdout:140: oid 25 version 48 is already newer than 48 2026-02-25T08:42:14.006 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 25 v 48 (ObjNum 91 snap 9 seq_num 91) dirty exists 2026-02-25T08:42:14.006 INFO:tasks.rados.rados.0.trial023.stdout:140: left oid 25 (ObjNum 91 snap 9 seq_num 91) 2026-02-25T08:42:14.006 INFO:tasks.rados.rados.0.trial023.stdout:141: finishing write tid 1 to trial02342524-46 2026-02-25T08:42:14.006 INFO:tasks.rados.rados.0.trial023.stdout:141: finishing write tid 2 to trial02342524-46 2026-02-25T08:42:14.006 INFO:tasks.rados.rados.0.trial023.stdout:139: done (4 left) 2026-02-25T08:42:14.006 INFO:tasks.rados.rados.0.trial023.stdout:140: done (3 left) 2026-02-25T08:42:14.006 INFO:tasks.rados.rados.0.trial023.stdout:142: done (2 left) 2026-02-25T08:42:14.006 INFO:tasks.rados.rados.0.trial023.stdout:143: done (1 left) 2026-02-25T08:42:14.006 INFO:tasks.rados.rados.0.trial023.stdout:144: snap_create 2026-02-25T08:42:14.006 INFO:tasks.rados.rados.0.trial023.stdout:141: finishing write tid 3 to trial02342524-46 2026-02-25T08:42:14.006 INFO:tasks.rados.rados.0.trial023.stdout:141: finishing write tid 4 to trial02342524-46 2026-02-25T08:42:14.007 INFO:tasks.rados.rados.0.trial023.stdout:141: finishing write tid 5 to trial02342524-46 2026-02-25T08:42:14.007 INFO:tasks.rados.rados.0.trial023.stdout:141: oid 46 updating version 0 to 42 2026-02-25T08:42:14.007 INFO:tasks.rados.rados.0.trial023.stdout:141: oid 46 version 42 is already newer than 42 2026-02-25T08:42:14.007 INFO:tasks.rados.rados.0.trial023.stdout:141: oid 46 version 42 is already newer than 42 2026-02-25T08:42:14.007 INFO:tasks.rados.rados.0.trial023.stdout:141: oid 46 version 42 is already newer than 39 2026-02-25T08:42:14.007 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 46 v 42 (ObjNum 92 snap 9 seq_num 92) dirty exists 2026-02-25T08:42:14.007 INFO:tasks.rados.rados.0.trial023.stdout:141: left oid 46 (ObjNum 92 snap 9 seq_num 92) 2026-02-25T08:42:14.848 INFO:tasks.rados.rados.0.trial023.stdout:141: done (1 left) 2026-02-25T08:42:14.848 INFO:tasks.rados.rados.0.trial023.stdout:144: done (0 left) 2026-02-25T08:42:14.848 INFO:tasks.rados.rados.0.trial023.stdout:145: read oid 1 snap -1 2026-02-25T08:42:14.849 INFO:tasks.rados.rados.0.trial023.stdout:145: expect (ObjNum 0 snap 0 seq_num 0) 2026-02-25T08:42:14.849 INFO:tasks.rados.rados.0.trial023.stdout:146: snap_remove snap 9 2026-02-25T08:42:15.602 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:42:15.679 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:15.677+0000 7fc9182cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:15.682 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:15.680+0000 7fc9182cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:15.696 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:42:15.724 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:15.724 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:15.850 INFO:tasks.rados.rados.0.trial023.stdout:146: done (1 left) 2026-02-25T08:42:15.851 INFO:tasks.rados.rados.0.trial023.stdout:147: delete oid 6 current snap is 10 2026-02-25T08:42:15.851 INFO:tasks.rados.rados.0.trial023.stdout:145: expect (ObjNum 0 snap 0 seq_num 0) 2026-02-25T08:42:15.851 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 328614 2026-02-25T08:42:15.865 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:42:15.894 INFO:tasks.rados.rados.0.trial023.stdout:145: done (1 left) 2026-02-25T08:42:15.895 INFO:tasks.rados.rados.0.trial023.stdout:147: done (0 left) 2026-02-25T08:42:15.895 INFO:tasks.rados.rados.0.trial023.stdout:148: write oid 41 current snap is 10 2026-02-25T08:42:15.895 INFO:tasks.rados.rados.0.trial023.stdout:148: seq_num 95 ranges {796665=544891,1759554=650689,2976243=620422} 2026-02-25T08:42:15.899 INFO:tasks.rados.rados.0.trial023.stdout:148: writing trial02342524-41 from 796665 to 1341556 tid 1 2026-02-25T08:42:15.903 INFO:tasks.rados.rados.0.trial023.stdout:148: writing trial02342524-41 from 1759554 to 2410243 tid 2 2026-02-25T08:42:15.908 INFO:tasks.rados.rados.0.trial023.stdout:148: writing trial02342524-41 from 2976243 to 3596665 tid 3 2026-02-25T08:42:15.908 INFO:tasks.rados.rados.0.trial023.stdout:149: write oid 20 current snap is 10 2026-02-25T08:42:15.908 INFO:tasks.rados.rados.0.trial023.stdout:149: seq_num 96 ranges {413472=620140,1818036=510700,3120019=493453} 2026-02-25T08:42:15.911 INFO:tasks.rados.rados.0.trial023.stdout:149: writing trial02342524-20 from 413472 to 1033612 tid 1 2026-02-25T08:42:15.916 INFO:tasks.rados.rados.0.trial023.stdout:149: writing trial02342524-20 from 1818036 to 2328736 tid 2 2026-02-25T08:42:15.921 INFO:tasks.rados.rados.0.trial023.stdout:149: writing trial02342524-20 from 3120019 to 3613472 tid 3 2026-02-25T08:42:15.921 INFO:tasks.rados.rados.0.trial023.stdout:150: delete oid 30 current snap is 10 2026-02-25T08:42:15.923 INFO:tasks.rados.rados.0.trial023.stdout:148: finishing write tid 1 to trial02342524-41 2026-02-25T08:42:15.931 INFO:tasks.rados.rados.0.trial023.stdout:149: finishing write tid 1 to trial02342524-20 2026-02-25T08:42:15.936 INFO:tasks.rados.rados.0.trial023.stdout:148: finishing write tid 2 to trial02342524-41 2026-02-25T08:42:15.938 INFO:tasks.rados.rados.0.trial023.stdout:149: finishing write tid 2 to trial02342524-20 2026-02-25T08:42:15.944 INFO:tasks.rados.rados.0.trial023.stdout:148: finishing write tid 3 to trial02342524-41 2026-02-25T08:42:15.944 INFO:tasks.rados.rados.0.trial023.stdout:149: finishing write tid 3 to trial02342524-20 2026-02-25T08:42:15.948 INFO:tasks.rados.rados.0.trial023.stdout:149: finishing write tid 4 to trial02342524-20 2026-02-25T08:42:15.952 INFO:tasks.rados.rados.0.trial023.stdout:149: finishing write tid 5 to trial02342524-20 2026-02-25T08:42:15.953 INFO:tasks.rados.rados.0.trial023.stdout:149: oid 20 updating version 0 to 53 2026-02-25T08:42:15.953 INFO:tasks.rados.rados.0.trial023.stdout:149: oid 20 version 53 is already newer than 53 2026-02-25T08:42:15.953 INFO:tasks.rados.rados.0.trial023.stdout:149: oid 20 version 53 is already newer than 53 2026-02-25T08:42:15.953 INFO:tasks.rados.rados.0.trial023.stdout:149: oid 20 version 53 is already newer than 53 2026-02-25T08:42:15.953 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 20 v 53 (ObjNum 96 snap 10 seq_num 96) dirty exists 2026-02-25T08:42:15.953 INFO:tasks.rados.rados.0.trial023.stdout:149: left oid 20 (ObjNum 96 snap 10 seq_num 96) 2026-02-25T08:42:15.953 INFO:tasks.rados.rados.0.trial023.stdout:148: finishing write tid 4 to trial02342524-41 2026-02-25T08:42:15.956 INFO:tasks.rados.rados.0.trial023.stdout:148: finishing write tid 5 to trial02342524-41 2026-02-25T08:42:15.956 INFO:tasks.rados.rados.0.trial023.stdout:148: oid 41 updating version 0 to 27 2026-02-25T08:42:15.956 INFO:tasks.rados.rados.0.trial023.stdout:148: oid 41 version 27 is already newer than 27 2026-02-25T08:42:15.956 INFO:tasks.rados.rados.0.trial023.stdout:148: oid 41 version 27 is already newer than 27 2026-02-25T08:42:15.956 INFO:tasks.rados.rados.0.trial023.stdout:148: oid 41 version 27 is already newer than 27 2026-02-25T08:42:15.956 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 41 v 27 (ObjNum 95 snap 10 seq_num 95) dirty exists 2026-02-25T08:42:15.956 INFO:tasks.rados.rados.0.trial023.stdout:148: left oid 41 (ObjNum 95 snap 10 seq_num 95) 2026-02-25T08:42:15.959 INFO:tasks.rados.rados.0.trial023.stdout:148: done (2 left) 2026-02-25T08:42:15.959 INFO:tasks.rados.rados.0.trial023.stdout:149: done (1 left) 2026-02-25T08:42:15.959 INFO:tasks.rados.rados.0.trial023.stdout:150: done (0 left) 2026-02-25T08:42:15.959 INFO:tasks.rados.rados.0.trial023.stdout:151: write (excl) oid 45 current snap is 10 2026-02-25T08:42:15.959 INFO:tasks.rados.rados.0.trial023.stdout:151: seq_num 98 ranges {447086=770638,1935000=798016,3458512=188574} 2026-02-25T08:42:15.963 INFO:tasks.rados.rados.0.trial023.stdout:151: writing trial02342524-45 from 447086 to 1217724 tid 1 2026-02-25T08:42:15.969 INFO:tasks.rados.rados.0.trial023.stdout:151: writing trial02342524-45 from 1935000 to 2733016 tid 2 2026-02-25T08:42:15.973 INFO:tasks.rados.rados.0.trial023.stdout:151: writing trial02342524-45 from 3458512 to 3647086 tid 3 2026-02-25T08:42:15.973 INFO:tasks.rados.rados.0.trial023.stdout:152: write (excl) oid 4 current snap is 10 2026-02-25T08:42:15.973 INFO:tasks.rados.rados.0.trial023.stdout:152: seq_num 99 ranges {463893=562240,1593482=658027,2685935=559341} 2026-02-25T08:42:15.977 INFO:tasks.rados.rados.0.trial023.stdout:152: writing trial02342524-4 from 463893 to 1026133 tid 1 2026-02-25T08:42:15.981 INFO:tasks.rados.rados.0.trial023.stdout:152: writing trial02342524-4 from 1593482 to 2251509 tid 2 2026-02-25T08:42:15.985 INFO:tasks.rados.rados.0.trial023.stdout:152: writing trial02342524-4 from 2685935 to 3245276 tid 3 2026-02-25T08:42:15.986 INFO:tasks.rados.rados.0.trial023.stdout:153: delete oid 30 current snap is 10 2026-02-25T08:42:15.986 INFO:tasks.rados.rados.0.trial023.stdout:151: finishing write tid 1 to trial02342524-45 2026-02-25T08:42:15.995 INFO:tasks.rados.rados.0.trial023.stdout:153: done (2 left) 2026-02-25T08:42:15.995 INFO:tasks.rados.rados.0.trial023.stdout:154: write (excl) oid 3 current snap is 10 2026-02-25T08:42:15.995 INFO:tasks.rados.rados.0.trial023.stdout:154: seq_num 101 ranges {497507=712738,1710446=661696,3140781=556726} 2026-02-25T08:42:15.999 INFO:tasks.rados.rados.0.trial023.stdout:154: writing trial02342524-3 from 497507 to 1210245 tid 1 2026-02-25T08:42:16.004 INFO:tasks.rados.rados.0.trial023.stdout:154: writing trial02342524-3 from 1710446 to 2372142 tid 2 2026-02-25T08:42:16.010 INFO:tasks.rados.rados.0.trial023.stdout:154: writing trial02342524-3 from 3140781 to 3697507 tid 3 2026-02-25T08:42:16.010 INFO:tasks.rados.rados.0.trial023.stdout:155: write oid 26 current snap is 10 2026-02-25T08:42:16.010 INFO:tasks.rados.rados.0.trial023.stdout:151: finishing write tid 2 to trial02342524-45 2026-02-25T08:42:16.010 INFO:tasks.rados.rados.0.trial023.stdout:151: finishing write tid 3 to trial02342524-45 2026-02-25T08:42:16.010 INFO:tasks.rados.rados.0.trial023.stdout:151: finishing write tid 4 to trial02342524-45 2026-02-25T08:42:16.010 INFO:tasks.rados.rados.0.trial023.stdout:155: seq_num 102 ranges {514314=787987,1768928=405354,2768204=509510} 2026-02-25T08:42:16.014 INFO:tasks.rados.rados.0.trial023.stdout:155: writing trial02342524-26 from 514314 to 1302301 tid 1 2026-02-25T08:42:16.018 INFO:tasks.rados.rados.0.trial023.stdout:155: writing trial02342524-26 from 1768928 to 2174282 tid 2 2026-02-25T08:42:16.022 INFO:tasks.rados.rados.0.trial023.stdout:155: writing trial02342524-26 from 2768204 to 3277714 tid 3 2026-02-25T08:42:16.022 INFO:tasks.rados.rados.0.trial023.stdout:156: snap_create 2026-02-25T08:42:16.022 INFO:tasks.rados.rados.0.trial023.stdout:151: finishing write tid 5 to trial02342524-45 2026-02-25T08:42:16.023 INFO:tasks.rados.rados.0.trial023.stdout:151: oid 45 updating version 0 to 54 2026-02-25T08:42:16.023 INFO:tasks.rados.rados.0.trial023.stdout:151: oid 45 updating version 54 to 57 2026-02-25T08:42:16.023 INFO:tasks.rados.rados.0.trial023.stdout:151: oid 45 version 57 is already newer than 57 2026-02-25T08:42:16.023 INFO:tasks.rados.rados.0.trial023.stdout:151: oid 45 version 57 is already newer than 57 2026-02-25T08:42:16.023 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 45 v 57 (ObjNum 98 snap 10 seq_num 98) dirty exists 2026-02-25T08:42:16.023 INFO:tasks.rados.rados.0.trial023.stdout:151: left oid 45 (ObjNum 98 snap 10 seq_num 98) 2026-02-25T08:42:16.023 INFO:tasks.rados.rados.0.trial023.stdout:152: finishing write tid 1 to trial02342524-4 2026-02-25T08:42:16.033 INFO:tasks.rados.rados.0.trial023.stdout:152: finishing write tid 2 to trial02342524-4 2026-02-25T08:42:16.035 INFO:tasks.rados.rados.0.trial023.stdout:155: finishing write tid 1 to trial02342524-26 2026-02-25T08:42:16.039 INFO:tasks.rados.rados.0.trial023.stdout:152: finishing write tid 3 to trial02342524-4 2026-02-25T08:42:16.043 INFO:tasks.rados.rados.0.trial023.stdout:155: finishing write tid 2 to trial02342524-26 2026-02-25T08:42:16.044 INFO:tasks.rados.rados.0.trial023.stdout:152: finishing write tid 4 to trial02342524-4 2026-02-25T08:42:16.046 INFO:tasks.rados.rados.0.trial023.stdout:152: finishing write tid 5 to trial02342524-4 2026-02-25T08:42:16.046 INFO:tasks.rados.rados.0.trial023.stdout:152: oid 4 updating version 0 to 58 2026-02-25T08:42:16.046 INFO:tasks.rados.rados.0.trial023.stdout:152: oid 4 updating version 58 to 61 2026-02-25T08:42:16.046 INFO:tasks.rados.rados.0.trial023.stdout:152: oid 4 version 61 is already newer than 61 2026-02-25T08:42:16.046 INFO:tasks.rados.rados.0.trial023.stdout:152: oid 4 version 61 is already newer than 61 2026-02-25T08:42:16.046 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 4 v 61 (ObjNum 99 snap 10 seq_num 99) dirty exists 2026-02-25T08:42:16.047 INFO:tasks.rados.rados.0.trial023.stdout:152: left oid 4 (ObjNum 99 snap 10 seq_num 99) 2026-02-25T08:42:16.049 INFO:tasks.rados.rados.0.trial023.stdout:155: finishing write tid 3 to trial02342524-26 2026-02-25T08:42:16.050 INFO:tasks.rados.rados.0.trial023.stdout:154: finishing write tid 1 to trial02342524-3 2026-02-25T08:42:16.054 INFO:tasks.rados.rados.0.trial023.stdout:155: finishing write tid 4 to trial02342524-26 2026-02-25T08:42:16.055 INFO:tasks.rados.rados.0.trial023.stdout:155: finishing write tid 5 to trial02342524-26 2026-02-25T08:42:16.056 INFO:tasks.rados.rados.0.trial023.stdout:155: oid 26 updating version 0 to 40 2026-02-25T08:42:16.056 INFO:tasks.rados.rados.0.trial023.stdout:155: oid 26 version 40 is already newer than 40 2026-02-25T08:42:16.056 INFO:tasks.rados.rados.0.trial023.stdout:155: oid 26 version 40 is already newer than 40 2026-02-25T08:42:16.056 INFO:tasks.rados.rados.0.trial023.stdout:155: oid 26 version 40 is already newer than 40 2026-02-25T08:42:16.056 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 26 v 40 (ObjNum 102 snap 10 seq_num 102) dirty exists 2026-02-25T08:42:16.056 INFO:tasks.rados.rados.0.trial023.stdout:155: left oid 26 (ObjNum 102 snap 10 seq_num 102) 2026-02-25T08:42:16.062 INFO:tasks.rados.rados.0.trial023.stdout:154: finishing write tid 2 to trial02342524-3 2026-02-25T08:42:16.068 INFO:tasks.rados.rados.0.trial023.stdout:154: finishing write tid 3 to trial02342524-3 2026-02-25T08:42:16.070 INFO:tasks.rados.rados.0.trial023.stdout:154: finishing write tid 4 to trial02342524-3 2026-02-25T08:42:16.072 INFO:tasks.rados.rados.0.trial023.stdout:154: finishing write tid 5 to trial02342524-3 2026-02-25T08:42:16.072 INFO:tasks.rados.rados.0.trial023.stdout:154: oid 3 updating version 0 to 62 2026-02-25T08:42:16.073 INFO:tasks.rados.rados.0.trial023.stdout:154: oid 3 updating version 62 to 65 2026-02-25T08:42:16.073 INFO:tasks.rados.rados.0.trial023.stdout:154: oid 3 version 65 is already newer than 65 2026-02-25T08:42:16.073 INFO:tasks.rados.rados.0.trial023.stdout:154: oid 3 version 65 is already newer than 65 2026-02-25T08:42:16.073 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 3 v 65 (ObjNum 101 snap 10 seq_num 101) dirty exists 2026-02-25T08:42:16.073 INFO:tasks.rados.rados.0.trial023.stdout:154: left oid 3 (ObjNum 101 snap 10 seq_num 101) 2026-02-25T08:42:16.856 INFO:tasks.rados.rados.0.trial023.stdout:151: done (4 left) 2026-02-25T08:42:16.856 INFO:tasks.rados.rados.0.trial023.stdout:152: done (3 left) 2026-02-25T08:42:16.856 INFO:tasks.rados.rados.0.trial023.stdout:154: done (2 left) 2026-02-25T08:42:16.856 INFO:tasks.rados.rados.0.trial023.stdout:155: done (1 left) 2026-02-25T08:42:16.856 INFO:tasks.rados.rados.0.trial023.stdout:156: done (0 left) 2026-02-25T08:42:16.856 INFO:tasks.rados.rados.0.trial023.stdout:157: snap_create 2026-02-25T08:42:17.164 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:42:17.237 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:42:17.240 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:42:17.241 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:42:17.726 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:42:17.802 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:17.799+0000 7f51edc5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:17.804 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:17.803+0000 7f51edc5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:17.819 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:42:17.852 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:17.853 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:17.857 INFO:tasks.rados.rados.0.trial023.stdout:157: done (0 left) 2026-02-25T08:42:17.857 INFO:tasks.rados.rados.0.trial023.stdout:158: read oid 2 snap -1 2026-02-25T08:42:17.857 INFO:tasks.rados.rados.0.trial023.stdout:158: expect (ObjNum 87 snap 8 seq_num 87) 2026-02-25T08:42:17.857 INFO:tasks.rados.rados.0.trial023.stdout:159: snap_remove snap 11 2026-02-25T08:42:18.861 INFO:tasks.rados.rados.0.trial023.stdout:159: done (1 left) 2026-02-25T08:42:18.861 INFO:tasks.rados.rados.0.trial023.stdout:160: write (excl) oid 40 current snap is 12 2026-02-25T08:42:18.861 INFO:tasks.rados.rados.0.trial023.stdout:160: seq_num 103 ranges {531121=463236,1711057=665365,3079274=651847} 2026-02-25T08:42:18.864 INFO:tasks.rados.rados.0.trial023.stdout:160: writing trial02342524-40 from 531121 to 994357 tid 1 2026-02-25T08:42:18.869 INFO:tasks.rados.rados.0.trial023.stdout:160: writing trial02342524-40 from 1711057 to 2376422 tid 2 2026-02-25T08:42:18.875 INFO:tasks.rados.rados.0.trial023.stdout:160: writing trial02342524-40 from 3079274 to 3731121 tid 3 2026-02-25T08:42:18.875 INFO:tasks.rados.rados.0.trial023.stdout:161: read oid 19 snap 10 2026-02-25T08:42:18.875 INFO:tasks.rados.rados.0.trial023.stdout:161: expect (ObjNum 18 snap 0 seq_num 18) 2026-02-25T08:42:18.875 INFO:tasks.rados.rados.0.trial023.stdout:162: write (excl) oid 34 current snap is 12 2026-02-25T08:42:18.875 INFO:tasks.rados.rados.0.trial023.stdout:162: seq_num 104 ranges {547928=538485,1769539=409023,2706697=532054} 2026-02-25T08:42:18.879 INFO:tasks.rados.rados.0.trial023.stdout:162: writing trial02342524-34 from 547928 to 1086413 tid 1 2026-02-25T08:42:18.883 INFO:tasks.rados.rados.0.trial023.stdout:162: writing trial02342524-34 from 1769539 to 2178562 tid 2 2026-02-25T08:42:18.887 INFO:tasks.rados.rados.0.trial023.stdout:162: writing trial02342524-34 from 2706697 to 3238751 tid 3 2026-02-25T08:42:18.888 INFO:tasks.rados.rados.0.trial023.stdout:163: read oid 5 snap 10 2026-02-25T08:42:18.888 INFO:tasks.rados.rados.0.trial023.stdout:163: expect (ObjNum 4 snap 0 seq_num 4) 2026-02-25T08:42:18.888 INFO:tasks.rados.rados.0.trial023.stdout:164: read oid 16 snap -1 2026-02-25T08:42:18.888 INFO:tasks.rados.rados.0.trial023.stdout:164: expect (ObjNum 90 snap 9 seq_num 90) 2026-02-25T08:42:18.888 INFO:tasks.rados.rados.0.trial023.stdout:165: write oid 4 current snap is 12 2026-02-25T08:42:18.888 INFO:tasks.rados.rados.0.trial023.stdout:158: expect (ObjNum 87 snap 8 seq_num 87) 2026-02-25T08:42:18.888 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:42:18.946 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 3281428 2026-02-25T08:42:18.948 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~39s 2026-02-25T08:42:18.954 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1211468 2026-02-25T08:42:18.980 INFO:tasks.rados.rados.0.trial023.stdout:165: seq_num 105 ranges {564735=613734,1828021=669034,3134120=630615} 2026-02-25T08:42:18.986 INFO:tasks.rados.rados.0.trial023.stdout:165: writing trial02342524-4 from 564735 to 1178469 tid 1 2026-02-25T08:42:18.991 INFO:tasks.rados.rados.0.trial023.stdout:165: writing trial02342524-4 from 1828021 to 2497055 tid 2 2026-02-25T08:42:18.996 INFO:tasks.rados.rados.0.trial023.stdout:165: writing trial02342524-4 from 3134120 to 3764735 tid 3 2026-02-25T08:42:18.996 INFO:tasks.rados.rados.0.trial023.stdout:158: done (6 left) 2026-02-25T08:42:18.996 INFO:tasks.rados.rados.0.trial023.stdout:166: delete oid 10 current snap is 12 2026-02-25T08:42:18.996 INFO:tasks.rados.rados.0.trial023.stdout:161: expect (ObjNum 18 snap 0 seq_num 18) 2026-02-25T08:42:19.000 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1073764 2026-02-25T08:42:19.049 INFO:tasks.rados.rados.0.trial023.stdout:160: finishing write tid 1 to trial02342524-40 2026-02-25T08:42:19.049 INFO:tasks.rados.rados.0.trial023.stdout:164: expect (ObjNum 90 snap 9 seq_num 90) 2026-02-25T08:42:19.189 INFO:tasks.rados.rados.0.trial023.stdout:163: expect (ObjNum 4 snap 0 seq_num 4) 2026-02-25T08:42:19.201 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1386226 2026-02-25T08:42:19.211 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:42:19.230 INFO:tasks.rados.rados.0.trial023.stdout:160: finishing write tid 2 to trial02342524-40 2026-02-25T08:42:19.230 INFO:tasks.rados.rados.0.trial023.stdout:160: finishing write tid 3 to trial02342524-40 2026-02-25T08:42:19.230 INFO:tasks.rados.rados.0.trial023.stdout:161: done (6 left) 2026-02-25T08:42:19.230 INFO:tasks.rados.rados.0.trial023.stdout:163: done (5 left) 2026-02-25T08:42:19.230 INFO:tasks.rados.rados.0.trial023.stdout:164: done (4 left) 2026-02-25T08:42:19.230 INFO:tasks.rados.rados.0.trial023.stdout:166: done (3 left) 2026-02-25T08:42:19.230 INFO:tasks.rados.rados.0.trial023.stdout:167: delete oid 49 current snap is 12 2026-02-25T08:42:19.230 INFO:tasks.rados.rados.0.trial023.stdout:160: finishing write tid 4 to trial02342524-40 2026-02-25T08:42:19.230 INFO:tasks.rados.rados.0.trial023.stdout:160: finishing write tid 5 to trial02342524-40 2026-02-25T08:42:19.230 INFO:tasks.rados.rados.0.trial023.stdout:160: oid 40 updating version 0 to 29 2026-02-25T08:42:19.230 INFO:tasks.rados.rados.0.trial023.stdout:160: oid 40 version 29 is already newer than 29 2026-02-25T08:42:19.230 INFO:tasks.rados.rados.0.trial023.stdout:160: oid 40 version 29 is already newer than 29 2026-02-25T08:42:19.231 INFO:tasks.rados.rados.0.trial023.stdout:160: oid 40 version 29 is already newer than 26 2026-02-25T08:42:19.231 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 40 v 29 (ObjNum 103 snap 12 seq_num 103) dirty exists 2026-02-25T08:42:19.231 INFO:tasks.rados.rados.0.trial023.stdout:160: left oid 40 (ObjNum 103 snap 12 seq_num 103) 2026-02-25T08:42:19.231 INFO:tasks.rados.rados.0.trial023.stdout:162: finishing write tid 1 to trial02342524-34 2026-02-25T08:42:19.231 INFO:tasks.rados.rados.0.trial023.stdout:162: finishing write tid 2 to trial02342524-34 2026-02-25T08:42:19.231 INFO:tasks.rados.rados.0.trial023.stdout:162: finishing write tid 3 to trial02342524-34 2026-02-25T08:42:19.231 INFO:tasks.rados.rados.0.trial023.stdout:162: finishing write tid 4 to trial02342524-34 2026-02-25T08:42:19.231 INFO:tasks.rados.rados.0.trial023.stdout:162: finishing write tid 5 to trial02342524-34 2026-02-25T08:42:19.231 INFO:tasks.rados.rados.0.trial023.stdout:162: oid 34 updating version 0 to 23 2026-02-25T08:42:19.231 INFO:tasks.rados.rados.0.trial023.stdout:162: oid 34 version 23 is already newer than 23 2026-02-25T08:42:19.231 INFO:tasks.rados.rados.0.trial023.stdout:162: oid 34 version 23 is already newer than 23 2026-02-25T08:42:19.231 INFO:tasks.rados.rados.0.trial023.stdout:162: oid 34 version 23 is already newer than 20 2026-02-25T08:42:19.231 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 34 v 23 (ObjNum 104 snap 12 seq_num 104) dirty exists 2026-02-25T08:42:19.231 INFO:tasks.rados.rados.0.trial023.stdout:162: left oid 34 (ObjNum 104 snap 12 seq_num 104) 2026-02-25T08:42:19.231 INFO:tasks.rados.rados.0.trial023.stdout:165: finishing write tid 1 to trial02342524-4 2026-02-25T08:42:19.231 INFO:tasks.rados.rados.0.trial023.stdout:165: finishing write tid 2 to trial02342524-4 2026-02-25T08:42:19.231 INFO:tasks.rados.rados.0.trial023.stdout:165: finishing write tid 3 to trial02342524-4 2026-02-25T08:42:19.231 INFO:tasks.rados.rados.0.trial023.stdout:165: finishing write tid 4 to trial02342524-4 2026-02-25T08:42:19.232 INFO:tasks.rados.rados.0.trial023.stdout:165: finishing write tid 5 to trial02342524-4 2026-02-25T08:42:19.232 INFO:tasks.rados.rados.0.trial023.stdout:165: oid 4 updating version 0 to 70 2026-02-25T08:42:19.232 INFO:tasks.rados.rados.0.trial023.stdout:165: oid 4 version 70 is already newer than 70 2026-02-25T08:42:19.232 INFO:tasks.rados.rados.0.trial023.stdout:165: oid 4 version 70 is already newer than 70 2026-02-25T08:42:19.232 INFO:tasks.rados.rados.0.trial023.stdout:165: oid 4 version 70 is already newer than 70 2026-02-25T08:42:19.232 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 4 v 70 (ObjNum 105 snap 12 seq_num 105) dirty exists 2026-02-25T08:42:19.232 INFO:tasks.rados.rados.0.trial023.stdout:165: left oid 4 (ObjNum 105 snap 12 seq_num 105) 2026-02-25T08:42:19.251 INFO:tasks.rados.rados.0.trial023.stdout:160: done (3 left) 2026-02-25T08:42:19.252 INFO:tasks.rados.rados.0.trial023.stdout:162: done (2 left) 2026-02-25T08:42:19.252 INFO:tasks.rados.rados.0.trial023.stdout:165: done (1 left) 2026-02-25T08:42:19.252 INFO:tasks.rados.rados.0.trial023.stdout:167: done (0 left) 2026-02-25T08:42:19.252 INFO:tasks.rados.rados.0.trial023.stdout:168: write (excl) oid 24 current snap is 12 2026-02-25T08:42:19.252 INFO:tasks.rados.rados.0.trial023.stdout:168: seq_num 108 ranges {615156=555834,1603467=416361,2816389=693495} 2026-02-25T08:42:19.256 INFO:tasks.rados.rados.0.trial023.stdout:168: writing trial02342524-24 from 615156 to 1170990 tid 1 2026-02-25T08:42:19.259 INFO:tasks.rados.rados.0.trial023.stdout:168: writing trial02342524-24 from 1603467 to 2019828 tid 2 2026-02-25T08:42:19.265 INFO:tasks.rados.rados.0.trial023.stdout:168: writing trial02342524-24 from 2816389 to 3509884 tid 3 2026-02-25T08:42:19.265 INFO:tasks.rados.rados.0.trial023.stdout:169: snap_remove snap 10 2026-02-25T08:42:19.854 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:42:19.864 INFO:tasks.rados.rados.0.trial023.stdout:169: done (1 left) 2026-02-25T08:42:19.864 INFO:tasks.rados.rados.0.trial023.stdout:170: write (excl) oid 49 current snap is 12 2026-02-25T08:42:19.865 INFO:tasks.rados.rados.0.trial023.stdout:168: finishing write tid 1 to trial02342524-24 2026-02-25T08:42:19.865 INFO:tasks.rados.rados.0.trial023.stdout:170: seq_num 109 ranges {631963=631083,2061949=560019,3243812=504767} 2026-02-25T08:42:19.869 INFO:tasks.rados.rados.0.trial023.stdout:170: writing trial02342524-49 from 631963 to 1263046 tid 1 2026-02-25T08:42:19.874 INFO:tasks.rados.rados.0.trial023.stdout:170: writing trial02342524-49 from 2061949 to 2621968 tid 2 2026-02-25T08:42:19.879 INFO:tasks.rados.rados.0.trial023.stdout:170: writing trial02342524-49 from 3243812 to 3748579 tid 3 2026-02-25T08:42:19.880 INFO:tasks.rados.rados.0.trial023.stdout:171: delete oid 13 current snap is 12 2026-02-25T08:42:19.880 INFO:tasks.rados.rados.0.trial023.stdout:168: finishing write tid 2 to trial02342524-24 2026-02-25T08:42:19.880 INFO:tasks.rados.rados.0.trial023.stdout:168: finishing write tid 3 to trial02342524-24 2026-02-25T08:42:19.880 INFO:tasks.rados.rados.0.trial023.stdout:168: finishing write tid 4 to trial02342524-24 2026-02-25T08:42:19.880 INFO:tasks.rados.rados.0.trial023.stdout:168: finishing write tid 5 to trial02342524-24 2026-02-25T08:42:19.880 INFO:tasks.rados.rados.0.trial023.stdout:168: oid 24 updating version 0 to 34 2026-02-25T08:42:19.880 INFO:tasks.rados.rados.0.trial023.stdout:168: oid 24 version 34 is already newer than 34 2026-02-25T08:42:19.880 INFO:tasks.rados.rados.0.trial023.stdout:168: oid 24 version 34 is already newer than 34 2026-02-25T08:42:19.880 INFO:tasks.rados.rados.0.trial023.stdout:168: oid 24 version 34 is already newer than 31 2026-02-25T08:42:19.880 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 24 v 34 (ObjNum 108 snap 12 seq_num 108) dirty exists 2026-02-25T08:42:19.880 INFO:tasks.rados.rados.0.trial023.stdout:168: left oid 24 (ObjNum 108 snap 12 seq_num 108) 2026-02-25T08:42:19.894 INFO:tasks.rados.rados.0.trial023.stdout:168: done (2 left) 2026-02-25T08:42:19.895 INFO:tasks.rados.rados.0.trial023.stdout:171: done (1 left) 2026-02-25T08:42:19.895 INFO:tasks.rados.rados.0.trial023.stdout:172: delete oid 29 current snap is 12 2026-02-25T08:42:19.898 INFO:tasks.rados.rados.0.trial023.stdout:170: finishing write tid 1 to trial02342524-49 2026-02-25T08:42:19.915 INFO:tasks.rados.rados.0.trial023.stdout:172: done (1 left) 2026-02-25T08:42:19.915 INFO:tasks.rados.rados.0.trial023.stdout:173: read oid 26 snap -1 2026-02-25T08:42:19.916 INFO:tasks.rados.rados.0.trial023.stdout:173: expect (ObjNum 102 snap 10 seq_num 102) 2026-02-25T08:42:19.916 INFO:tasks.rados.rados.0.trial023.stdout:174: write oid 3 current snap is 12 2026-02-25T08:42:19.916 INFO:tasks.rados.rados.0.trial023.stdout:174: seq_num 112 ranges {682384=456830,1721042=423699,2809728=454936,3731357=151027} 2026-02-25T08:42:19.920 INFO:tasks.rados.rados.0.trial023.stdout:174: writing trial02342524-3 from 682384 to 1139214 tid 1 2026-02-25T08:42:19.923 INFO:tasks.rados.rados.0.trial023.stdout:174: writing trial02342524-3 from 1721042 to 2144741 tid 2 2026-02-25T08:42:19.928 INFO:tasks.rados.rados.0.trial023.stdout:174: writing trial02342524-3 from 2809728 to 3264664 tid 3 2026-02-25T08:42:19.931 INFO:tasks.rados.rados.0.trial023.stdout:174: writing trial02342524-3 from 3731357 to 3882384 tid 4 2026-02-25T08:42:19.931 INFO:tasks.rados.rados.0.trial023.stdout:175: write (excl) oid 37 current snap is 12 2026-02-25T08:42:19.931 INFO:tasks.rados.rados.0.trial023.stdout:170: finishing write tid 2 to trial02342524-49 2026-02-25T08:42:19.931 INFO:tasks.rados.rados.0.trial023.stdout:170: finishing write tid 3 to trial02342524-49 2026-02-25T08:42:19.931 INFO:tasks.rados.rados.0.trial023.stdout:175: seq_num 113 ranges {699191=532079,1779524=567357,2837151=666208} 2026-02-25T08:42:19.933 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:19.931+0000 7fde5b3b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:19.935 INFO:tasks.rados.rados.0.trial023.stdout:175: writing trial02342524-37 from 699191 to 1231270 tid 1 2026-02-25T08:42:19.937 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:19.936+0000 7fde5b3b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:19.939 INFO:tasks.rados.rados.0.trial023.stdout:175: writing trial02342524-37 from 1779524 to 2346881 tid 2 2026-02-25T08:42:19.944 INFO:tasks.rados.rados.0.trial023.stdout:175: writing trial02342524-37 from 2837151 to 3503359 tid 3 2026-02-25T08:42:19.944 INFO:tasks.rados.rados.0.trial023.stdout:176: delete oid 45 current snap is 12 2026-02-25T08:42:19.945 INFO:tasks.rados.rados.0.trial023.stdout:173: expect (ObjNum 102 snap 10 seq_num 102) 2026-02-25T08:42:19.945 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:42:19.958 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:42:19.991 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:19.992 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:20.010 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:42:20.089 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2082847 2026-02-25T08:42:20.132 INFO:tasks.rados.rados.0.trial023.stdout:174: finishing write tid 1 to trial02342524-3 2026-02-25T08:42:20.132 INFO:tasks.rados.rados.0.trial023.stdout:175: finishing write tid 1 to trial02342524-37 2026-02-25T08:42:20.132 INFO:tasks.rados.rados.0.trial023.stdout:174: finishing write tid 2 to trial02342524-3 2026-02-25T08:42:20.132 INFO:tasks.rados.rados.0.trial023.stdout:170: finishing write tid 4 to trial02342524-49 2026-02-25T08:42:20.132 INFO:tasks.rados.rados.0.trial023.stdout:170: finishing write tid 5 to trial02342524-49 2026-02-25T08:42:20.132 INFO:tasks.rados.rados.0.trial023.stdout:170: oid 49 updating version 0 to 23 2026-02-25T08:42:20.132 INFO:tasks.rados.rados.0.trial023.stdout:170: oid 49 updating version 23 to 25 2026-02-25T08:42:20.132 INFO:tasks.rados.rados.0.trial023.stdout:170: oid 49 version 25 is already newer than 25 2026-02-25T08:42:20.132 INFO:tasks.rados.rados.0.trial023.stdout:170: oid 49 updating version 25 to 28 2026-02-25T08:42:20.132 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 49 v 28 (ObjNum 109 snap 12 seq_num 109) dirty exists 2026-02-25T08:42:20.133 INFO:tasks.rados.rados.0.trial023.stdout:170: left oid 49 (ObjNum 109 snap 12 seq_num 109) 2026-02-25T08:42:20.133 INFO:tasks.rados.rados.0.trial023.stdout:174: finishing write tid 3 to trial02342524-3 2026-02-25T08:42:20.133 INFO:tasks.rados.rados.0.trial023.stdout:175: finishing write tid 2 to trial02342524-37 2026-02-25T08:42:20.133 INFO:tasks.rados.rados.0.trial023.stdout:170: done (4 left) 2026-02-25T08:42:20.133 INFO:tasks.rados.rados.0.trial023.stdout:173: done (3 left) 2026-02-25T08:42:20.133 INFO:tasks.rados.rados.0.trial023.stdout:176: done (2 left) 2026-02-25T08:42:20.133 INFO:tasks.rados.rados.0.trial023.stdout:177: read oid 20 snap -1 2026-02-25T08:42:20.133 INFO:tasks.rados.rados.0.trial023.stdout:177: expect (ObjNum 96 snap 10 seq_num 96) 2026-02-25T08:42:20.133 INFO:tasks.rados.rados.0.trial023.stdout:178: write (excl) oid 33 current snap is 12 2026-02-25T08:42:20.133 INFO:tasks.rados.rados.0.trial023.stdout:174: finishing write tid 4 to trial02342524-3 2026-02-25T08:42:20.133 INFO:tasks.rados.rados.0.trial023.stdout:174: finishing write tid 5 to trial02342524-3 2026-02-25T08:42:20.133 INFO:tasks.rados.rados.0.trial023.stdout:174: finishing write tid 6 to trial02342524-3 2026-02-25T08:42:20.134 INFO:tasks.rados.rados.0.trial023.stdout:174: oid 3 updating version 0 to 75 2026-02-25T08:42:20.134 INFO:tasks.rados.rados.0.trial023.stdout:174: oid 3 version 75 is already newer than 74 2026-02-25T08:42:20.134 INFO:tasks.rados.rados.0.trial023.stdout:174: oid 3 version 75 is already newer than 75 2026-02-25T08:42:20.134 INFO:tasks.rados.rados.0.trial023.stdout:174: oid 3 version 75 is already newer than 75 2026-02-25T08:42:20.134 INFO:tasks.rados.rados.0.trial023.stdout:174: oid 3 version 75 is already newer than 75 2026-02-25T08:42:20.134 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 3 v 75 (ObjNum 112 snap 12 seq_num 112) dirty exists 2026-02-25T08:42:20.134 INFO:tasks.rados.rados.0.trial023.stdout:174: left oid 3 (ObjNum 112 snap 12 seq_num 112) 2026-02-25T08:42:20.134 INFO:tasks.rados.rados.0.trial023.stdout:175: finishing write tid 3 to trial02342524-37 2026-02-25T08:42:20.134 INFO:tasks.rados.rados.0.trial023.stdout:175: finishing write tid 4 to trial02342524-37 2026-02-25T08:42:20.134 INFO:tasks.rados.rados.0.trial023.stdout:178: seq_num 115 ranges {732805=798930,2296488=571026,3291997=405105} 2026-02-25T08:42:20.137 INFO:tasks.rados.rados.0.trial023.stdout:178: writing trial02342524-33 from 732805 to 1531735 tid 1 2026-02-25T08:42:20.142 INFO:tasks.rados.rados.0.trial023.stdout:178: writing trial02342524-33 from 2296488 to 2867514 tid 2 2026-02-25T08:42:20.147 INFO:tasks.rados.rados.0.trial023.stdout:178: writing trial02342524-33 from 3291997 to 3697102 tid 3 2026-02-25T08:42:20.147 INFO:tasks.rados.rados.0.trial023.stdout:174: done (3 left) 2026-02-25T08:42:20.147 INFO:tasks.rados.rados.0.trial023.stdout:179: write oid 29 current snap is 12 2026-02-25T08:42:20.147 INFO:tasks.rados.rados.0.trial023.stdout:175: finishing write tid 5 to trial02342524-37 2026-02-25T08:42:20.147 INFO:tasks.rados.rados.0.trial023.stdout:175: oid 37 updating version 0 to 48 2026-02-25T08:42:20.147 INFO:tasks.rados.rados.0.trial023.stdout:175: oid 37 version 48 is already newer than 48 2026-02-25T08:42:20.147 INFO:tasks.rados.rados.0.trial023.stdout:175: oid 37 version 48 is already newer than 48 2026-02-25T08:42:20.147 INFO:tasks.rados.rados.0.trial023.stdout:175: oid 37 version 48 is already newer than 45 2026-02-25T08:42:20.147 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 37 v 48 (ObjNum 113 snap 12 seq_num 113) dirty exists 2026-02-25T08:42:20.147 INFO:tasks.rados.rados.0.trial023.stdout:175: left oid 37 (ObjNum 113 snap 12 seq_num 113) 2026-02-25T08:42:20.148 INFO:tasks.rados.rados.0.trial023.stdout:179: seq_num 116 ranges {749612=474179,1954970=431037,2919420=616377} 2026-02-25T08:42:20.151 INFO:tasks.rados.rados.0.trial023.stdout:179: writing trial02342524-29 from 749612 to 1223791 tid 1 2026-02-25T08:42:20.156 INFO:tasks.rados.rados.0.trial023.stdout:179: writing trial02342524-29 from 1954970 to 2386007 tid 2 2026-02-25T08:42:20.161 INFO:tasks.rados.rados.0.trial023.stdout:179: writing trial02342524-29 from 2919420 to 3535797 tid 3 2026-02-25T08:42:20.161 INFO:tasks.rados.rados.0.trial023.stdout:175: done (3 left) 2026-02-25T08:42:20.161 INFO:tasks.rados.rados.0.trial023.stdout:180: write (excl) oid 19 current snap is 12 2026-02-25T08:42:20.161 INFO:tasks.rados.rados.0.trial023.stdout:180: seq_num 117 ranges {766419=549428,2013452=574695,3346843=427649} 2026-02-25T08:42:20.165 INFO:tasks.rados.rados.0.trial023.stdout:180: writing trial02342524-19 from 766419 to 1315847 tid 1 2026-02-25T08:42:20.170 INFO:tasks.rados.rados.0.trial023.stdout:180: writing trial02342524-19 from 2013452 to 2588147 tid 2 2026-02-25T08:42:20.175 INFO:tasks.rados.rados.0.trial023.stdout:180: writing trial02342524-19 from 3346843 to 3774492 tid 3 2026-02-25T08:42:20.175 INFO:tasks.rados.rados.0.trial023.stdout:181: read oid 10 snap -1 2026-02-25T08:42:20.175 INFO:tasks.rados.rados.0.trial023.stdout:181: expect deleted 2026-02-25T08:42:20.175 INFO:tasks.rados.rados.0.trial023.stdout:182: delete oid 6 current snap is 12 2026-02-25T08:42:20.176 INFO:tasks.rados.rados.0.trial023.stdout:177: expect (ObjNum 96 snap 10 seq_num 96) 2026-02-25T08:42:20.182 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2480621 2026-02-25T08:42:20.224 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:42:20.256 INFO:tasks.rados.rados.0.trial023.stdout:178: finishing write tid 1 to trial02342524-33 2026-02-25T08:42:20.256 INFO:tasks.rados.rados.0.trial023.stdout:177: done (5 left) 2026-02-25T08:42:20.256 INFO:tasks.rados.rados.0.trial023.stdout:182: done (4 left) 2026-02-25T08:42:20.256 INFO:tasks.rados.rados.0.trial023.stdout:183: read oid 11 snap -1 2026-02-25T08:42:20.256 INFO:tasks.rados.rados.0.trial023.stdout:183: expect deleted 2026-02-25T08:42:20.256 INFO:tasks.rados.rados.0.trial023.stdout:184: delete oid 2 current snap is 12 2026-02-25T08:42:20.256 INFO:tasks.rados.rados.0.trial023.stdout:179: finishing write tid 1 to trial02342524-29 2026-02-25T08:42:20.256 INFO:tasks.rados.rados.0.trial023.stdout:179: finishing write tid 2 to trial02342524-29 2026-02-25T08:42:20.257 INFO:tasks.rados.rados.0.trial023.stdout:179: finishing write tid 3 to trial02342524-29 2026-02-25T08:42:20.257 INFO:tasks.rados.rados.0.trial023.stdout:180: finishing write tid 1 to trial02342524-19 2026-02-25T08:42:20.257 INFO:tasks.rados.rados.0.trial023.stdout:179: finishing write tid 4 to trial02342524-29 2026-02-25T08:42:20.257 INFO:tasks.rados.rados.0.trial023.stdout:179: finishing write tid 5 to trial02342524-29 2026-02-25T08:42:20.257 INFO:tasks.rados.rados.0.trial023.stdout:179: oid 29 updating version 0 to 52 2026-02-25T08:42:20.257 INFO:tasks.rados.rados.0.trial023.stdout:179: oid 29 version 52 is already newer than 52 2026-02-25T08:42:20.257 INFO:tasks.rados.rados.0.trial023.stdout:179: oid 29 version 52 is already newer than 52 2026-02-25T08:42:20.257 INFO:tasks.rados.rados.0.trial023.stdout:179: oid 29 version 52 is already newer than 52 2026-02-25T08:42:20.257 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 29 v 52 (ObjNum 116 snap 12 seq_num 116) dirty exists 2026-02-25T08:42:20.257 INFO:tasks.rados.rados.0.trial023.stdout:179: left oid 29 (ObjNum 116 snap 12 seq_num 116) 2026-02-25T08:42:20.257 INFO:tasks.rados.rados.0.trial023.stdout:178: finishing write tid 2 to trial02342524-33 2026-02-25T08:42:20.257 INFO:tasks.rados.rados.0.trial023.stdout:178: finishing write tid 3 to trial02342524-33 2026-02-25T08:42:20.257 INFO:tasks.rados.rados.0.trial023.stdout:180: finishing write tid 2 to trial02342524-19 2026-02-25T08:42:20.258 INFO:tasks.rados.rados.0.trial023.stdout:178: finishing write tid 4 to trial02342524-33 2026-02-25T08:42:20.258 INFO:tasks.rados.rados.0.trial023.stdout:180: finishing write tid 3 to trial02342524-19 2026-02-25T08:42:20.258 INFO:tasks.rados.rados.0.trial023.stdout:178: finishing write tid 5 to trial02342524-33 2026-02-25T08:42:20.258 INFO:tasks.rados.rados.0.trial023.stdout:178: oid 33 updating version 0 to 12 2026-02-25T08:42:20.258 INFO:tasks.rados.rados.0.trial023.stdout:178: oid 33 version 12 is already newer than 9 2026-02-25T08:42:20.258 INFO:tasks.rados.rados.0.trial023.stdout:178: oid 33 version 12 is already newer than 12 2026-02-25T08:42:20.258 INFO:tasks.rados.rados.0.trial023.stdout:178: oid 33 version 12 is already newer than 12 2026-02-25T08:42:20.258 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 33 v 12 (ObjNum 115 snap 12 seq_num 115) dirty exists 2026-02-25T08:42:20.258 INFO:tasks.rados.rados.0.trial023.stdout:178: left oid 33 (ObjNum 115 snap 12 seq_num 115) 2026-02-25T08:42:20.258 INFO:tasks.rados.rados.0.trial023.stdout:180: finishing write tid 4 to trial02342524-19 2026-02-25T08:42:20.258 INFO:tasks.rados.rados.0.trial023.stdout:180: finishing write tid 5 to trial02342524-19 2026-02-25T08:42:20.258 INFO:tasks.rados.rados.0.trial023.stdout:180: oid 19 updating version 0 to 43 2026-02-25T08:42:20.258 INFO:tasks.rados.rados.0.trial023.stdout:180: oid 19 version 43 is already newer than 43 2026-02-25T08:42:20.259 INFO:tasks.rados.rados.0.trial023.stdout:180: oid 19 version 43 is already newer than 43 2026-02-25T08:42:20.259 INFO:tasks.rados.rados.0.trial023.stdout:180: oid 19 version 43 is already newer than 40 2026-02-25T08:42:20.259 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 19 v 43 (ObjNum 117 snap 12 seq_num 117) dirty exists 2026-02-25T08:42:20.259 INFO:tasks.rados.rados.0.trial023.stdout:180: left oid 19 (ObjNum 117 snap 12 seq_num 117) 2026-02-25T08:42:20.269 INFO:tasks.rados.rados.0.trial023.stdout:178: done (5 left) 2026-02-25T08:42:20.270 INFO:tasks.rados.rados.0.trial023.stdout:179: done (4 left) 2026-02-25T08:42:20.270 INFO:tasks.rados.rados.0.trial023.stdout:180: done (3 left) 2026-02-25T08:42:20.270 INFO:tasks.rados.rados.0.trial023.stdout:181: done (2 left) 2026-02-25T08:42:20.270 INFO:tasks.rados.rados.0.trial023.stdout:183: done (1 left) 2026-02-25T08:42:20.270 INFO:tasks.rados.rados.0.trial023.stdout:184: done (0 left) 2026-02-25T08:42:20.270 INFO:tasks.rados.rados.0.trial023.stdout:185: read oid 24 snap -1 2026-02-25T08:42:20.270 INFO:tasks.rados.rados.0.trial023.stdout:185: expect (ObjNum 108 snap 12 seq_num 108) 2026-02-25T08:42:20.270 INFO:tasks.rados.rados.0.trial023.stdout:186: snap_create 2026-02-25T08:42:20.289 INFO:tasks.rados.rados.0.trial023.stdout:185: expect (ObjNum 108 snap 12 seq_num 108) 2026-02-25T08:42:20.869 INFO:tasks.rados.rados.0.trial023.stdout:185: done (1 left) 2026-02-25T08:42:20.869 INFO:tasks.rados.rados.0.trial023.stdout:186: done (0 left) 2026-02-25T08:42:20.869 INFO:tasks.rados.rados.0.trial023.stdout:187: read oid 36 snap -1 2026-02-25T08:42:20.869 INFO:tasks.rados.rados.0.trial023.stdout:187: expect deleted 2026-02-25T08:42:20.870 INFO:tasks.rados.rados.0.trial023.stdout:188: write oid 35 current snap is 13 2026-02-25T08:42:20.870 INFO:tasks.rados.rados.0.trial023.stdout:188: seq_num 120 ranges {416840=775175,1672545=344295} 2026-02-25T08:42:20.873 INFO:tasks.rados.rados.0.trial023.stdout:188: writing trial02342524-35 from 416840 to 1192015 tid 1 2026-02-25T08:42:20.877 INFO:tasks.rados.rados.0.trial023.stdout:188: writing trial02342524-35 from 1672545 to 2016840 tid 2 2026-02-25T08:42:20.877 INFO:tasks.rados.rados.0.trial023.stdout:189: read oid 34 snap -1 2026-02-25T08:42:20.877 INFO:tasks.rados.rados.0.trial023.stdout:189: expect (ObjNum 104 snap 12 seq_num 104) 2026-02-25T08:42:20.877 INFO:tasks.rados.rados.0.trial023.stdout:190: read oid 29 snap -1 2026-02-25T08:42:20.877 INFO:tasks.rados.rados.0.trial023.stdout:190: expect (ObjNum 116 snap 12 seq_num 116) 2026-02-25T08:42:20.877 INFO:tasks.rados.rados.0.trial023.stdout:191: read oid 26 snap -1 2026-02-25T08:42:20.877 INFO:tasks.rados.rados.0.trial023.stdout:191: expect (ObjNum 102 snap 10 seq_num 102) 2026-02-25T08:42:20.877 INFO:tasks.rados.rados.0.trial023.stdout:192: read oid 21 snap -1 2026-02-25T08:42:20.877 INFO:tasks.rados.rados.0.trial023.stdout:192: expect (ObjNum 50 snap 0 seq_num 50) 2026-02-25T08:42:20.877 INFO:tasks.rados.rados.0.trial023.stdout:193: snap_create 2026-02-25T08:42:20.898 INFO:tasks.rados.rados.0.trial023.stdout:192: expect (ObjNum 50 snap 0 seq_num 50) 2026-02-25T08:42:20.898 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:42:20.969 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1915529 2026-02-25T08:42:20.984 INFO:tasks.rados.rados.0.trial023.stdout:189: expect (ObjNum 104 snap 12 seq_num 104) 2026-02-25T08:42:21.103 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2357571 2026-02-25T08:42:21.120 INFO:tasks.rados.rados.0.trial023.stdout:190: expect (ObjNum 116 snap 12 seq_num 116) 2026-02-25T08:42:21.120 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 346323 2026-02-25T08:42:21.222 INFO:tasks.rados.rados.0.trial023.stdout:188: finishing write tid 1 to trial02342524-35 2026-02-25T08:42:21.222 INFO:tasks.rados.rados.0.trial023.stdout:191: expect (ObjNum 102 snap 10 seq_num 102) 2026-02-25T08:42:21.288 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:42:21.410 INFO:tasks.rados.rados.0.trial023.stdout:188: finishing write tid 2 to trial02342524-35 2026-02-25T08:42:21.410 INFO:tasks.rados.rados.0.trial023.stdout:188: finishing write tid 3 to trial02342524-35 2026-02-25T08:42:21.410 INFO:tasks.rados.rados.0.trial023.stdout:188: finishing write tid 4 to trial02342524-35 2026-02-25T08:42:21.410 INFO:tasks.rados.rados.0.trial023.stdout:188: oid 35 updating version 0 to 46 2026-02-25T08:42:21.410 INFO:tasks.rados.rados.0.trial023.stdout:188: oid 35 version 46 is already newer than 46 2026-02-25T08:42:21.410 INFO:tasks.rados.rados.0.trial023.stdout:188: oid 35 version 46 is already newer than 46 2026-02-25T08:42:21.411 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 35 v 46 (ObjNum 120 snap 13 seq_num 120) dirty exists 2026-02-25T08:42:21.411 INFO:tasks.rados.rados.0.trial023.stdout:188: left oid 35 (ObjNum 120 snap 13 seq_num 120) 2026-02-25T08:42:21.871 INFO:tasks.rados.rados.0.trial023.stdout:187: done (6 left) 2026-02-25T08:42:21.871 INFO:tasks.rados.rados.0.trial023.stdout:188: done (5 left) 2026-02-25T08:42:21.871 INFO:tasks.rados.rados.0.trial023.stdout:189: done (4 left) 2026-02-25T08:42:21.872 INFO:tasks.rados.rados.0.trial023.stdout:190: done (3 left) 2026-02-25T08:42:21.872 INFO:tasks.rados.rados.0.trial023.stdout:191: done (2 left) 2026-02-25T08:42:21.872 INFO:tasks.rados.rados.0.trial023.stdout:192: done (1 left) 2026-02-25T08:42:21.872 INFO:tasks.rados.rados.0.trial023.stdout:193: done (0 left) 2026-02-25T08:42:21.872 INFO:tasks.rados.rados.0.trial023.stdout:194: read oid 44 snap -1 2026-02-25T08:42:21.872 INFO:tasks.rados.rados.0.trial023.stdout:194: expect (ObjNum 43 snap 0 seq_num 43) 2026-02-25T08:42:21.872 INFO:tasks.rados.rados.0.trial023.stdout:195: snap_remove snap 13 2026-02-25T08:42:21.993 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:42:22.070 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:22.067+0000 7f245ca5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:22.072 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:22.071+0000 7f245ca5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:22.088 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:42:22.115 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:22.116 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:22.242 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:42:22.313 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:42:22.316 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:42:22.316 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:42:22.876 INFO:tasks.rados.rados.0.trial023.stdout:195: done (1 left) 2026-02-25T08:42:22.876 INFO:tasks.rados.rados.0.trial023.stdout:196: write (excl) oid 17 current snap is 14 2026-02-25T08:42:22.876 INFO:tasks.rados.rados.0.trial023.stdout:196: seq_num 121 ranges {433647=450424,1331027=582033} 2026-02-25T08:42:22.879 INFO:tasks.rados.rados.0.trial023.stdout:196: writing trial02342524-17 from 433647 to 884071 tid 1 2026-02-25T08:42:22.883 INFO:tasks.rados.rados.0.trial023.stdout:196: writing trial02342524-17 from 1331027 to 1913060 tid 2 2026-02-25T08:42:22.883 INFO:tasks.rados.rados.0.trial023.stdout:197: delete oid 46 current snap is 14 2026-02-25T08:42:22.883 INFO:tasks.rados.rados.0.trial023.stdout:194: expect (ObjNum 43 snap 0 seq_num 43) 2026-02-25T08:42:22.893 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 621680 2026-02-25T08:42:22.912 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 130767 2026-02-25T08:42:22.934 INFO:tasks.rados.rados.0.trial023.stdout:194: done (2 left) 2026-02-25T08:42:22.934 INFO:tasks.rados.rados.0.trial023.stdout:197: done (1 left) 2026-02-25T08:42:22.934 INFO:tasks.rados.rados.0.trial023.stdout:198: snap_remove snap 12 2026-02-25T08:42:23.879 INFO:tasks.rados.rados.0.trial023.stdout:198: done (1 left) 2026-02-25T08:42:23.879 INFO:tasks.rados.rados.0.trial023.stdout:199: delete oid 6 current snap is 14 2026-02-25T08:42:23.879 INFO:tasks.rados.rados.0.trial023.stdout:196: finishing write tid 1 to trial02342524-17 2026-02-25T08:42:23.879 INFO:tasks.rados.rados.0.trial023.stdout:196: finishing write tid 2 to trial02342524-17 2026-02-25T08:42:23.879 INFO:tasks.rados.rados.0.trial023.stdout:196: finishing write tid 3 to trial02342524-17 2026-02-25T08:42:23.879 INFO:tasks.rados.rados.0.trial023.stdout:196: finishing write tid 4 to trial02342524-17 2026-02-25T08:42:23.879 INFO:tasks.rados.rados.0.trial023.stdout:196: oid 17 updating version 0 to 7 2026-02-25T08:42:23.879 INFO:tasks.rados.rados.0.trial023.stdout:196: oid 17 version 7 is already newer than 5 2026-02-25T08:42:23.879 INFO:tasks.rados.rados.0.trial023.stdout:196: oid 17 version 7 is already newer than 7 2026-02-25T08:42:23.880 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 17 v 7 (ObjNum 121 snap 14 seq_num 121) dirty exists 2026-02-25T08:42:23.880 INFO:tasks.rados.rados.0.trial023.stdout:196: left oid 17 (ObjNum 121 snap 14 seq_num 121) 2026-02-25T08:42:23.902 INFO:tasks.rados.rados.0.trial023.stdout:196: done (1 left) 2026-02-25T08:42:23.903 INFO:tasks.rados.rados.0.trial023.stdout:199: done (0 left) 2026-02-25T08:42:23.903 INFO:tasks.rados.rados.0.trial023.stdout:200: write (excl) oid 13 current snap is 14 2026-02-25T08:42:23.903 INFO:tasks.rados.rados.0.trial023.stdout:200: seq_num 124 ranges {484068=792524,1906473=177595} 2026-02-25T08:42:23.908 INFO:tasks.rados.rados.0.trial023.stdout:200: writing trial02342524-13 from 484068 to 1276592 tid 1 2026-02-25T08:42:23.911 INFO:tasks.rados.rados.0.trial023.stdout:200: writing trial02342524-13 from 1906473 to 2084068 tid 2 2026-02-25T08:42:23.911 INFO:tasks.rados.rados.0.trial023.stdout:201: write oid 29 current snap is 14 2026-02-25T08:42:23.911 INFO:tasks.rados.rados.0.trial023.stdout:201: seq_num 125 ranges {500875=467773,1564955=535920} 2026-02-25T08:42:23.914 INFO:tasks.rados.rados.0.trial023.stdout:201: writing trial02342524-29 from 500875 to 968648 tid 1 2026-02-25T08:42:23.919 INFO:tasks.rados.rados.0.trial023.stdout:201: writing trial02342524-29 from 1564955 to 2100875 tid 2 2026-02-25T08:42:23.919 INFO:tasks.rados.rados.0.trial023.stdout:202: snap_create 2026-02-25T08:42:23.924 INFO:tasks.rados.rados.0.trial023.stdout:200: finishing write tid 1 to trial02342524-13 2026-02-25T08:42:23.939 INFO:tasks.rados.rados.0.trial023.stdout:200: finishing write tid 2 to trial02342524-13 2026-02-25T08:42:23.943 INFO:tasks.rados.rados.0.trial023.stdout:200: finishing write tid 3 to trial02342524-13 2026-02-25T08:42:23.946 INFO:tasks.rados.rados.0.trial023.stdout:200: finishing write tid 4 to trial02342524-13 2026-02-25T08:42:23.946 INFO:tasks.rados.rados.0.trial023.stdout:200: oid 13 updating version 0 to 55 2026-02-25T08:42:23.946 INFO:tasks.rados.rados.0.trial023.stdout:200: oid 13 version 55 is already newer than 53 2026-02-25T08:42:23.946 INFO:tasks.rados.rados.0.trial023.stdout:200: oid 13 version 55 is already newer than 55 2026-02-25T08:42:23.946 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 13 v 55 (ObjNum 124 snap 14 seq_num 124) dirty exists 2026-02-25T08:42:23.946 INFO:tasks.rados.rados.0.trial023.stdout:200: left oid 13 (ObjNum 124 snap 14 seq_num 124) 2026-02-25T08:42:23.955 INFO:tasks.rados.rados.0.trial023.stdout:201: finishing write tid 1 to trial02342524-29 2026-02-25T08:42:23.957 INFO:tasks.rados.rados.0.trial023.stdout:201: finishing write tid 2 to trial02342524-29 2026-02-25T08:42:23.962 INFO:tasks.rados.rados.0.trial023.stdout:201: finishing write tid 3 to trial02342524-29 2026-02-25T08:42:23.964 INFO:tasks.rados.rados.0.trial023.stdout:201: finishing write tid 4 to trial02342524-29 2026-02-25T08:42:23.964 INFO:tasks.rados.rados.0.trial023.stdout:201: oid 29 updating version 0 to 58 2026-02-25T08:42:23.964 INFO:tasks.rados.rados.0.trial023.stdout:201: oid 29 version 58 is already newer than 58 2026-02-25T08:42:23.964 INFO:tasks.rados.rados.0.trial023.stdout:201: oid 29 version 58 is already newer than 58 2026-02-25T08:42:23.964 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 29 v 58 (ObjNum 125 snap 14 seq_num 125) dirty exists 2026-02-25T08:42:23.964 INFO:tasks.rados.rados.0.trial023.stdout:201: left oid 29 (ObjNum 125 snap 14 seq_num 125) 2026-02-25T08:42:24.116 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:42:24.194 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:24.192+0000 7f6f62c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:24.197 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:24.196+0000 7f6f62c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:24.212 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:42:24.238 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:24.239 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:24.451 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~44s 2026-02-25T08:42:24.884 INFO:tasks.rados.rados.0.trial023.stdout:200: done (2 left) 2026-02-25T08:42:24.884 INFO:tasks.rados.rados.0.trial023.stdout:201: done (1 left) 2026-02-25T08:42:24.884 INFO:tasks.rados.rados.0.trial023.stdout:202: done (0 left) 2026-02-25T08:42:24.884 INFO:tasks.rados.rados.0.trial023.stdout:203: write (excl) oid 45 current snap is 15 2026-02-25T08:42:24.884 INFO:tasks.rados.rados.0.trial023.stdout:203: seq_num 126 ranges {517682=543022,1623437=494245} 2026-02-25T08:42:24.887 INFO:tasks.rados.rados.0.trial023.stdout:203: writing trial02342524-45 from 517682 to 1060704 tid 1 2026-02-25T08:42:24.892 INFO:tasks.rados.rados.0.trial023.stdout:203: writing trial02342524-45 from 1623437 to 2117682 tid 2 2026-02-25T08:42:24.892 INFO:tasks.rados.rados.0.trial023.stdout:204: snap_remove snap 14 2026-02-25T08:42:25.884 INFO:tasks.rados.rados.0.trial023.stdout:204: done (1 left) 2026-02-25T08:42:25.885 INFO:tasks.rados.rados.0.trial023.stdout:205: write (excl) oid 9 current snap is 15 2026-02-25T08:42:25.885 INFO:tasks.rados.rados.0.trial023.stdout:203: finishing write tid 1 to trial02342524-45 2026-02-25T08:42:25.885 INFO:tasks.rados.rados.0.trial023.stdout:205: seq_num 127 ranges {534489=618271,1565566=568923} 2026-02-25T08:42:25.888 INFO:tasks.rados.rados.0.trial023.stdout:205: writing trial02342524-9 from 534489 to 1152760 tid 1 2026-02-25T08:42:25.893 INFO:tasks.rados.rados.0.trial023.stdout:205: writing trial02342524-9 from 1565566 to 2134489 tid 2 2026-02-25T08:42:25.893 INFO:tasks.rados.rados.0.trial023.stdout:206: write (excl) oid 20 current snap is 15 2026-02-25T08:42:25.893 INFO:tasks.rados.rados.0.trial023.stdout:203: finishing write tid 2 to trial02342524-45 2026-02-25T08:42:25.893 INFO:tasks.rados.rados.0.trial023.stdout:203: finishing write tid 3 to trial02342524-45 2026-02-25T08:42:25.893 INFO:tasks.rados.rados.0.trial023.stdout:203: finishing write tid 4 to trial02342524-45 2026-02-25T08:42:25.893 INFO:tasks.rados.rados.0.trial023.stdout:203: oid 45 updating version 0 to 79 2026-02-25T08:42:25.893 INFO:tasks.rados.rados.0.trial023.stdout:203: oid 45 version 79 is already newer than 79 2026-02-25T08:42:25.893 INFO:tasks.rados.rados.0.trial023.stdout:203: oid 45 version 79 is already newer than 77 2026-02-25T08:42:25.894 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 45 v 79 (ObjNum 126 snap 15 seq_num 126) dirty exists 2026-02-25T08:42:25.894 INFO:tasks.rados.rados.0.trial023.stdout:203: left oid 45 (ObjNum 126 snap 15 seq_num 126) 2026-02-25T08:42:25.894 INFO:tasks.rados.rados.0.trial023.stdout:206: seq_num 128 ranges {551296=693520,2024048=127248} 2026-02-25T08:42:25.897 INFO:tasks.rados.rados.0.trial023.stdout:206: writing trial02342524-20 from 551296 to 1244816 tid 1 2026-02-25T08:42:25.901 INFO:tasks.rados.rados.0.trial023.stdout:206: writing trial02342524-20 from 2024048 to 2151296 tid 2 2026-02-25T08:42:25.901 INFO:tasks.rados.rados.0.trial023.stdout:203: done (2 left) 2026-02-25T08:42:25.901 INFO:tasks.rados.rados.0.trial023.stdout:207: snap_create 2026-02-25T08:42:25.916 INFO:tasks.rados.rados.0.trial023.stdout:206: finishing write tid 1 to trial02342524-20 2026-02-25T08:42:25.930 INFO:tasks.rados.rados.0.trial023.stdout:206: finishing write tid 2 to trial02342524-20 2026-02-25T08:42:25.941 INFO:tasks.rados.rados.0.trial023.stdout:206: finishing write tid 3 to trial02342524-20 2026-02-25T08:42:25.944 INFO:tasks.rados.rados.0.trial023.stdout:206: finishing write tid 4 to trial02342524-20 2026-02-25T08:42:25.944 INFO:tasks.rados.rados.0.trial023.stdout:206: oid 20 updating version 0 to 80 2026-02-25T08:42:25.944 INFO:tasks.rados.rados.0.trial023.stdout:206: oid 20 updating version 80 to 82 2026-02-25T08:42:25.944 INFO:tasks.rados.rados.0.trial023.stdout:206: oid 20 version 82 is already newer than 82 2026-02-25T08:42:25.944 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 20 v 82 (ObjNum 128 snap 15 seq_num 128) dirty exists 2026-02-25T08:42:25.945 INFO:tasks.rados.rados.0.trial023.stdout:206: left oid 20 (ObjNum 128 snap 15 seq_num 128) 2026-02-25T08:42:25.951 INFO:tasks.rados.rados.0.trial023.stdout:205: finishing write tid 1 to trial02342524-9 2026-02-25T08:42:25.965 INFO:tasks.rados.rados.0.trial023.stdout:205: finishing write tid 2 to trial02342524-9 2026-02-25T08:42:25.975 INFO:tasks.rados.rados.0.trial023.stdout:205: finishing write tid 3 to trial02342524-9 2026-02-25T08:42:25.977 INFO:tasks.rados.rados.0.trial023.stdout:205: finishing write tid 4 to trial02342524-9 2026-02-25T08:42:25.977 INFO:tasks.rados.rados.0.trial023.stdout:205: oid 9 updating version 0 to 55 2026-02-25T08:42:25.977 INFO:tasks.rados.rados.0.trial023.stdout:205: oid 9 version 55 is already newer than 53 2026-02-25T08:42:25.977 INFO:tasks.rados.rados.0.trial023.stdout:205: oid 9 version 55 is already newer than 55 2026-02-25T08:42:25.977 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 9 v 55 (ObjNum 127 snap 15 seq_num 127) dirty exists 2026-02-25T08:42:25.977 INFO:tasks.rados.rados.0.trial023.stdout:205: left oid 9 (ObjNum 127 snap 15 seq_num 127) 2026-02-25T08:42:26.241 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:42:26.316 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:26.314+0000 7ff33b4c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:26.318 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:26.317+0000 7ff33b4c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:26.333 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:42:26.362 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:26.363 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:26.890 INFO:tasks.rados.rados.0.trial023.stdout:205: done (2 left) 2026-02-25T08:42:26.890 INFO:tasks.rados.rados.0.trial023.stdout:206: done (1 left) 2026-02-25T08:42:26.890 INFO:tasks.rados.rados.0.trial023.stdout:207: done (0 left) 2026-02-25T08:42:26.890 INFO:tasks.rados.rados.0.trial023.stdout:208: snap_create 2026-02-25T08:42:27.318 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:42:27.388 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:42:27.390 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:42:27.391 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:42:27.892 INFO:tasks.rados.rados.0.trial023.stdout:208: done (0 left) 2026-02-25T08:42:27.892 INFO:tasks.rados.rados.0.trial023.stdout:209: snap_remove snap 16 2026-02-25T08:42:28.364 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:42:28.440 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:28.438+0000 7f49d5d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:28.443 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:28.441+0000 7f49d5d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:28.457 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:42:28.484 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:28.484 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:28.896 INFO:tasks.rados.rados.0.trial023.stdout:209: done (0 left) 2026-02-25T08:42:28.896 INFO:tasks.rados.rados.0.trial023.stdout:210: write (excl) oid 9 current snap is 17 2026-02-25T08:42:28.897 INFO:tasks.rados.rados.0.trial023.stdout:210: seq_num 129 ranges {568103=768769,2082530=85573} 2026-02-25T08:42:28.901 INFO:tasks.rados.rados.0.trial023.stdout:210: writing trial02342524-9 from 568103 to 1336872 tid 1 2026-02-25T08:42:28.904 INFO:tasks.rados.rados.0.trial023.stdout:210: writing trial02342524-9 from 2082530 to 2168103 tid 2 2026-02-25T08:42:28.905 INFO:tasks.rados.rados.0.trial023.stdout:211: snap_create 2026-02-25T08:42:28.939 INFO:tasks.rados.rados.0.trial023.stdout:210: finishing write tid 1 to trial02342524-9 2026-02-25T08:42:28.951 INFO:tasks.rados.rados.0.trial023.stdout:210: finishing write tid 2 to trial02342524-9 2026-02-25T08:42:28.956 INFO:tasks.rados.rados.0.trial023.stdout:210: finishing write tid 3 to trial02342524-9 2026-02-25T08:42:28.957 INFO:tasks.rados.rados.0.trial023.stdout:210: finishing write tid 4 to trial02342524-9 2026-02-25T08:42:28.958 INFO:tasks.rados.rados.0.trial023.stdout:210: oid 9 updating version 0 to 59 2026-02-25T08:42:28.958 INFO:tasks.rados.rados.0.trial023.stdout:210: oid 9 version 59 is already newer than 59 2026-02-25T08:42:28.958 INFO:tasks.rados.rados.0.trial023.stdout:210: oid 9 version 59 is already newer than 57 2026-02-25T08:42:28.958 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 9 v 59 (ObjNum 129 snap 17 seq_num 129) dirty exists 2026-02-25T08:42:28.958 INFO:tasks.rados.rados.0.trial023.stdout:210: left oid 9 (ObjNum 129 snap 17 seq_num 129) 2026-02-25T08:42:29.900 INFO:tasks.rados.rados.0.trial023.stdout:210: done (1 left) 2026-02-25T08:42:29.900 INFO:tasks.rados.rados.0.trial023.stdout:211: done (0 left) 2026-02-25T08:42:29.901 INFO:tasks.rados.rados.0.trial023.stdout:212: read oid 33 snap -1 2026-02-25T08:42:29.901 INFO:tasks.rados.rados.0.trial023.stdout:212: expect (ObjNum 115 snap 12 seq_num 115) 2026-02-25T08:42:29.901 INFO:tasks.rados.rados.0.trial023.stdout:213: snap_remove snap 15 2026-02-25T08:42:29.954 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~50s 2026-02-25T08:42:30.486 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:42:30.563 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:30.561+0000 7f903d6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:30.565 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:30.564+0000 7f903d6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:30.581 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:42:30.608 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:30.609 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:30.906 INFO:tasks.rados.rados.0.trial023.stdout:213: done (1 left) 2026-02-25T08:42:30.906 INFO:tasks.rados.rados.0.trial023.stdout:214: write (excl) oid 31 current snap is 18 2026-02-25T08:42:30.906 INFO:tasks.rados.rados.0.trial023.stdout:214: seq_num 130 ranges {584910=560371,1857365=327545} 2026-02-25T08:42:30.910 INFO:tasks.rados.rados.0.trial023.stdout:214: writing trial02342524-31 from 584910 to 1145281 tid 1 2026-02-25T08:42:30.914 INFO:tasks.rados.rados.0.trial023.stdout:214: writing trial02342524-31 from 1857365 to 2184910 tid 2 2026-02-25T08:42:30.915 INFO:tasks.rados.rados.0.trial023.stdout:215: delete oid 28 current snap is 18 2026-02-25T08:42:30.915 INFO:tasks.rados.rados.0.trial023.stdout:212: expect (ObjNum 115 snap 12 seq_num 115) 2026-02-25T08:42:30.968 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:42:31.076 INFO:tasks.rados.rados.0.trial023.stdout:212: done (2 left) 2026-02-25T08:42:31.076 INFO:tasks.rados.rados.0.trial023.stdout:215: done (1 left) 2026-02-25T08:42:31.076 INFO:tasks.rados.rados.0.trial023.stdout:216: delete oid 42 current snap is 18 2026-02-25T08:42:31.076 INFO:tasks.rados.rados.0.trial023.stdout:214: finishing write tid 1 to trial02342524-31 2026-02-25T08:42:31.076 INFO:tasks.rados.rados.0.trial023.stdout:214: finishing write tid 2 to trial02342524-31 2026-02-25T08:42:31.076 INFO:tasks.rados.rados.0.trial023.stdout:214: finishing write tid 3 to trial02342524-31 2026-02-25T08:42:31.076 INFO:tasks.rados.rados.0.trial023.stdout:214: finishing write tid 4 to trial02342524-31 2026-02-25T08:42:31.077 INFO:tasks.rados.rados.0.trial023.stdout:214: oid 31 updating version 0 to 18 2026-02-25T08:42:31.077 INFO:tasks.rados.rados.0.trial023.stdout:214: oid 31 updating version 18 to 20 2026-02-25T08:42:31.077 INFO:tasks.rados.rados.0.trial023.stdout:214: oid 31 version 20 is already newer than 20 2026-02-25T08:42:31.077 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 31 v 20 (ObjNum 130 snap 18 seq_num 130) dirty exists 2026-02-25T08:42:31.077 INFO:tasks.rados.rados.0.trial023.stdout:214: left oid 31 (ObjNum 130 snap 18 seq_num 130) 2026-02-25T08:42:31.099 INFO:tasks.rados.rados.0.trial023.stdout:214: done (1 left) 2026-02-25T08:42:31.100 INFO:tasks.rados.rados.0.trial023.stdout:216: done (0 left) 2026-02-25T08:42:31.100 INFO:tasks.rados.rados.0.trial023.stdout:217: read oid 3 snap -1 2026-02-25T08:42:31.100 INFO:tasks.rados.rados.0.trial023.stdout:217: expect (ObjNum 112 snap 12 seq_num 112) 2026-02-25T08:42:31.100 INFO:tasks.rados.rados.0.trial023.stdout:218: write (excl) oid 37 current snap is 18 2026-02-25T08:42:31.100 INFO:tasks.rados.rados.0.trial023.stdout:218: seq_num 133 ranges {635331=786118,1916458=318873} 2026-02-25T08:42:31.105 INFO:tasks.rados.rados.0.trial023.stdout:218: writing trial02342524-37 from 635331 to 1421449 tid 1 2026-02-25T08:42:31.108 INFO:tasks.rados.rados.0.trial023.stdout:218: writing trial02342524-37 from 1916458 to 2235331 tid 2 2026-02-25T08:42:31.108 INFO:tasks.rados.rados.0.trial023.stdout:219: write (excl) oid 11 current snap is 18 2026-02-25T08:42:31.108 INFO:tasks.rados.rados.0.trial023.stdout:219: seq_num 134 ranges {652138=461367,1574940=677198} 2026-02-25T08:42:31.112 INFO:tasks.rados.rados.0.trial023.stdout:219: writing trial02342524-11 from 652138 to 1113505 tid 1 2026-02-25T08:42:31.117 INFO:tasks.rados.rados.0.trial023.stdout:219: writing trial02342524-11 from 1574940 to 2252138 tid 2 2026-02-25T08:42:31.117 INFO:tasks.rados.rados.0.trial023.stdout:220: snap_remove snap 17 2026-02-25T08:42:31.909 INFO:tasks.rados.rados.0.trial023.stdout:220: done (3 left) 2026-02-25T08:42:31.909 INFO:tasks.rados.rados.0.trial023.stdout:221: write (excl) oid 19 current snap is 18 2026-02-25T08:42:31.909 INFO:tasks.rados.rados.0.trial023.stdout:221: seq_num 135 ranges {668945=536616,1633422=491363} 2026-02-25T08:42:31.913 INFO:tasks.rados.rados.0.trial023.stdout:221: writing trial02342524-19 from 668945 to 1205561 tid 1 2026-02-25T08:42:31.917 INFO:tasks.rados.rados.0.trial023.stdout:221: writing trial02342524-19 from 1633422 to 2124785 tid 2 2026-02-25T08:42:31.917 INFO:tasks.rados.rados.0.trial023.stdout:222: write oid 23 current snap is 18 2026-02-25T08:42:31.917 INFO:tasks.rados.rados.0.trial023.stdout:222: seq_num 136 ranges {685752=611865,1975551=310201} 2026-02-25T08:42:31.922 INFO:tasks.rados.rados.0.trial023.stdout:222: writing trial02342524-23 from 685752 to 1297617 tid 1 2026-02-25T08:42:31.926 INFO:tasks.rados.rados.0.trial023.stdout:222: writing trial02342524-23 from 1975551 to 2285752 tid 2 2026-02-25T08:42:31.926 INFO:tasks.rados.rados.0.trial023.stdout:223: snap_create 2026-02-25T08:42:31.926 INFO:tasks.rados.rados.0.trial023.stdout:217: expect (ObjNum 112 snap 12 seq_num 112) 2026-02-25T08:42:31.940 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2021752 2026-02-25T08:42:32.027 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 3136295 2026-02-25T08:42:32.038 INFO:tasks.rados.rados.0.trial023.stdout:219: finishing write tid 1 to trial02342524-11 2026-02-25T08:42:32.038 INFO:tasks.rados.rados.0.trial023.stdout:218: finishing write tid 1 to trial02342524-37 2026-02-25T08:42:32.039 INFO:tasks.rados.rados.0.trial023.stdout:219: finishing write tid 2 to trial02342524-11 2026-02-25T08:42:32.039 INFO:tasks.rados.rados.0.trial023.stdout:219: finishing write tid 3 to trial02342524-11 2026-02-25T08:42:32.039 INFO:tasks.rados.rados.0.trial023.stdout:218: finishing write tid 2 to trial02342524-37 2026-02-25T08:42:32.039 INFO:tasks.rados.rados.0.trial023.stdout:219: finishing write tid 4 to trial02342524-11 2026-02-25T08:42:32.039 INFO:tasks.rados.rados.0.trial023.stdout:219: oid 11 updating version 0 to 85 2026-02-25T08:42:32.039 INFO:tasks.rados.rados.0.trial023.stdout:219: oid 11 version 85 is already newer than 85 2026-02-25T08:42:32.039 INFO:tasks.rados.rados.0.trial023.stdout:219: oid 11 version 85 is already newer than 83 2026-02-25T08:42:32.039 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 11 v 85 (ObjNum 134 snap 18 seq_num 134) dirty exists 2026-02-25T08:42:32.039 INFO:tasks.rados.rados.0.trial023.stdout:219: left oid 11 (ObjNum 134 snap 18 seq_num 134) 2026-02-25T08:42:32.039 INFO:tasks.rados.rados.0.trial023.stdout:218: finishing write tid 3 to trial02342524-37 2026-02-25T08:42:32.039 INFO:tasks.rados.rados.0.trial023.stdout:218: finishing write tid 4 to trial02342524-37 2026-02-25T08:42:32.039 INFO:tasks.rados.rados.0.trial023.stdout:218: oid 37 updating version 0 to 62 2026-02-25T08:42:32.039 INFO:tasks.rados.rados.0.trial023.stdout:218: oid 37 version 62 is already newer than 62 2026-02-25T08:42:32.039 INFO:tasks.rados.rados.0.trial023.stdout:218: oid 37 version 62 is already newer than 60 2026-02-25T08:42:32.039 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 37 v 62 (ObjNum 133 snap 18 seq_num 133) dirty exists 2026-02-25T08:42:32.039 INFO:tasks.rados.rados.0.trial023.stdout:218: left oid 37 (ObjNum 133 snap 18 seq_num 133) 2026-02-25T08:42:32.039 INFO:tasks.rados.rados.0.trial023.stdout:222: finishing write tid 1 to trial02342524-23 2026-02-25T08:42:32.039 INFO:tasks.rados.rados.0.trial023.stdout:221: finishing write tid 1 to trial02342524-19 2026-02-25T08:42:32.039 INFO:tasks.rados.rados.0.trial023.stdout:222: finishing write tid 2 to trial02342524-23 2026-02-25T08:42:32.039 INFO:tasks.rados.rados.0.trial023.stdout:222: finishing write tid 3 to trial02342524-23 2026-02-25T08:42:32.039 INFO:tasks.rados.rados.0.trial023.stdout:222: finishing write tid 4 to trial02342524-23 2026-02-25T08:42:32.039 INFO:tasks.rados.rados.0.trial023.stdout:222: oid 23 updating version 0 to 65 2026-02-25T08:42:32.039 INFO:tasks.rados.rados.0.trial023.stdout:222: oid 23 version 65 is already newer than 65 2026-02-25T08:42:32.039 INFO:tasks.rados.rados.0.trial023.stdout:222: oid 23 version 65 is already newer than 65 2026-02-25T08:42:32.039 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 23 v 65 (ObjNum 136 snap 18 seq_num 136) dirty exists 2026-02-25T08:42:32.039 INFO:tasks.rados.rados.0.trial023.stdout:222: left oid 23 (ObjNum 136 snap 18 seq_num 136) 2026-02-25T08:42:32.039 INFO:tasks.rados.rados.0.trial023.stdout:221: finishing write tid 2 to trial02342524-19 2026-02-25T08:42:32.039 INFO:tasks.rados.rados.0.trial023.stdout:221: finishing write tid 3 to trial02342524-19 2026-02-25T08:42:32.039 INFO:tasks.rados.rados.0.trial023.stdout:221: finishing write tid 4 to trial02342524-19 2026-02-25T08:42:32.039 INFO:tasks.rados.rados.0.trial023.stdout:221: oid 19 updating version 0 to 46 2026-02-25T08:42:32.039 INFO:tasks.rados.rados.0.trial023.stdout:221: oid 19 version 46 is already newer than 46 2026-02-25T08:42:32.039 INFO:tasks.rados.rados.0.trial023.stdout:221: oid 19 version 46 is already newer than 44 2026-02-25T08:42:32.039 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 19 v 46 (ObjNum 135 snap 18 seq_num 135) dirty exists 2026-02-25T08:42:32.039 INFO:tasks.rados.rados.0.trial023.stdout:221: left oid 19 (ObjNum 135 snap 18 seq_num 135) 2026-02-25T08:42:32.392 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:42:32.463 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:42:32.466 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:42:32.466 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:42:32.610 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:42:32.687 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:32.684+0000 7f9157174640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:32.690 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:32.688+0000 7f9157174640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:32.705 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:42:32.730 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:32.731 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:32.913 INFO:tasks.rados.rados.0.trial023.stdout:217: done (5 left) 2026-02-25T08:42:32.914 INFO:tasks.rados.rados.0.trial023.stdout:218: done (4 left) 2026-02-25T08:42:32.914 INFO:tasks.rados.rados.0.trial023.stdout:219: done (3 left) 2026-02-25T08:42:32.914 INFO:tasks.rados.rados.0.trial023.stdout:221: done (2 left) 2026-02-25T08:42:32.914 INFO:tasks.rados.rados.0.trial023.stdout:222: done (1 left) 2026-02-25T08:42:32.914 INFO:tasks.rados.rados.0.trial023.stdout:223: done (0 left) 2026-02-25T08:42:32.914 INFO:tasks.rados.rados.0.trial023.stdout:224: read oid 13 snap -1 2026-02-25T08:42:32.914 INFO:tasks.rados.rados.0.trial023.stdout:224: expect (ObjNum 124 snap 14 seq_num 124) 2026-02-25T08:42:32.914 INFO:tasks.rados.rados.0.trial023.stdout:225: delete oid 37 current snap is 19 2026-02-25T08:42:32.950 INFO:tasks.rados.rados.0.trial023.stdout:224: expect (ObjNum 124 snap 14 seq_num 124) 2026-02-25T08:42:32.969 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 308650 2026-02-25T08:42:33.002 INFO:tasks.rados.rados.0.trial023.stdout:224: done (1 left) 2026-02-25T08:42:33.003 INFO:tasks.rados.rados.0.trial023.stdout:225: done (0 left) 2026-02-25T08:42:33.003 INFO:tasks.rados.rados.0.trial023.stdout:226: read oid 31 snap 18 2026-02-25T08:42:33.003 INFO:tasks.rados.rados.0.trial023.stdout:226: expect (ObjNum 130 snap 18 seq_num 130) 2026-02-25T08:42:33.003 INFO:tasks.rados.rados.0.trial023.stdout:227: read oid 10 snap -1 2026-02-25T08:42:33.003 INFO:tasks.rados.rados.0.trial023.stdout:227: expect deleted 2026-02-25T08:42:33.003 INFO:tasks.rados.rados.0.trial023.stdout:228: write oid 45 current snap is 19 2026-02-25T08:42:33.003 INFO:tasks.rados.rados.0.trial023.stdout:228: seq_num 138 ranges {719366=478716,1808868=510498} 2026-02-25T08:42:33.007 INFO:tasks.rados.rados.0.trial023.stdout:228: writing trial02342524-45 from 719366 to 1198082 tid 1 2026-02-25T08:42:33.012 INFO:tasks.rados.rados.0.trial023.stdout:228: writing trial02342524-45 from 1808868 to 2319366 tid 2 2026-02-25T08:42:33.012 INFO:tasks.rados.rados.0.trial023.stdout:229: write (excl) oid 1 current snap is 19 2026-02-25T08:42:33.012 INFO:tasks.rados.rados.0.trial023.stdout:229: seq_num 139 ranges {736173=553965,1867350=468823} 2026-02-25T08:42:33.017 INFO:tasks.rados.rados.0.trial023.stdout:229: writing trial02342524-1 from 736173 to 1290138 tid 1 2026-02-25T08:42:33.022 INFO:tasks.rados.rados.0.trial023.stdout:229: writing trial02342524-1 from 1867350 to 2336173 tid 2 2026-02-25T08:42:33.022 INFO:tasks.rados.rados.0.trial023.stdout:230: write (excl) oid 38 current snap is 19 2026-02-25T08:42:33.022 INFO:tasks.rados.rados.0.trial023.stdout:230: seq_num 140 ranges {752980=629214,1809479=543501} 2026-02-25T08:42:33.027 INFO:tasks.rados.rados.0.trial023.stdout:230: writing trial02342524-38 from 752980 to 1382194 tid 1 2026-02-25T08:42:33.031 INFO:tasks.rados.rados.0.trial023.stdout:230: writing trial02342524-38 from 1809479 to 2352980 tid 2 2026-02-25T08:42:33.031 INFO:tasks.rados.rados.0.trial023.stdout:231: snap_create 2026-02-25T08:42:33.031 INFO:tasks.rados.rados.0.trial023.stdout:226: expect (ObjNum 130 snap 18 seq_num 130) 2026-02-25T08:42:33.031 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:42:33.056 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 483607 2026-02-25T08:42:33.093 INFO:tasks.rados.rados.0.trial023.stdout:228: finishing write tid 1 to trial02342524-45 2026-02-25T08:42:33.093 INFO:tasks.rados.rados.0.trial023.stdout:229: finishing write tid 1 to trial02342524-1 2026-02-25T08:42:33.093 INFO:tasks.rados.rados.0.trial023.stdout:228: finishing write tid 2 to trial02342524-45 2026-02-25T08:42:33.093 INFO:tasks.rados.rados.0.trial023.stdout:228: finishing write tid 3 to trial02342524-45 2026-02-25T08:42:33.093 INFO:tasks.rados.rados.0.trial023.stdout:228: finishing write tid 4 to trial02342524-45 2026-02-25T08:42:33.093 INFO:tasks.rados.rados.0.trial023.stdout:228: oid 45 updating version 0 to 89 2026-02-25T08:42:33.093 INFO:tasks.rados.rados.0.trial023.stdout:228: oid 45 version 89 is already newer than 89 2026-02-25T08:42:33.093 INFO:tasks.rados.rados.0.trial023.stdout:228: oid 45 version 89 is already newer than 89 2026-02-25T08:42:33.093 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 45 v 89 (ObjNum 138 snap 19 seq_num 138) dirty exists 2026-02-25T08:42:33.094 INFO:tasks.rados.rados.0.trial023.stdout:228: left oid 45 (ObjNum 138 snap 19 seq_num 138) 2026-02-25T08:42:33.094 INFO:tasks.rados.rados.0.trial023.stdout:230: finishing write tid 1 to trial02342524-38 2026-02-25T08:42:33.094 INFO:tasks.rados.rados.0.trial023.stdout:229: finishing write tid 2 to trial02342524-1 2026-02-25T08:42:33.094 INFO:tasks.rados.rados.0.trial023.stdout:229: finishing write tid 3 to trial02342524-1 2026-02-25T08:42:33.094 INFO:tasks.rados.rados.0.trial023.stdout:229: finishing write tid 4 to trial02342524-1 2026-02-25T08:42:33.094 INFO:tasks.rados.rados.0.trial023.stdout:229: oid 1 updating version 0 to 32 2026-02-25T08:42:33.094 INFO:tasks.rados.rados.0.trial023.stdout:229: oid 1 version 32 is already newer than 32 2026-02-25T08:42:33.094 INFO:tasks.rados.rados.0.trial023.stdout:229: oid 1 version 32 is already newer than 30 2026-02-25T08:42:33.094 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 1 v 32 (ObjNum 139 snap 19 seq_num 139) dirty exists 2026-02-25T08:42:33.094 INFO:tasks.rados.rados.0.trial023.stdout:229: left oid 1 (ObjNum 139 snap 19 seq_num 139) 2026-02-25T08:42:33.094 INFO:tasks.rados.rados.0.trial023.stdout:230: finishing write tid 2 to trial02342524-38 2026-02-25T08:42:33.094 INFO:tasks.rados.rados.0.trial023.stdout:230: finishing write tid 3 to trial02342524-38 2026-02-25T08:42:33.095 INFO:tasks.rados.rados.0.trial023.stdout:230: finishing write tid 4 to trial02342524-38 2026-02-25T08:42:33.095 INFO:tasks.rados.rados.0.trial023.stdout:230: oid 38 updating version 0 to 48 2026-02-25T08:42:33.095 INFO:tasks.rados.rados.0.trial023.stdout:230: oid 38 updating version 48 to 50 2026-02-25T08:42:33.095 INFO:tasks.rados.rados.0.trial023.stdout:230: oid 38 version 50 is already newer than 50 2026-02-25T08:42:33.095 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 38 v 50 (ObjNum 140 snap 19 seq_num 140) dirty exists 2026-02-25T08:42:33.095 INFO:tasks.rados.rados.0.trial023.stdout:230: left oid 38 (ObjNum 140 snap 19 seq_num 140) 2026-02-25T08:42:33.914 INFO:tasks.rados.rados.0.trial023.stdout:226: done (5 left) 2026-02-25T08:42:33.915 INFO:tasks.rados.rados.0.trial023.stdout:227: done (4 left) 2026-02-25T08:42:33.915 INFO:tasks.rados.rados.0.trial023.stdout:228: done (3 left) 2026-02-25T08:42:33.915 INFO:tasks.rados.rados.0.trial023.stdout:229: done (2 left) 2026-02-25T08:42:33.915 INFO:tasks.rados.rados.0.trial023.stdout:230: done (1 left) 2026-02-25T08:42:33.915 INFO:tasks.rados.rados.0.trial023.stdout:231: done (0 left) 2026-02-25T08:42:33.915 INFO:tasks.rados.rados.0.trial023.stdout:232: read oid 33 snap -1 2026-02-25T08:42:33.915 INFO:tasks.rados.rados.0.trial023.stdout:232: expect (ObjNum 115 snap 12 seq_num 115) 2026-02-25T08:42:33.915 INFO:tasks.rados.rados.0.trial023.stdout:233: delete oid 2 current snap is 20 2026-02-25T08:42:33.928 INFO:tasks.rados.rados.0.trial023.stdout:233: done (1 left) 2026-02-25T08:42:33.928 INFO:tasks.rados.rados.0.trial023.stdout:234: write (excl) oid 44 current snap is 20 2026-02-25T08:42:33.928 INFO:tasks.rados.rados.0.trial023.stdout:234: seq_num 142 ranges {786594=779712,2326443=60151} 2026-02-25T08:42:33.934 INFO:tasks.rados.rados.0.trial023.stdout:234: writing trial02342524-44 from 786594 to 1566306 tid 1 2026-02-25T08:42:33.937 INFO:tasks.rados.rados.0.trial023.stdout:234: writing trial02342524-44 from 2326443 to 2386594 tid 2 2026-02-25T08:42:33.937 INFO:tasks.rados.rados.0.trial023.stdout:235: snap_remove snap 19 2026-02-25T08:42:34.732 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:42:34.809 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:34.805+0000 7fa253ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:34.811 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:34.809+0000 7fa253ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:34.827 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:42:34.853 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:34.854 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:34.918 INFO:tasks.rados.rados.0.trial023.stdout:235: done (2 left) 2026-02-25T08:42:34.918 INFO:tasks.rados.rados.0.trial023.stdout:236: read oid 19 snap -1 2026-02-25T08:42:34.919 INFO:tasks.rados.rados.0.trial023.stdout:236: expect (ObjNum 135 snap 18 seq_num 135) 2026-02-25T08:42:34.919 INFO:tasks.rados.rados.0.trial023.stdout:237: read oid 39 snap -1 2026-02-25T08:42:34.919 INFO:tasks.rados.rados.0.trial023.stdout:237: expect (ObjNum 38 snap 0 seq_num 38) 2026-02-25T08:42:34.919 INFO:tasks.rados.rados.0.trial023.stdout:238: snap_create 2026-02-25T08:42:34.919 INFO:tasks.rados.rados.0.trial023.stdout:232: expect (ObjNum 115 snap 12 seq_num 115) 2026-02-25T08:42:34.919 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 169062 2026-02-25T08:42:34.971 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:42:35.077 INFO:tasks.rados.rados.0.trial023.stdout:234: finishing write tid 1 to trial02342524-44 2026-02-25T08:42:35.077 INFO:tasks.rados.rados.0.trial023.stdout:234: finishing write tid 2 to trial02342524-44 2026-02-25T08:42:35.078 INFO:tasks.rados.rados.0.trial023.stdout:234: finishing write tid 3 to trial02342524-44 2026-02-25T08:42:35.078 INFO:tasks.rados.rados.0.trial023.stdout:234: finishing write tid 4 to trial02342524-44 2026-02-25T08:42:35.078 INFO:tasks.rados.rados.0.trial023.stdout:234: oid 44 updating version 0 to 10 2026-02-25T08:42:35.078 INFO:tasks.rados.rados.0.trial023.stdout:234: oid 44 version 10 is already newer than 10 2026-02-25T08:42:35.078 INFO:tasks.rados.rados.0.trial023.stdout:234: oid 44 version 10 is already newer than 8 2026-02-25T08:42:35.078 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 44 v 10 (ObjNum 142 snap 20 seq_num 142) dirty exists 2026-02-25T08:42:35.078 INFO:tasks.rados.rados.0.trial023.stdout:234: left oid 44 (ObjNum 142 snap 20 seq_num 142) 2026-02-25T08:42:35.078 INFO:tasks.rados.rados.0.trial023.stdout:237: expect (ObjNum 38 snap 0 seq_num 38) 2026-02-25T08:42:35.144 INFO:tasks.rados.rados.0.trial023.stdout:236: expect (ObjNum 135 snap 18 seq_num 135) 2026-02-25T08:42:35.176 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:42:35.225 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1950652 2026-02-25T08:42:35.457 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~55s 2026-02-25T08:42:35.920 INFO:tasks.rados.rados.0.trial023.stdout:232: done (4 left) 2026-02-25T08:42:35.921 INFO:tasks.rados.rados.0.trial023.stdout:234: done (3 left) 2026-02-25T08:42:35.921 INFO:tasks.rados.rados.0.trial023.stdout:236: done (2 left) 2026-02-25T08:42:35.921 INFO:tasks.rados.rados.0.trial023.stdout:237: done (1 left) 2026-02-25T08:42:35.921 INFO:tasks.rados.rados.0.trial023.stdout:238: done (0 left) 2026-02-25T08:42:35.921 INFO:tasks.rados.rados.0.trial023.stdout:239: read oid 16 snap -1 2026-02-25T08:42:35.921 INFO:tasks.rados.rados.0.trial023.stdout:239: expect (ObjNum 90 snap 9 seq_num 90) 2026-02-25T08:42:35.921 INFO:tasks.rados.rados.0.trial023.stdout:240: delete oid 15 current snap is 21 2026-02-25T08:42:35.953 INFO:tasks.rados.rados.0.trial023.stdout:239: expect (ObjNum 90 snap 9 seq_num 90) 2026-02-25T08:42:35.953 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:42:36.006 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1405030 2026-02-25T08:42:36.044 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:42:36.093 INFO:tasks.rados.rados.0.trial023.stdout:239: done (1 left) 2026-02-25T08:42:36.093 INFO:tasks.rados.rados.0.trial023.stdout:240: done (0 left) 2026-02-25T08:42:36.093 INFO:tasks.rados.rados.0.trial023.stdout:241: write oid 13 current snap is 21 2026-02-25T08:42:36.093 INFO:tasks.rados.rados.0.trial023.stdout:241: seq_num 144 ranges {420208=530210,1527054=766050} 2026-02-25T08:42:36.096 INFO:tasks.rados.rados.0.trial023.stdout:241: writing trial02342524-13 from 420208 to 950418 tid 1 2026-02-25T08:42:36.101 INFO:tasks.rados.rados.0.trial023.stdout:241: writing trial02342524-13 from 1527054 to 2293104 tid 2 2026-02-25T08:42:36.102 INFO:tasks.rados.rados.0.trial023.stdout:242: snap_remove snap 18 2026-02-25T08:42:36.855 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:42:36.925 INFO:tasks.rados.rados.0.trial023.stdout:242: done (1 left) 2026-02-25T08:42:36.925 INFO:tasks.rados.rados.0.trial023.stdout:243: read oid 28 snap 20 2026-02-25T08:42:36.925 INFO:tasks.rados.rados.0.trial023.stdout:243: expect deleted 2026-02-25T08:42:36.925 INFO:tasks.rados.rados.0.trial023.stdout:244: snap_create 2026-02-25T08:42:36.925 INFO:tasks.rados.rados.0.trial023.stdout:241: finishing write tid 1 to trial02342524-13 2026-02-25T08:42:36.925 INFO:tasks.rados.rados.0.trial023.stdout:241: finishing write tid 2 to trial02342524-13 2026-02-25T08:42:36.925 INFO:tasks.rados.rados.0.trial023.stdout:241: finishing write tid 3 to trial02342524-13 2026-02-25T08:42:36.926 INFO:tasks.rados.rados.0.trial023.stdout:241: finishing write tid 4 to trial02342524-13 2026-02-25T08:42:36.926 INFO:tasks.rados.rados.0.trial023.stdout:241: oid 13 updating version 0 to 71 2026-02-25T08:42:36.926 INFO:tasks.rados.rados.0.trial023.stdout:241: oid 13 version 71 is already newer than 71 2026-02-25T08:42:36.926 INFO:tasks.rados.rados.0.trial023.stdout:241: oid 13 version 71 is already newer than 71 2026-02-25T08:42:36.926 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 13 v 71 (ObjNum 144 snap 21 seq_num 144) dirty exists 2026-02-25T08:42:36.926 INFO:tasks.rados.rados.0.trial023.stdout:241: left oid 13 (ObjNum 144 snap 21 seq_num 144) 2026-02-25T08:42:36.933 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:36.930+0000 7f6c3390b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:36.936 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:36.934+0000 7f6c3390b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:36.967 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:42:36.992 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:36.993 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:37.468 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:42:37.538 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:42:37.541 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:42:37.541 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:42:37.927 INFO:tasks.rados.rados.0.trial023.stdout:241: done (2 left) 2026-02-25T08:42:37.927 INFO:tasks.rados.rados.0.trial023.stdout:243: done (1 left) 2026-02-25T08:42:37.927 INFO:tasks.rados.rados.0.trial023.stdout:244: done (0 left) 2026-02-25T08:42:37.927 INFO:tasks.rados.rados.0.trial023.stdout:245: snap_remove snap 21 2026-02-25T08:42:38.932 INFO:tasks.rados.rados.0.trial023.stdout:245: done (0 left) 2026-02-25T08:42:38.932 INFO:tasks.rados.rados.0.trial023.stdout:246: snap_create 2026-02-25T08:42:38.994 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:42:39.069 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:39.066+0000 7f176a65a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:39.072 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:39.069+0000 7f176a65a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:39.086 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:42:39.111 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:39.112 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:39.933 INFO:tasks.rados.rados.0.trial023.stdout:246: done (0 left) 2026-02-25T08:42:39.933 INFO:tasks.rados.rados.0.trial023.stdout:247: read oid 34 snap -1 2026-02-25T08:42:39.934 INFO:tasks.rados.rados.0.trial023.stdout:247: expect (ObjNum 104 snap 12 seq_num 104) 2026-02-25T08:42:39.934 INFO:tasks.rados.rados.0.trial023.stdout:248: snap_create 2026-02-25T08:42:39.967 INFO:tasks.rados.rados.0.trial023.stdout:247: expect (ObjNum 104 snap 12 seq_num 104) 2026-02-25T08:42:40.018 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:42:40.068 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:42:40.938 INFO:tasks.rados.rados.0.trial023.stdout:247: done (1 left) 2026-02-25T08:42:40.938 INFO:tasks.rados.rados.0.trial023.stdout:248: done (0 left) 2026-02-25T08:42:40.939 INFO:tasks.rados.rados.0.trial023.stdout:249: read oid 2 snap -1 2026-02-25T08:42:40.939 INFO:tasks.rados.rados.0.trial023.stdout:249: expect deleted 2026-02-25T08:42:40.939 INFO:tasks.rados.rados.0.trial023.stdout:250: delete oid 41 current snap is 24 2026-02-25T08:42:40.961 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~61s 2026-02-25T08:42:40.982 INFO:tasks.rados.rados.0.trial023.stdout:249: done (1 left) 2026-02-25T08:42:40.982 INFO:tasks.rados.rados.0.trial023.stdout:250: done (0 left) 2026-02-25T08:42:40.982 INFO:tasks.rados.rados.0.trial023.stdout:251: snap_create 2026-02-25T08:42:41.113 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:42:41.190 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:41.187+0000 7f6286635640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:41.193 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:41.191+0000 7f6286635640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:41.208 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:42:41.234 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:41.235 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:41.941 INFO:tasks.rados.rados.0.trial023.stdout:251: done (0 left) 2026-02-25T08:42:41.941 INFO:tasks.rados.rados.0.trial023.stdout:252: delete oid 35 current snap is 25 2026-02-25T08:42:41.984 INFO:tasks.rados.rados.0.trial023.stdout:252: done (0 left) 2026-02-25T08:42:41.984 INFO:tasks.rados.rados.0.trial023.stdout:253: delete oid 46 current snap is 25 2026-02-25T08:42:41.992 INFO:tasks.rados.rados.0.trial023.stdout:253: done (0 left) 2026-02-25T08:42:41.992 INFO:tasks.rados.rados.0.trial023.stdout:254: write oid 26 current snap is 25 2026-02-25T08:42:41.992 INFO:tasks.rados.rados.0.trial023.stdout:254: seq_num 148 ranges {487436=547559,1760982=657035} 2026-02-25T08:42:41.997 INFO:tasks.rados.rados.0.trial023.stdout:254: writing trial02342524-26 from 487436 to 1034995 tid 1 2026-02-25T08:42:42.002 INFO:tasks.rados.rados.0.trial023.stdout:254: writing trial02342524-26 from 1760982 to 2418017 tid 2 2026-02-25T08:42:42.002 INFO:tasks.rados.rados.0.trial023.stdout:255: snap_remove snap 22 2026-02-25T08:42:42.542 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:42:42.613 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:42:42.615 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:42:42.616 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:42:42.945 INFO:tasks.rados.rados.0.trial023.stdout:255: done (1 left) 2026-02-25T08:42:42.946 INFO:tasks.rados.rados.0.trial023.stdout:256: snap_create 2026-02-25T08:42:42.946 INFO:tasks.rados.rados.0.trial023.stdout:254: finishing write tid 1 to trial02342524-26 2026-02-25T08:42:42.946 INFO:tasks.rados.rados.0.trial023.stdout:254: finishing write tid 2 to trial02342524-26 2026-02-25T08:42:42.946 INFO:tasks.rados.rados.0.trial023.stdout:254: finishing write tid 3 to trial02342524-26 2026-02-25T08:42:42.946 INFO:tasks.rados.rados.0.trial023.stdout:254: finishing write tid 4 to trial02342524-26 2026-02-25T08:42:42.946 INFO:tasks.rados.rados.0.trial023.stdout:254: oid 26 updating version 0 to 75 2026-02-25T08:42:42.946 INFO:tasks.rados.rados.0.trial023.stdout:254: oid 26 version 75 is already newer than 75 2026-02-25T08:42:42.946 INFO:tasks.rados.rados.0.trial023.stdout:254: oid 26 version 75 is already newer than 75 2026-02-25T08:42:42.946 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 26 v 75 (ObjNum 148 snap 25 seq_num 148) dirty exists 2026-02-25T08:42:42.946 INFO:tasks.rados.rados.0.trial023.stdout:254: left oid 26 (ObjNum 148 snap 25 seq_num 148) 2026-02-25T08:42:43.237 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:42:43.314 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:43.311+0000 7fb49dddc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:43.317 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:43.315+0000 7fb49dddc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:43.333 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:42:43.361 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:43.361 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:43.948 INFO:tasks.rados.rados.0.trial023.stdout:254: done (1 left) 2026-02-25T08:42:43.948 INFO:tasks.rados.rados.0.trial023.stdout:256: done (0 left) 2026-02-25T08:42:43.948 INFO:tasks.rados.rados.0.trial023.stdout:257: write (excl) oid 29 current snap is 26 2026-02-25T08:42:43.948 INFO:tasks.rados.rados.0.trial023.stdout:257: seq_num 149 ranges {504243=622808,1819464=517046} 2026-02-25T08:42:43.952 INFO:tasks.rados.rados.0.trial023.stdout:257: writing trial02342524-29 from 504243 to 1127051 tid 1 2026-02-25T08:42:43.957 INFO:tasks.rados.rados.0.trial023.stdout:257: writing trial02342524-29 from 1819464 to 2336510 tid 2 2026-02-25T08:42:43.957 INFO:tasks.rados.rados.0.trial023.stdout:258: snap_remove snap 25 2026-02-25T08:42:44.801 INFO:tasks.rados.rados.0.trial023.stdout:258: done (1 left) 2026-02-25T08:42:44.801 INFO:tasks.rados.rados.0.trial023.stdout:259: write (excl) oid 28 current snap is 26 2026-02-25T08:42:44.801 INFO:tasks.rados.rados.0.trial023.stdout:257: finishing write tid 1 to trial02342524-29 2026-02-25T08:42:44.801 INFO:tasks.rados.rados.0.trial023.stdout:259: seq_num 150 ranges {521050=698057,1877946=643104} 2026-02-25T08:42:44.805 INFO:tasks.rados.rados.0.trial023.stdout:259: writing trial02342524-28 from 521050 to 1219107 tid 1 2026-02-25T08:42:44.811 INFO:tasks.rados.rados.0.trial023.stdout:259: writing trial02342524-28 from 1877946 to 2521050 tid 2 2026-02-25T08:42:44.811 INFO:tasks.rados.rados.0.trial023.stdout:260: write oid 19 current snap is 26 2026-02-25T08:42:44.811 INFO:tasks.rados.rados.0.trial023.stdout:257: finishing write tid 2 to trial02342524-29 2026-02-25T08:42:44.811 INFO:tasks.rados.rados.0.trial023.stdout:260: seq_num 151 ranges {537857=773306,1936428=520715} 2026-02-25T08:42:44.815 INFO:tasks.rados.rados.0.trial023.stdout:260: writing trial02342524-19 from 537857 to 1311163 tid 1 2026-02-25T08:42:44.820 INFO:tasks.rados.rados.0.trial023.stdout:260: writing trial02342524-19 from 1936428 to 2457143 tid 2 2026-02-25T08:42:44.820 INFO:tasks.rados.rados.0.trial023.stdout:261: read oid 41 snap 24 2026-02-25T08:42:44.820 INFO:tasks.rados.rados.0.trial023.stdout:261: expect deleted 2026-02-25T08:42:44.820 INFO:tasks.rados.rados.0.trial023.stdout:262: snap_remove snap 23 2026-02-25T08:42:45.362 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:42:45.438 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:45.435+0000 7f029d8cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:45.440 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:45.438+0000 7f029d8cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:45.455 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:42:45.483 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:45.484 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:45.802 INFO:tasks.rados.rados.0.trial023.stdout:262: done (4 left) 2026-02-25T08:42:45.803 INFO:tasks.rados.rados.0.trial023.stdout:263: read oid 20 snap -1 2026-02-25T08:42:45.803 INFO:tasks.rados.rados.0.trial023.stdout:263: expect (ObjNum 128 snap 15 seq_num 128) 2026-02-25T08:42:45.803 INFO:tasks.rados.rados.0.trial023.stdout:264: read oid 32 snap -1 2026-02-25T08:42:45.803 INFO:tasks.rados.rados.0.trial023.stdout:264: expect (ObjNum 31 snap 0 seq_num 31) 2026-02-25T08:42:45.803 INFO:tasks.rados.rados.0.trial023.stdout:265: write (excl) oid 1 current snap is 26 2026-02-25T08:42:45.803 INFO:tasks.rados.rados.0.trial023.stdout:257: finishing write tid 3 to trial02342524-29 2026-02-25T08:42:45.803 INFO:tasks.rados.rados.0.trial023.stdout:265: seq_num 152 ranges {554664=448555,1478557=664373} 2026-02-25T08:42:45.806 INFO:tasks.rados.rados.0.trial023.stdout:265: writing trial02342524-1 from 554664 to 1003219 tid 1 2026-02-25T08:42:45.811 INFO:tasks.rados.rados.0.trial023.stdout:265: writing trial02342524-1 from 1478557 to 2142930 tid 2 2026-02-25T08:42:45.811 INFO:tasks.rados.rados.0.trial023.stdout:266: snap_create 2026-02-25T08:42:45.811 INFO:tasks.rados.rados.0.trial023.stdout:257: finishing write tid 4 to trial02342524-29 2026-02-25T08:42:45.811 INFO:tasks.rados.rados.0.trial023.stdout:257: oid 29 updating version 0 to 79 2026-02-25T08:42:45.811 INFO:tasks.rados.rados.0.trial023.stdout:257: oid 29 version 79 is already newer than 79 2026-02-25T08:42:45.811 INFO:tasks.rados.rados.0.trial023.stdout:257: oid 29 version 79 is already newer than 77 2026-02-25T08:42:45.812 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 29 v 79 (ObjNum 149 snap 26 seq_num 149) dirty exists 2026-02-25T08:42:45.812 INFO:tasks.rados.rados.0.trial023.stdout:257: left oid 29 (ObjNum 149 snap 26 seq_num 149) 2026-02-25T08:42:45.812 INFO:tasks.rados.rados.0.trial023.stdout:259: finishing write tid 1 to trial02342524-28 2026-02-25T08:42:45.812 INFO:tasks.rados.rados.0.trial023.stdout:259: finishing write tid 2 to trial02342524-28 2026-02-25T08:42:45.812 INFO:tasks.rados.rados.0.trial023.stdout:259: finishing write tid 3 to trial02342524-28 2026-02-25T08:42:45.812 INFO:tasks.rados.rados.0.trial023.stdout:259: finishing write tid 4 to trial02342524-28 2026-02-25T08:42:45.812 INFO:tasks.rados.rados.0.trial023.stdout:259: oid 28 updating version 0 to 22 2026-02-25T08:42:45.812 INFO:tasks.rados.rados.0.trial023.stdout:259: oid 28 version 22 is already newer than 22 2026-02-25T08:42:45.812 INFO:tasks.rados.rados.0.trial023.stdout:259: oid 28 version 22 is already newer than 20 2026-02-25T08:42:45.812 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 28 v 22 (ObjNum 150 snap 26 seq_num 150) dirty exists 2026-02-25T08:42:45.812 INFO:tasks.rados.rados.0.trial023.stdout:259: left oid 28 (ObjNum 150 snap 26 seq_num 150) 2026-02-25T08:42:45.812 INFO:tasks.rados.rados.0.trial023.stdout:260: finishing write tid 1 to trial02342524-19 2026-02-25T08:42:45.813 INFO:tasks.rados.rados.0.trial023.stdout:260: finishing write tid 2 to trial02342524-19 2026-02-25T08:42:45.813 INFO:tasks.rados.rados.0.trial023.stdout:260: finishing write tid 3 to trial02342524-19 2026-02-25T08:42:45.813 INFO:tasks.rados.rados.0.trial023.stdout:260: finishing write tid 4 to trial02342524-19 2026-02-25T08:42:45.813 INFO:tasks.rados.rados.0.trial023.stdout:260: oid 19 updating version 0 to 54 2026-02-25T08:42:45.813 INFO:tasks.rados.rados.0.trial023.stdout:260: oid 19 version 54 is already newer than 54 2026-02-25T08:42:45.813 INFO:tasks.rados.rados.0.trial023.stdout:260: oid 19 version 54 is already newer than 54 2026-02-25T08:42:45.813 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 19 v 54 (ObjNum 151 snap 26 seq_num 151) dirty exists 2026-02-25T08:42:45.813 INFO:tasks.rados.rados.0.trial023.stdout:260: left oid 19 (ObjNum 151 snap 26 seq_num 151) 2026-02-25T08:42:45.842 INFO:tasks.rados.rados.0.trial023.stdout:263: expect (ObjNum 128 snap 15 seq_num 128) 2026-02-25T08:42:45.886 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:42:45.912 INFO:tasks.rados.rados.0.trial023.stdout:264: expect (ObjNum 31 snap 0 seq_num 31) 2026-02-25T08:42:45.912 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:42:45.932 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:42:45.953 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:42:45.975 INFO:tasks.rados.rados.0.trial023.stdout:265: finishing write tid 1 to trial02342524-1 2026-02-25T08:42:45.975 INFO:tasks.rados.rados.0.trial023.stdout:265: finishing write tid 2 to trial02342524-1 2026-02-25T08:42:45.975 INFO:tasks.rados.rados.0.trial023.stdout:265: finishing write tid 3 to trial02342524-1 2026-02-25T08:42:45.975 INFO:tasks.rados.rados.0.trial023.stdout:265: finishing write tid 4 to trial02342524-1 2026-02-25T08:42:45.975 INFO:tasks.rados.rados.0.trial023.stdout:265: oid 1 updating version 0 to 36 2026-02-25T08:42:45.975 INFO:tasks.rados.rados.0.trial023.stdout:265: oid 1 updating version 36 to 38 2026-02-25T08:42:45.975 INFO:tasks.rados.rados.0.trial023.stdout:265: oid 1 version 38 is already newer than 38 2026-02-25T08:42:45.975 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 1 v 38 (ObjNum 152 snap 26 seq_num 152) dirty exists 2026-02-25T08:42:45.975 INFO:tasks.rados.rados.0.trial023.stdout:265: left oid 1 (ObjNum 152 snap 26 seq_num 152) 2026-02-25T08:42:46.465 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~66s 2026-02-25T08:42:46.802 INFO:tasks.rados.rados.0.trial023.stdout:257: done (7 left) 2026-02-25T08:42:46.803 INFO:tasks.rados.rados.0.trial023.stdout:259: done (6 left) 2026-02-25T08:42:46.803 INFO:tasks.rados.rados.0.trial023.stdout:260: done (5 left) 2026-02-25T08:42:46.803 INFO:tasks.rados.rados.0.trial023.stdout:261: done (4 left) 2026-02-25T08:42:46.803 INFO:tasks.rados.rados.0.trial023.stdout:263: done (3 left) 2026-02-25T08:42:46.803 INFO:tasks.rados.rados.0.trial023.stdout:264: done (2 left) 2026-02-25T08:42:46.803 INFO:tasks.rados.rados.0.trial023.stdout:265: done (1 left) 2026-02-25T08:42:46.803 INFO:tasks.rados.rados.0.trial023.stdout:266: done (0 left) 2026-02-25T08:42:46.803 INFO:tasks.rados.rados.0.trial023.stdout:267: write oid 41 current snap is 27 2026-02-25T08:42:46.803 INFO:tasks.rados.rados.0.trial023.stdout:267: seq_num 153 ranges {571471=640157,1653392=524384} 2026-02-25T08:42:46.807 INFO:tasks.rados.rados.0.trial023.stdout:267: writing trial02342524-41 from 571471 to 1211628 tid 1 2026-02-25T08:42:46.810 INFO:tasks.rados.rados.0.trial023.stdout:267: writing trial02342524-41 from 1653392 to 2177776 tid 2 2026-02-25T08:42:46.811 INFO:tasks.rados.rados.0.trial023.stdout:268: snap_remove snap 26 2026-02-25T08:42:47.484 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:42:47.562 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:47.558+0000 7f1aa9251640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:47.564 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:47.562+0000 7f1aa9251640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:47.579 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:42:47.608 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:47.609 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:47.617 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:42:47.685 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:42:47.688 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:42:47.689 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:42:47.804 INFO:tasks.rados.rados.0.trial023.stdout:268: done (1 left) 2026-02-25T08:42:47.804 INFO:tasks.rados.rados.0.trial023.stdout:269: snap_create 2026-02-25T08:42:47.804 INFO:tasks.rados.rados.0.trial023.stdout:267: finishing write tid 1 to trial02342524-41 2026-02-25T08:42:47.804 INFO:tasks.rados.rados.0.trial023.stdout:267: finishing write tid 2 to trial02342524-41 2026-02-25T08:42:47.804 INFO:tasks.rados.rados.0.trial023.stdout:267: finishing write tid 3 to trial02342524-41 2026-02-25T08:42:47.804 INFO:tasks.rados.rados.0.trial023.stdout:267: finishing write tid 4 to trial02342524-41 2026-02-25T08:42:47.804 INFO:tasks.rados.rados.0.trial023.stdout:267: oid 41 updating version 0 to 39 2026-02-25T08:42:47.804 INFO:tasks.rados.rados.0.trial023.stdout:267: oid 41 version 39 is already newer than 39 2026-02-25T08:42:47.805 INFO:tasks.rados.rados.0.trial023.stdout:267: oid 41 version 39 is already newer than 39 2026-02-25T08:42:47.805 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 41 v 39 (ObjNum 153 snap 27 seq_num 153) dirty exists 2026-02-25T08:42:47.805 INFO:tasks.rados.rados.0.trial023.stdout:267: left oid 41 (ObjNum 153 snap 27 seq_num 153) 2026-02-25T08:42:48.804 INFO:tasks.rados.rados.0.trial023.stdout:267: done (1 left) 2026-02-25T08:42:48.804 INFO:tasks.rados.rados.0.trial023.stdout:269: done (0 left) 2026-02-25T08:42:48.804 INFO:tasks.rados.rados.0.trial023.stdout:270: delete oid 33 current snap is 28 2026-02-25T08:42:48.848 INFO:tasks.rados.rados.0.trial023.stdout:270: done (0 left) 2026-02-25T08:42:48.848 INFO:tasks.rados.rados.0.trial023.stdout:271: snap_create 2026-02-25T08:42:49.610 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:42:49.684 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:49.681+0000 7f6b40ddc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:49.687 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:49.685+0000 7f6b40ddc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:49.702 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:42:49.730 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:49.730 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:49.813 INFO:tasks.rados.rados.0.trial023.stdout:271: done (0 left) 2026-02-25T08:42:49.813 INFO:tasks.rados.rados.0.trial023.stdout:272: write (excl) oid 13 current snap is 29 2026-02-25T08:42:49.813 INFO:tasks.rados.rados.0.trial023.stdout:272: seq_num 155 ranges {605085=790655,2170356=434729} 2026-02-25T08:42:49.818 INFO:tasks.rados.rados.0.trial023.stdout:272: writing trial02342524-13 from 605085 to 1395740 tid 1 2026-02-25T08:42:49.823 INFO:tasks.rados.rados.0.trial023.stdout:272: writing trial02342524-13 from 2170356 to 2605085 tid 2 2026-02-25T08:42:49.823 INFO:tasks.rados.rados.0.trial023.stdout:273: snap_create 2026-02-25T08:42:49.862 INFO:tasks.rados.rados.0.trial023.stdout:272: finishing write tid 1 to trial02342524-13 2026-02-25T08:42:49.877 INFO:tasks.rados.rados.0.trial023.stdout:272: finishing write tid 2 to trial02342524-13 2026-02-25T08:42:49.880 INFO:tasks.rados.rados.0.trial023.stdout:272: finishing write tid 3 to trial02342524-13 2026-02-25T08:42:49.882 INFO:tasks.rados.rados.0.trial023.stdout:272: finishing write tid 4 to trial02342524-13 2026-02-25T08:42:49.882 INFO:tasks.rados.rados.0.trial023.stdout:272: oid 13 updating version 0 to 83 2026-02-25T08:42:49.882 INFO:tasks.rados.rados.0.trial023.stdout:272: oid 13 version 83 is already newer than 83 2026-02-25T08:42:49.882 INFO:tasks.rados.rados.0.trial023.stdout:272: oid 13 version 83 is already newer than 81 2026-02-25T08:42:49.883 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 13 v 83 (ObjNum 155 snap 29 seq_num 155) dirty exists 2026-02-25T08:42:49.883 INFO:tasks.rados.rados.0.trial023.stdout:272: left oid 13 (ObjNum 155 snap 29 seq_num 155) 2026-02-25T08:42:50.817 INFO:tasks.rados.rados.0.trial023.stdout:272: done (1 left) 2026-02-25T08:42:50.818 INFO:tasks.rados.rados.0.trial023.stdout:273: done (0 left) 2026-02-25T08:42:50.818 INFO:tasks.rados.rados.0.trial023.stdout:274: read oid 49 snap -1 2026-02-25T08:42:50.818 INFO:tasks.rados.rados.0.trial023.stdout:274: expect (ObjNum 109 snap 12 seq_num 109) 2026-02-25T08:42:50.818 INFO:tasks.rados.rados.0.trial023.stdout:275: snap_create 2026-02-25T08:42:50.849 INFO:tasks.rados.rados.0.trial023.stdout:274: expect (ObjNum 109 snap 12 seq_num 109) 2026-02-25T08:42:50.849 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:42:50.914 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 449399 2026-02-25T08:42:51.731 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:42:51.810 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:51.807+0000 7fda5a451640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:51.813 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:51.811+0000 7fda5a451640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:51.821 INFO:tasks.rados.rados.0.trial023.stdout:274: done (1 left) 2026-02-25T08:42:51.821 INFO:tasks.rados.rados.0.trial023.stdout:275: done (0 left) 2026-02-25T08:42:51.821 INFO:tasks.rados.rados.0.trial023.stdout:276: read oid 20 snap -1 2026-02-25T08:42:51.821 INFO:tasks.rados.rados.0.trial023.stdout:276: expect (ObjNum 128 snap 15 seq_num 128) 2026-02-25T08:42:51.821 INFO:tasks.rados.rados.0.trial023.stdout:277: snap_remove snap 30 2026-02-25T08:42:51.830 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:42:51.859 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:51.859 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:51.968 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~72s 2026-02-25T08:42:52.690 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:42:52.761 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:42:52.764 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:42:52.765 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:42:52.825 INFO:tasks.rados.rados.0.trial023.stdout:277: done (1 left) 2026-02-25T08:42:52.825 INFO:tasks.rados.rados.0.trial023.stdout:278: snap_remove snap 29 2026-02-25T08:42:53.828 INFO:tasks.rados.rados.0.trial023.stdout:278: done (1 left) 2026-02-25T08:42:53.828 INFO:tasks.rados.rados.0.trial023.stdout:279: delete oid 21 current snap is 31 2026-02-25T08:42:53.828 INFO:tasks.rados.rados.0.trial023.stdout:276: expect (ObjNum 128 snap 15 seq_num 128) 2026-02-25T08:42:53.828 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:42:53.861 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:42:53.881 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 542470 2026-02-25T08:42:53.902 INFO:tasks.rados.rados.0.trial023.stdout:276: done (1 left) 2026-02-25T08:42:53.902 INFO:tasks.rados.rados.0.trial023.stdout:279: done (0 left) 2026-02-25T08:42:53.902 INFO:tasks.rados.rados.0.trial023.stdout:280: write (excl) oid 24 current snap is 31 2026-02-25T08:42:53.902 INFO:tasks.rados.rados.0.trial023.stdout:280: seq_num 157 ranges {638699=541153,1770967=531722} 2026-02-25T08:42:53.906 INFO:tasks.rados.rados.0.trial023.stdout:280: writing trial02342524-24 from 638699 to 1179852 tid 1 2026-02-25T08:42:53.910 INFO:tasks.rados.rados.0.trial023.stdout:280: writing trial02342524-24 from 1770967 to 2302689 tid 2 2026-02-25T08:42:53.911 INFO:tasks.rados.rados.0.trial023.stdout:281: read oid 7 snap -1 2026-02-25T08:42:53.911 INFO:tasks.rados.rados.0.trial023.stdout:281: expect (ObjNum 6 snap 0 seq_num 6) 2026-02-25T08:42:53.911 INFO:tasks.rados.rados.0.trial023.stdout:282: read oid 45 snap -1 2026-02-25T08:42:53.911 INFO:tasks.rados.rados.0.trial023.stdout:282: expect (ObjNum 138 snap 19 seq_num 138) 2026-02-25T08:42:53.911 INFO:tasks.rados.rados.0.trial023.stdout:283: snap_remove snap 20 2026-02-25T08:42:53.938 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:53.935+0000 7f82f9fcc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:53.941 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:53.939+0000 7f82f9fcc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:53.974 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:42:54.000 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:54.001 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:54.803 INFO:tasks.rados.rados.0.trial023.stdout:283: done (3 left) 2026-02-25T08:42:54.803 INFO:tasks.rados.rados.0.trial023.stdout:284: delete oid 36 current snap is 31 2026-02-25T08:42:54.803 INFO:tasks.rados.rados.0.trial023.stdout:282: expect (ObjNum 138 snap 19 seq_num 138) 2026-02-25T08:42:54.844 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2175323 2026-02-25T08:42:54.861 INFO:tasks.rados.rados.0.trial023.stdout:282: done (3 left) 2026-02-25T08:42:54.861 INFO:tasks.rados.rados.0.trial023.stdout:284: done (2 left) 2026-02-25T08:42:54.861 INFO:tasks.rados.rados.0.trial023.stdout:285: snap_create 2026-02-25T08:42:54.861 INFO:tasks.rados.rados.0.trial023.stdout:280: finishing write tid 1 to trial02342524-24 2026-02-25T08:42:54.861 INFO:tasks.rados.rados.0.trial023.stdout:280: finishing write tid 2 to trial02342524-24 2026-02-25T08:42:54.862 INFO:tasks.rados.rados.0.trial023.stdout:281: expect (ObjNum 6 snap 0 seq_num 6) 2026-02-25T08:42:54.862 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:42:54.881 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1109038 2026-02-25T08:42:54.894 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1242326 2026-02-25T08:42:54.904 INFO:tasks.rados.rados.0.trial023.stdout:280: finishing write tid 3 to trial02342524-24 2026-02-25T08:42:54.904 INFO:tasks.rados.rados.0.trial023.stdout:280: finishing write tid 4 to trial02342524-24 2026-02-25T08:42:54.904 INFO:tasks.rados.rados.0.trial023.stdout:280: oid 24 updating version 0 to 43 2026-02-25T08:42:54.904 INFO:tasks.rados.rados.0.trial023.stdout:280: oid 24 version 43 is already newer than 41 2026-02-25T08:42:54.904 INFO:tasks.rados.rados.0.trial023.stdout:280: oid 24 version 43 is already newer than 43 2026-02-25T08:42:54.904 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 24 v 43 (ObjNum 157 snap 31 seq_num 157) dirty exists 2026-02-25T08:42:54.904 INFO:tasks.rados.rados.0.trial023.stdout:280: left oid 24 (ObjNum 157 snap 31 seq_num 157) 2026-02-25T08:42:55.834 INFO:tasks.rados.rados.0.trial023.stdout:280: done (2 left) 2026-02-25T08:42:55.834 INFO:tasks.rados.rados.0.trial023.stdout:281: done (1 left) 2026-02-25T08:42:55.834 INFO:tasks.rados.rados.0.trial023.stdout:285: done (0 left) 2026-02-25T08:42:55.835 INFO:tasks.rados.rados.0.trial023.stdout:286: snap_create 2026-02-25T08:42:56.002 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:42:56.079 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:56.075+0000 7f98a790b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:56.081 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:56.079+0000 7f98a790b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:56.096 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:42:56.124 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:56.125 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:56.835 INFO:tasks.rados.rados.0.trial023.stdout:286: done (0 left) 2026-02-25T08:42:56.835 INFO:tasks.rados.rados.0.trial023.stdout:287: snap_remove snap 32 2026-02-25T08:42:57.472 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~77s 2026-02-25T08:42:57.766 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:42:57.838 INFO:tasks.rados.rados.0.trial023.stdout:287: done (0 left) 2026-02-25T08:42:57.838 INFO:tasks.rados.rados.0.trial023.stdout:288: snap_create 2026-02-25T08:42:57.839 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:42:57.842 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:42:57.842 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:42:58.125 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:42:58.202 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:58.199+0000 7f76dfdad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:58.204 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:42:58.202+0000 7f76dfdad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:58.219 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:42:58.247 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:58.248 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:58.843 INFO:tasks.rados.rados.0.trial023.stdout:288: done (0 left) 2026-02-25T08:42:58.844 INFO:tasks.rados.rados.0.trial023.stdout:289: write (excl) oid 4 current snap is 34 2026-02-25T08:42:58.844 INFO:tasks.rados.rados.0.trial023.stdout:289: seq_num 159 ranges {672313=691651,1887931=419038} 2026-02-25T08:42:58.848 INFO:tasks.rados.rados.0.trial023.stdout:289: writing trial02342524-4 from 672313 to 1363964 tid 1 2026-02-25T08:42:58.852 INFO:tasks.rados.rados.0.trial023.stdout:289: writing trial02342524-4 from 1887931 to 2306969 tid 2 2026-02-25T08:42:58.853 INFO:tasks.rados.rados.0.trial023.stdout:290: write (excl) oid 22 current snap is 34 2026-02-25T08:42:58.853 INFO:tasks.rados.rados.0.trial023.stdout:290: seq_num 160 ranges {689120=483253,1946413=679049} 2026-02-25T08:42:58.857 INFO:tasks.rados.rados.0.trial023.stdout:290: writing trial02342524-22 from 689120 to 1172373 tid 1 2026-02-25T08:42:58.863 INFO:tasks.rados.rados.0.trial023.stdout:290: writing trial02342524-22 from 1946413 to 2625462 tid 2 2026-02-25T08:42:58.863 INFO:tasks.rados.rados.0.trial023.stdout:291: write oid 34 current snap is 34 2026-02-25T08:42:58.863 INFO:tasks.rados.rados.0.trial023.stdout:291: seq_num 161 ranges {705927=558502,2004895=422707} 2026-02-25T08:42:58.867 INFO:tasks.rados.rados.0.trial023.stdout:291: writing trial02342524-34 from 705927 to 1264429 tid 1 2026-02-25T08:42:58.872 INFO:tasks.rados.rados.0.trial023.stdout:291: writing trial02342524-34 from 2004895 to 2427602 tid 2 2026-02-25T08:42:58.872 INFO:tasks.rados.rados.0.trial023.stdout:292: write oid 15 current snap is 34 2026-02-25T08:42:58.872 INFO:tasks.rados.rados.0.trial023.stdout:292: seq_num 162 ranges {722734=633751,2063377=659357} 2026-02-25T08:42:58.877 INFO:tasks.rados.rados.0.trial023.stdout:292: writing trial02342524-15 from 722734 to 1356485 tid 1 2026-02-25T08:42:58.882 INFO:tasks.rados.rados.0.trial023.stdout:292: writing trial02342524-15 from 2063377 to 2722734 tid 2 2026-02-25T08:42:58.882 INFO:tasks.rados.rados.0.trial023.stdout:293: read oid 10 snap 33 2026-02-25T08:42:58.882 INFO:tasks.rados.rados.0.trial023.stdout:293: expect deleted 2026-02-25T08:42:58.883 INFO:tasks.rados.rados.0.trial023.stdout:294: delete oid 11 current snap is 34 2026-02-25T08:42:58.893 INFO:tasks.rados.rados.0.trial023.stdout:289: finishing write tid 1 to trial02342524-4 2026-02-25T08:42:58.893 INFO:tasks.rados.rados.0.trial023.stdout:290: finishing write tid 1 to trial02342524-22 2026-02-25T08:42:58.900 INFO:tasks.rados.rados.0.trial023.stdout:292: finishing write tid 1 to trial02342524-15 2026-02-25T08:42:58.909 INFO:tasks.rados.rados.0.trial023.stdout:289: finishing write tid 2 to trial02342524-4 2026-02-25T08:42:58.910 INFO:tasks.rados.rados.0.trial023.stdout:292: finishing write tid 2 to trial02342524-15 2026-02-25T08:42:58.918 INFO:tasks.rados.rados.0.trial023.stdout:289: finishing write tid 3 to trial02342524-4 2026-02-25T08:42:58.920 INFO:tasks.rados.rados.0.trial023.stdout:289: finishing write tid 4 to trial02342524-4 2026-02-25T08:42:58.921 INFO:tasks.rados.rados.0.trial023.stdout:289: oid 4 updating version 0 to 91 2026-02-25T08:42:58.921 INFO:tasks.rados.rados.0.trial023.stdout:289: oid 4 updating version 91 to 93 2026-02-25T08:42:58.921 INFO:tasks.rados.rados.0.trial023.stdout:289: oid 4 version 93 is already newer than 93 2026-02-25T08:42:58.921 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 4 v 93 (ObjNum 159 snap 34 seq_num 159) dirty exists 2026-02-25T08:42:58.921 INFO:tasks.rados.rados.0.trial023.stdout:289: left oid 4 (ObjNum 159 snap 34 seq_num 159) 2026-02-25T08:42:58.921 INFO:tasks.rados.rados.0.trial023.stdout:292: finishing write tid 3 to trial02342524-15 2026-02-25T08:42:58.923 INFO:tasks.rados.rados.0.trial023.stdout:292: finishing write tid 4 to trial02342524-15 2026-02-25T08:42:58.923 INFO:tasks.rados.rados.0.trial023.stdout:292: oid 15 updating version 0 to 39 2026-02-25T08:42:58.923 INFO:tasks.rados.rados.0.trial023.stdout:292: oid 15 version 39 is already newer than 39 2026-02-25T08:42:58.923 INFO:tasks.rados.rados.0.trial023.stdout:292: oid 15 version 39 is already newer than 39 2026-02-25T08:42:58.923 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 15 v 39 (ObjNum 162 snap 34 seq_num 162) dirty exists 2026-02-25T08:42:58.924 INFO:tasks.rados.rados.0.trial023.stdout:292: left oid 15 (ObjNum 162 snap 34 seq_num 162) 2026-02-25T08:42:58.928 INFO:tasks.rados.rados.0.trial023.stdout:290: finishing write tid 2 to trial02342524-22 2026-02-25T08:42:58.931 INFO:tasks.rados.rados.0.trial023.stdout:289: done (5 left) 2026-02-25T08:42:58.931 INFO:tasks.rados.rados.0.trial023.stdout:292: done (4 left) 2026-02-25T08:42:58.932 INFO:tasks.rados.rados.0.trial023.stdout:293: done (3 left) 2026-02-25T08:42:58.932 INFO:tasks.rados.rados.0.trial023.stdout:294: done (2 left) 2026-02-25T08:42:58.932 INFO:tasks.rados.rados.0.trial023.stdout:295: delete oid 37 current snap is 34 2026-02-25T08:42:58.934 INFO:tasks.rados.rados.0.trial023.stdout:295: done (2 left) 2026-02-25T08:42:58.934 INFO:tasks.rados.rados.0.trial023.stdout:296: write (excl) oid 8 current snap is 34 2026-02-25T08:42:58.934 INFO:tasks.rados.rados.0.trial023.stdout:296: seq_num 165 ranges {773155=459498,1722470=430045,2648676=124479} 2026-02-25T08:42:58.939 INFO:tasks.rados.rados.0.trial023.stdout:296: writing trial02342524-8 from 773155 to 1232653 tid 1 2026-02-25T08:42:58.943 INFO:tasks.rados.rados.0.trial023.stdout:296: writing trial02342524-8 from 1722470 to 2152515 tid 2 2026-02-25T08:42:58.945 INFO:tasks.rados.rados.0.trial023.stdout:296: writing trial02342524-8 from 2648676 to 2773155 tid 3 2026-02-25T08:42:58.945 INFO:tasks.rados.rados.0.trial023.stdout:297: snap_create 2026-02-25T08:42:58.946 INFO:tasks.rados.rados.0.trial023.stdout:290: finishing write tid 3 to trial02342524-22 2026-02-25T08:42:58.946 INFO:tasks.rados.rados.0.trial023.stdout:290: finishing write tid 4 to trial02342524-22 2026-02-25T08:42:58.946 INFO:tasks.rados.rados.0.trial023.stdout:290: oid 22 updating version 0 to 25 2026-02-25T08:42:58.946 INFO:tasks.rados.rados.0.trial023.stdout:290: oid 22 updating version 25 to 27 2026-02-25T08:42:58.946 INFO:tasks.rados.rados.0.trial023.stdout:290: oid 22 version 27 is already newer than 27 2026-02-25T08:42:58.946 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 22 v 27 (ObjNum 160 snap 34 seq_num 160) dirty exists 2026-02-25T08:42:58.946 INFO:tasks.rados.rados.0.trial023.stdout:290: left oid 22 (ObjNum 160 snap 34 seq_num 160) 2026-02-25T08:42:58.956 INFO:tasks.rados.rados.0.trial023.stdout:291: finishing write tid 1 to trial02342524-34 2026-02-25T08:42:58.967 INFO:tasks.rados.rados.0.trial023.stdout:291: finishing write tid 2 to trial02342524-34 2026-02-25T08:42:58.975 INFO:tasks.rados.rados.0.trial023.stdout:291: finishing write tid 3 to trial02342524-34 2026-02-25T08:42:58.977 INFO:tasks.rados.rados.0.trial023.stdout:291: finishing write tid 4 to trial02342524-34 2026-02-25T08:42:58.977 INFO:tasks.rados.rados.0.trial023.stdout:291: oid 34 updating version 0 to 31 2026-02-25T08:42:58.978 INFO:tasks.rados.rados.0.trial023.stdout:291: oid 34 version 31 is already newer than 31 2026-02-25T08:42:58.978 INFO:tasks.rados.rados.0.trial023.stdout:291: oid 34 version 31 is already newer than 31 2026-02-25T08:42:58.978 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 34 v 31 (ObjNum 161 snap 34 seq_num 161) dirty exists 2026-02-25T08:42:58.978 INFO:tasks.rados.rados.0.trial023.stdout:291: left oid 34 (ObjNum 161 snap 34 seq_num 161) 2026-02-25T08:42:58.988 INFO:tasks.rados.rados.0.trial023.stdout:296: finishing write tid 1 to trial02342524-8 2026-02-25T08:42:59.002 INFO:tasks.rados.rados.0.trial023.stdout:296: finishing write tid 2 to trial02342524-8 2026-02-25T08:42:59.008 INFO:tasks.rados.rados.0.trial023.stdout:296: finishing write tid 3 to trial02342524-8 2026-02-25T08:42:59.013 INFO:tasks.rados.rados.0.trial023.stdout:296: finishing write tid 4 to trial02342524-8 2026-02-25T08:42:59.015 INFO:tasks.rados.rados.0.trial023.stdout:296: finishing write tid 5 to trial02342524-8 2026-02-25T08:42:59.015 INFO:tasks.rados.rados.0.trial023.stdout:296: oid 8 updating version 0 to 13 2026-02-25T08:42:59.015 INFO:tasks.rados.rados.0.trial023.stdout:296: oid 8 updating version 13 to 16 2026-02-25T08:42:59.015 INFO:tasks.rados.rados.0.trial023.stdout:296: oid 8 version 16 is already newer than 16 2026-02-25T08:42:59.015 INFO:tasks.rados.rados.0.trial023.stdout:296: oid 8 version 16 is already newer than 16 2026-02-25T08:42:59.015 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 8 v 16 (ObjNum 165 snap 34 seq_num 165) dirty exists 2026-02-25T08:42:59.015 INFO:tasks.rados.rados.0.trial023.stdout:296: left oid 8 (ObjNum 165 snap 34 seq_num 165) 2026-02-25T08:42:59.843 INFO:tasks.rados.rados.0.trial023.stdout:290: done (3 left) 2026-02-25T08:42:59.843 INFO:tasks.rados.rados.0.trial023.stdout:291: done (2 left) 2026-02-25T08:42:59.844 INFO:tasks.rados.rados.0.trial023.stdout:296: done (1 left) 2026-02-25T08:42:59.844 INFO:tasks.rados.rados.0.trial023.stdout:297: done (0 left) 2026-02-25T08:42:59.844 INFO:tasks.rados.rados.0.trial023.stdout:298: write oid 33 current snap is 35 2026-02-25T08:42:59.844 INFO:tasks.rados.rados.0.trial023.stdout:298: seq_num 166 ranges {789962=534747,1780952=690056} 2026-02-25T08:42:59.848 INFO:tasks.rados.rados.0.trial023.stdout:298: writing trial02342524-33 from 789962 to 1324709 tid 1 2026-02-25T08:42:59.853 INFO:tasks.rados.rados.0.trial023.stdout:298: writing trial02342524-33 from 1780952 to 2471008 tid 2 2026-02-25T08:42:59.853 INFO:tasks.rados.rados.0.trial023.stdout:299: read oid 5 snap -1 2026-02-25T08:42:59.853 INFO:tasks.rados.rados.0.trial023.stdout:299: expect (ObjNum 4 snap 0 seq_num 4) 2026-02-25T08:42:59.853 INFO:tasks.rados.rados.0.trial023.stdout:300: snap_create 2026-02-25T08:42:59.874 INFO:tasks.rados.rados.0.trial023.stdout:298: finishing write tid 1 to trial02342524-33 2026-02-25T08:42:59.880 INFO:tasks.rados.rados.0.trial023.stdout:298: finishing write tid 2 to trial02342524-33 2026-02-25T08:42:59.880 INFO:tasks.rados.rados.0.trial023.stdout:299: expect (ObjNum 4 snap 0 seq_num 4) 2026-02-25T08:42:59.881 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:42:59.898 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:42:59.933 INFO:tasks.rados.rados.0.trial023.stdout:298: finishing write tid 3 to trial02342524-33 2026-02-25T08:42:59.933 INFO:tasks.rados.rados.0.trial023.stdout:298: finishing write tid 4 to trial02342524-33 2026-02-25T08:42:59.933 INFO:tasks.rados.rados.0.trial023.stdout:298: oid 33 updating version 0 to 17 2026-02-25T08:42:59.933 INFO:tasks.rados.rados.0.trial023.stdout:298: oid 33 version 17 is already newer than 17 2026-02-25T08:42:59.933 INFO:tasks.rados.rados.0.trial023.stdout:298: oid 33 version 17 is already newer than 17 2026-02-25T08:42:59.933 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 33 v 17 (ObjNum 166 snap 35 seq_num 166) dirty exists 2026-02-25T08:42:59.934 INFO:tasks.rados.rados.0.trial023.stdout:298: left oid 33 (ObjNum 166 snap 35 seq_num 166) 2026-02-25T08:43:00.249 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:00.325 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:00.323+0000 7fcabe10b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:00.327 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:00.326+0000 7fcabe10b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:00.342 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:43:00.369 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:00.370 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:00.847 INFO:tasks.rados.rados.0.trial023.stdout:298: done (2 left) 2026-02-25T08:43:00.848 INFO:tasks.rados.rados.0.trial023.stdout:299: done (1 left) 2026-02-25T08:43:00.848 INFO:tasks.rados.rados.0.trial023.stdout:300: done (0 left) 2026-02-25T08:43:00.848 INFO:tasks.rados.rados.0.trial023.stdout:301: write oid 38 current snap is 36 2026-02-25T08:43:00.848 INFO:tasks.rados.rados.0.trial023.stdout:301: seq_num 167 ranges {406769=609996,1439434=433714,2303522=503247} 2026-02-25T08:43:00.851 INFO:tasks.rados.rados.0.trial023.stdout:301: writing trial02342524-38 from 406769 to 1016765 tid 1 2026-02-25T08:43:00.855 INFO:tasks.rados.rados.0.trial023.stdout:301: writing trial02342524-38 from 1439434 to 1873148 tid 2 2026-02-25T08:43:00.859 INFO:tasks.rados.rados.0.trial023.stdout:301: writing trial02342524-38 from 2303522 to 2806769 tid 3 2026-02-25T08:43:00.859 INFO:tasks.rados.rados.0.trial023.stdout:302: read oid 40 snap -1 2026-02-25T08:43:00.859 INFO:tasks.rados.rados.0.trial023.stdout:302: expect (ObjNum 103 snap 12 seq_num 103) 2026-02-25T08:43:00.859 INFO:tasks.rados.rados.0.trial023.stdout:303: read oid 13 snap 33 2026-02-25T08:43:00.859 INFO:tasks.rados.rados.0.trial023.stdout:303: expect (ObjNum 155 snap 29 seq_num 155) 2026-02-25T08:43:00.859 INFO:tasks.rados.rados.0.trial023.stdout:304: read oid 30 snap -1 2026-02-25T08:43:00.859 INFO:tasks.rados.rados.0.trial023.stdout:304: expect deleted 2026-02-25T08:43:00.859 INFO:tasks.rados.rados.0.trial023.stdout:305: delete oid 27 current snap is 36 2026-02-25T08:43:00.878 INFO:tasks.rados.rados.0.trial023.stdout:302: expect (ObjNum 103 snap 12 seq_num 103) 2026-02-25T08:43:00.878 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 20754 2026-02-25T08:43:00.932 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2424407 2026-02-25T08:43:00.945 INFO:tasks.rados.rados.0.trial023.stdout:303: expect (ObjNum 155 snap 29 seq_num 155) 2026-02-25T08:43:01.019 INFO:tasks.rados.rados.0.trial023.stdout:302: done (4 left) 2026-02-25T08:43:01.020 INFO:tasks.rados.rados.0.trial023.stdout:303: done (3 left) 2026-02-25T08:43:01.020 INFO:tasks.rados.rados.0.trial023.stdout:304: done (2 left) 2026-02-25T08:43:01.020 INFO:tasks.rados.rados.0.trial023.stdout:305: done (1 left) 2026-02-25T08:43:01.020 INFO:tasks.rados.rados.0.trial023.stdout:306: read oid 27 snap -1 2026-02-25T08:43:01.020 INFO:tasks.rados.rados.0.trial023.stdout:306: expect deleted 2026-02-25T08:43:01.020 INFO:tasks.rados.rados.0.trial023.stdout:307: snap_create 2026-02-25T08:43:01.020 INFO:tasks.rados.rados.0.trial023.stdout:301: finishing write tid 1 to trial02342524-38 2026-02-25T08:43:01.020 INFO:tasks.rados.rados.0.trial023.stdout:301: finishing write tid 2 to trial02342524-38 2026-02-25T08:43:01.020 INFO:tasks.rados.rados.0.trial023.stdout:301: finishing write tid 3 to trial02342524-38 2026-02-25T08:43:01.020 INFO:tasks.rados.rados.0.trial023.stdout:301: finishing write tid 4 to trial02342524-38 2026-02-25T08:43:01.021 INFO:tasks.rados.rados.0.trial023.stdout:301: finishing write tid 5 to trial02342524-38 2026-02-25T08:43:01.021 INFO:tasks.rados.rados.0.trial023.stdout:301: oid 38 updating version 0 to 61 2026-02-25T08:43:01.021 INFO:tasks.rados.rados.0.trial023.stdout:301: oid 38 version 61 is already newer than 61 2026-02-25T08:43:01.021 INFO:tasks.rados.rados.0.trial023.stdout:301: oid 38 version 61 is already newer than 61 2026-02-25T08:43:01.021 INFO:tasks.rados.rados.0.trial023.stdout:301: oid 38 version 61 is already newer than 61 2026-02-25T08:43:01.021 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 38 v 61 (ObjNum 167 snap 36 seq_num 167) dirty exists 2026-02-25T08:43:01.021 INFO:tasks.rados.rados.0.trial023.stdout:301: left oid 38 (ObjNum 167 snap 36 seq_num 167) 2026-02-25T08:43:01.852 INFO:tasks.rados.rados.0.trial023.stdout:301: done (2 left) 2026-02-25T08:43:01.852 INFO:tasks.rados.rados.0.trial023.stdout:306: done (1 left) 2026-02-25T08:43:01.853 INFO:tasks.rados.rados.0.trial023.stdout:307: done (0 left) 2026-02-25T08:43:01.853 INFO:tasks.rados.rados.0.trial023.stdout:308: snap_create 2026-02-25T08:43:02.370 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:02.448 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:02.446+0000 7fc648c11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:02.450 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:02.449+0000 7fc648c11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:02.465 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:43:02.492 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:02.493 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:02.843 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:43:02.854 INFO:tasks.rados.rados.0.trial023.stdout:308: done (0 left) 2026-02-25T08:43:02.854 INFO:tasks.rados.rados.0.trial023.stdout:309: read oid 13 snap 36 2026-02-25T08:43:02.854 INFO:tasks.rados.rados.0.trial023.stdout:309: expect (ObjNum 155 snap 29 seq_num 155) 2026-02-25T08:43:02.854 INFO:tasks.rados.rados.0.trial023.stdout:310: snap_remove snap 28 2026-02-25T08:43:02.916 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:43:02.918 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:43:02.919 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:43:02.977 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~83s 2026-02-25T08:43:03.858 INFO:tasks.rados.rados.0.trial023.stdout:310: done (1 left) 2026-02-25T08:43:03.858 INFO:tasks.rados.rados.0.trial023.stdout:311: write (excl) oid 36 current snap is 38 2026-02-25T08:43:03.858 INFO:tasks.rados.rados.0.trial023.stdout:311: seq_num 169 ranges {440383=476847,1556398=437383,2474721=365662} 2026-02-25T08:43:03.861 INFO:tasks.rados.rados.0.trial023.stdout:311: writing trial02342524-36 from 440383 to 917230 tid 1 2026-02-25T08:43:03.865 INFO:tasks.rados.rados.0.trial023.stdout:311: writing trial02342524-36 from 1556398 to 1993781 tid 2 2026-02-25T08:43:03.869 INFO:tasks.rados.rados.0.trial023.stdout:311: writing trial02342524-36 from 2474721 to 2840383 tid 3 2026-02-25T08:43:03.869 INFO:tasks.rados.rados.0.trial023.stdout:312: snap_create 2026-02-25T08:43:03.869 INFO:tasks.rados.rados.0.trial023.stdout:309: expect (ObjNum 155 snap 29 seq_num 155) 2026-02-25T08:43:03.917 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:43:03.946 INFO:tasks.rados.rados.0.trial023.stdout:311: finishing write tid 1 to trial02342524-36 2026-02-25T08:43:03.946 INFO:tasks.rados.rados.0.trial023.stdout:311: finishing write tid 2 to trial02342524-36 2026-02-25T08:43:03.946 INFO:tasks.rados.rados.0.trial023.stdout:311: finishing write tid 3 to trial02342524-36 2026-02-25T08:43:03.946 INFO:tasks.rados.rados.0.trial023.stdout:311: finishing write tid 4 to trial02342524-36 2026-02-25T08:43:03.946 INFO:tasks.rados.rados.0.trial023.stdout:311: finishing write tid 5 to trial02342524-36 2026-02-25T08:43:03.946 INFO:tasks.rados.rados.0.trial023.stdout:311: oid 36 updating version 0 to 43 2026-02-25T08:43:03.947 INFO:tasks.rados.rados.0.trial023.stdout:311: oid 36 version 43 is already newer than 43 2026-02-25T08:43:03.947 INFO:tasks.rados.rados.0.trial023.stdout:311: oid 36 version 43 is already newer than 40 2026-02-25T08:43:03.947 INFO:tasks.rados.rados.0.trial023.stdout:311: oid 36 version 43 is already newer than 43 2026-02-25T08:43:03.947 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 36 v 43 (ObjNum 169 snap 38 seq_num 169) dirty exists 2026-02-25T08:43:03.947 INFO:tasks.rados.rados.0.trial023.stdout:311: left oid 36 (ObjNum 169 snap 38 seq_num 169) 2026-02-25T08:43:04.494 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:04.572 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:04.570+0000 7fc58a61a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:04.575 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:04.574+0000 7fc58a61a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:04.590 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:43:04.617 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:04.617 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:04.806 INFO:tasks.rados.rados.0.trial023.stdout:309: done (2 left) 2026-02-25T08:43:04.806 INFO:tasks.rados.rados.0.trial023.stdout:311: done (1 left) 2026-02-25T08:43:04.806 INFO:tasks.rados.rados.0.trial023.stdout:312: done (0 left) 2026-02-25T08:43:04.806 INFO:tasks.rados.rados.0.trial023.stdout:313: read oid 26 snap 35 2026-02-25T08:43:04.806 INFO:tasks.rados.rados.0.trial023.stdout:313: expect (ObjNum 148 snap 25 seq_num 148) 2026-02-25T08:43:04.807 INFO:tasks.rados.rados.0.trial023.stdout:314: write oid 23 current snap is 39 2026-02-25T08:43:04.807 INFO:tasks.rados.rados.0.trial023.stdout:314: seq_num 170 ranges {457190=552096,1614880=581041,2785791=71399} 2026-02-25T08:43:04.809 INFO:tasks.rados.rados.0.trial023.stdout:314: writing trial02342524-23 from 457190 to 1009286 tid 1 2026-02-25T08:43:04.814 INFO:tasks.rados.rados.0.trial023.stdout:314: writing trial02342524-23 from 1614880 to 2195921 tid 2 2026-02-25T08:43:04.816 INFO:tasks.rados.rados.0.trial023.stdout:314: writing trial02342524-23 from 2785791 to 2857190 tid 3 2026-02-25T08:43:04.816 INFO:tasks.rados.rados.0.trial023.stdout:315: read oid 1 snap -1 2026-02-25T08:43:04.817 INFO:tasks.rados.rados.0.trial023.stdout:315: expect (ObjNum 152 snap 26 seq_num 152) 2026-02-25T08:43:04.817 INFO:tasks.rados.rados.0.trial023.stdout:316: snap_create 2026-02-25T08:43:04.834 INFO:tasks.rados.rados.0.trial023.stdout:315: expect (ObjNum 152 snap 26 seq_num 152) 2026-02-25T08:43:04.885 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:43:04.917 INFO:tasks.rados.rados.0.trial023.stdout:313: expect (ObjNum 148 snap 25 seq_num 148) 2026-02-25T08:43:04.917 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:43:04.957 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:43:05.039 INFO:tasks.rados.rados.0.trial023.stdout:314: finishing write tid 1 to trial02342524-23 2026-02-25T08:43:05.039 INFO:tasks.rados.rados.0.trial023.stdout:314: finishing write tid 2 to trial02342524-23 2026-02-25T08:43:05.039 INFO:tasks.rados.rados.0.trial023.stdout:314: finishing write tid 3 to trial02342524-23 2026-02-25T08:43:05.040 INFO:tasks.rados.rados.0.trial023.stdout:314: finishing write tid 4 to trial02342524-23 2026-02-25T08:43:05.040 INFO:tasks.rados.rados.0.trial023.stdout:314: finishing write tid 5 to trial02342524-23 2026-02-25T08:43:05.040 INFO:tasks.rados.rados.0.trial023.stdout:314: oid 23 updating version 0 to 88 2026-02-25T08:43:05.040 INFO:tasks.rados.rados.0.trial023.stdout:314: oid 23 version 88 is already newer than 88 2026-02-25T08:43:05.040 INFO:tasks.rados.rados.0.trial023.stdout:314: oid 23 version 88 is already newer than 88 2026-02-25T08:43:05.040 INFO:tasks.rados.rados.0.trial023.stdout:314: oid 23 version 88 is already newer than 88 2026-02-25T08:43:05.040 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 23 v 88 (ObjNum 170 snap 39 seq_num 170) dirty exists 2026-02-25T08:43:05.040 INFO:tasks.rados.rados.0.trial023.stdout:314: left oid 23 (ObjNum 170 snap 39 seq_num 170) 2026-02-25T08:43:05.806 INFO:tasks.rados.rados.0.trial023.stdout:313: done (3 left) 2026-02-25T08:43:05.807 INFO:tasks.rados.rados.0.trial023.stdout:314: done (2 left) 2026-02-25T08:43:05.807 INFO:tasks.rados.rados.0.trial023.stdout:315: done (1 left) 2026-02-25T08:43:05.807 INFO:tasks.rados.rados.0.trial023.stdout:316: done (0 left) 2026-02-25T08:43:05.807 INFO:tasks.rados.rados.0.trial023.stdout:317: write oid 37 current snap is 40 2026-02-25T08:43:05.807 INFO:tasks.rados.rados.0.trial023.stdout:317: seq_num 171 ranges {473997=627345,1673362=441052,2529567=344430} 2026-02-25T08:43:05.810 INFO:tasks.rados.rados.0.trial023.stdout:317: writing trial02342524-37 from 473997 to 1101342 tid 1 2026-02-25T08:43:05.814 INFO:tasks.rados.rados.0.trial023.stdout:317: writing trial02342524-37 from 1673362 to 2114414 tid 2 2026-02-25T08:43:05.818 INFO:tasks.rados.rados.0.trial023.stdout:317: writing trial02342524-37 from 2529567 to 2873997 tid 3 2026-02-25T08:43:05.818 INFO:tasks.rados.rados.0.trial023.stdout:318: write oid 9 current snap is 40 2026-02-25T08:43:05.818 INFO:tasks.rados.rados.0.trial023.stdout:318: seq_num 172 ranges {490804=702594,1615491=584710,2724284=166520} 2026-02-25T08:43:05.822 INFO:tasks.rados.rados.0.trial023.stdout:318: writing trial02342524-9 from 490804 to 1193398 tid 1 2026-02-25T08:43:05.826 INFO:tasks.rados.rados.0.trial023.stdout:318: writing trial02342524-9 from 1615491 to 2200201 tid 2 2026-02-25T08:43:05.829 INFO:tasks.rados.rados.0.trial023.stdout:318: writing trial02342524-9 from 2724284 to 2890804 tid 3 2026-02-25T08:43:05.829 INFO:tasks.rados.rados.0.trial023.stdout:319: delete oid 24 current snap is 40 2026-02-25T08:43:05.837 INFO:tasks.rados.rados.0.trial023.stdout:317: finishing write tid 1 to trial02342524-37 2026-02-25T08:43:05.845 INFO:tasks.rados.rados.0.trial023.stdout:317: finishing write tid 2 to trial02342524-37 2026-02-25T08:43:05.862 INFO:tasks.rados.rados.0.trial023.stdout:317: finishing write tid 3 to trial02342524-37 2026-02-25T08:43:05.865 INFO:tasks.rados.rados.0.trial023.stdout:317: finishing write tid 4 to trial02342524-37 2026-02-25T08:43:05.867 INFO:tasks.rados.rados.0.trial023.stdout:318: finishing write tid 1 to trial02342524-9 2026-02-25T08:43:05.868 INFO:tasks.rados.rados.0.trial023.stdout:319: done (2 left) 2026-02-25T08:43:05.869 INFO:tasks.rados.rados.0.trial023.stdout:320: snap_remove snap 39 2026-02-25T08:43:06.618 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:06.696 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:06.694+0000 7f8a2f1fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:06.698 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:06.697+0000 7f8a2f1fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:06.713 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:43:06.738 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:06.739 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:06.808 INFO:tasks.rados.rados.0.trial023.stdout:320: done (2 left) 2026-02-25T08:43:06.808 INFO:tasks.rados.rados.0.trial023.stdout:321: read oid 48 snap -1 2026-02-25T08:43:06.808 INFO:tasks.rados.rados.0.trial023.stdout:321: expect deleted 2026-02-25T08:43:06.809 INFO:tasks.rados.rados.0.trial023.stdout:322: write (excl) oid 32 current snap is 40 2026-02-25T08:43:06.809 INFO:tasks.rados.rados.0.trial023.stdout:317: finishing write tid 5 to trial02342524-37 2026-02-25T08:43:06.809 INFO:tasks.rados.rados.0.trial023.stdout:317: oid 37 updating version 0 to 92 2026-02-25T08:43:06.809 INFO:tasks.rados.rados.0.trial023.stdout:317: oid 37 version 92 is already newer than 92 2026-02-25T08:43:06.809 INFO:tasks.rados.rados.0.trial023.stdout:317: oid 37 version 92 is already newer than 92 2026-02-25T08:43:06.809 INFO:tasks.rados.rados.0.trial023.stdout:317: oid 37 version 92 is already newer than 92 2026-02-25T08:43:06.809 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 37 v 92 (ObjNum 171 snap 40 seq_num 171) dirty exists 2026-02-25T08:43:06.809 INFO:tasks.rados.rados.0.trial023.stdout:317: left oid 37 (ObjNum 171 snap 40 seq_num 171) 2026-02-25T08:43:06.809 INFO:tasks.rados.rados.0.trial023.stdout:322: seq_num 174 ranges {524418=453092,1732455=588379,2779130=145288} 2026-02-25T08:43:06.811 INFO:tasks.rados.rados.0.trial023.stdout:322: writing trial02342524-32 from 524418 to 977510 tid 1 2026-02-25T08:43:06.816 INFO:tasks.rados.rados.0.trial023.stdout:322: writing trial02342524-32 from 1732455 to 2320834 tid 2 2026-02-25T08:43:06.819 INFO:tasks.rados.rados.0.trial023.stdout:322: writing trial02342524-32 from 2779130 to 2924418 tid 3 2026-02-25T08:43:06.820 INFO:tasks.rados.rados.0.trial023.stdout:317: done (3 left) 2026-02-25T08:43:06.820 INFO:tasks.rados.rados.0.trial023.stdout:323: read oid 15 snap -1 2026-02-25T08:43:06.820 INFO:tasks.rados.rados.0.trial023.stdout:323: expect (ObjNum 162 snap 34 seq_num 162) 2026-02-25T08:43:06.820 INFO:tasks.rados.rados.0.trial023.stdout:324: read oid 12 snap -1 2026-02-25T08:43:06.820 INFO:tasks.rados.rados.0.trial023.stdout:324: expect (ObjNum 75 snap 3 seq_num 75) 2026-02-25T08:43:06.820 INFO:tasks.rados.rados.0.trial023.stdout:325: write (excl) oid 39 current snap is 40 2026-02-25T08:43:06.820 INFO:tasks.rados.rados.0.trial023.stdout:318: finishing write tid 2 to trial02342524-9 2026-02-25T08:43:06.820 INFO:tasks.rados.rados.0.trial023.stdout:325: seq_num 175 ranges {541225=644694,1907290=448390} 2026-02-25T08:43:06.824 INFO:tasks.rados.rados.0.trial023.stdout:325: writing trial02342524-39 from 541225 to 1185919 tid 1 2026-02-25T08:43:06.829 INFO:tasks.rados.rados.0.trial023.stdout:325: writing trial02342524-39 from 1907290 to 2355680 tid 2 2026-02-25T08:43:06.829 INFO:tasks.rados.rados.0.trial023.stdout:326: read oid 27 snap -1 2026-02-25T08:43:06.829 INFO:tasks.rados.rados.0.trial023.stdout:326: expect deleted 2026-02-25T08:43:06.829 INFO:tasks.rados.rados.0.trial023.stdout:327: snap_remove snap 31 2026-02-25T08:43:07.811 INFO:tasks.rados.rados.0.trial023.stdout:327: done (7 left) 2026-02-25T08:43:07.811 INFO:tasks.rados.rados.0.trial023.stdout:328: snap_remove snap 38 2026-02-25T08:43:07.920 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:43:07.991 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:43:07.993 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:43:07.993 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:43:08.480 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~88s 2026-02-25T08:43:08.740 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:08.812 INFO:tasks.rados.rados.0.trial023.stdout:328: done (7 left) 2026-02-25T08:43:08.813 INFO:tasks.rados.rados.0.trial023.stdout:329: read oid 19 snap -1 2026-02-25T08:43:08.813 INFO:tasks.rados.rados.0.trial023.stdout:329: expect (ObjNum 151 snap 26 seq_num 151) 2026-02-25T08:43:08.813 INFO:tasks.rados.rados.0.trial023.stdout:330: delete oid 45 current snap is 40 2026-02-25T08:43:08.813 INFO:tasks.rados.rados.0.trial023.stdout:318: finishing write tid 3 to trial02342524-9 2026-02-25T08:43:08.813 INFO:tasks.rados.rados.0.trial023.stdout:318: finishing write tid 4 to trial02342524-9 2026-02-25T08:43:08.813 INFO:tasks.rados.rados.0.trial023.stdout:318: finishing write tid 5 to trial02342524-9 2026-02-25T08:43:08.813 INFO:tasks.rados.rados.0.trial023.stdout:318: oid 9 updating version 0 to 79 2026-02-25T08:43:08.813 INFO:tasks.rados.rados.0.trial023.stdout:318: oid 9 version 79 is already newer than 79 2026-02-25T08:43:08.813 INFO:tasks.rados.rados.0.trial023.stdout:318: oid 9 version 79 is already newer than 79 2026-02-25T08:43:08.814 INFO:tasks.rados.rados.0.trial023.stdout:318: oid 9 version 79 is already newer than 79 2026-02-25T08:43:08.814 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 9 v 79 (ObjNum 172 snap 40 seq_num 172) dirty exists 2026-02-25T08:43:08.814 INFO:tasks.rados.rados.0.trial023.stdout:318: left oid 9 (ObjNum 172 snap 40 seq_num 172) 2026-02-25T08:43:08.814 INFO:tasks.rados.rados.0.trial023.stdout:323: expect (ObjNum 162 snap 34 seq_num 162) 2026-02-25T08:43:08.817 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:08.815+0000 7f8456a8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:08.820 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:08.819+0000 7f8456a8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:08.828 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1674582 2026-02-25T08:43:08.862 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:43:08.864 INFO:tasks.rados.rados.0.trial023.stdout:322: finishing write tid 1 to trial02342524-32 2026-02-25T08:43:08.864 INFO:tasks.rados.rados.0.trial023.stdout:324: expect (ObjNum 75 snap 3 seq_num 75) 2026-02-25T08:43:08.887 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:08.888 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:08.908 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1049120 2026-02-25T08:43:08.987 INFO:tasks.rados.rados.0.trial023.stdout:325: finishing write tid 1 to trial02342524-39 2026-02-25T08:43:08.988 INFO:tasks.rados.rados.0.trial023.stdout:318: done (8 left) 2026-02-25T08:43:08.988 INFO:tasks.rados.rados.0.trial023.stdout:321: done (7 left) 2026-02-25T08:43:08.988 INFO:tasks.rados.rados.0.trial023.stdout:323: done (6 left) 2026-02-25T08:43:08.988 INFO:tasks.rados.rados.0.trial023.stdout:324: done (5 left) 2026-02-25T08:43:08.988 INFO:tasks.rados.rados.0.trial023.stdout:326: done (4 left) 2026-02-25T08:43:08.988 INFO:tasks.rados.rados.0.trial023.stdout:330: done (3 left) 2026-02-25T08:43:08.988 INFO:tasks.rados.rados.0.trial023.stdout:331: read oid 1 snap -1 2026-02-25T08:43:08.988 INFO:tasks.rados.rados.0.trial023.stdout:331: expect (ObjNum 152 snap 26 seq_num 152) 2026-02-25T08:43:08.988 INFO:tasks.rados.rados.0.trial023.stdout:332: snap_remove snap 35 2026-02-25T08:43:09.818 INFO:tasks.rados.rados.0.trial023.stdout:332: done (4 left) 2026-02-25T08:43:09.818 INFO:tasks.rados.rados.0.trial023.stdout:333: snap_create 2026-02-25T08:43:09.818 INFO:tasks.rados.rados.0.trial023.stdout:322: finishing write tid 2 to trial02342524-32 2026-02-25T08:43:09.818 INFO:tasks.rados.rados.0.trial023.stdout:322: finishing write tid 3 to trial02342524-32 2026-02-25T08:43:09.818 INFO:tasks.rados.rados.0.trial023.stdout:322: finishing write tid 4 to trial02342524-32 2026-02-25T08:43:09.818 INFO:tasks.rados.rados.0.trial023.stdout:322: finishing write tid 5 to trial02342524-32 2026-02-25T08:43:09.818 INFO:tasks.rados.rados.0.trial023.stdout:322: oid 32 updating version 0 to 50 2026-02-25T08:43:09.819 INFO:tasks.rados.rados.0.trial023.stdout:322: oid 32 version 50 is already newer than 47 2026-02-25T08:43:09.819 INFO:tasks.rados.rados.0.trial023.stdout:322: oid 32 version 50 is already newer than 50 2026-02-25T08:43:09.819 INFO:tasks.rados.rados.0.trial023.stdout:322: oid 32 version 50 is already newer than 50 2026-02-25T08:43:09.819 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 32 v 50 (ObjNum 174 snap 40 seq_num 174) dirty exists 2026-02-25T08:43:09.819 INFO:tasks.rados.rados.0.trial023.stdout:322: left oid 32 (ObjNum 174 snap 40 seq_num 174) 2026-02-25T08:43:09.819 INFO:tasks.rados.rados.0.trial023.stdout:325: finishing write tid 2 to trial02342524-39 2026-02-25T08:43:09.819 INFO:tasks.rados.rados.0.trial023.stdout:325: finishing write tid 3 to trial02342524-39 2026-02-25T08:43:09.819 INFO:tasks.rados.rados.0.trial023.stdout:325: finishing write tid 4 to trial02342524-39 2026-02-25T08:43:09.819 INFO:tasks.rados.rados.0.trial023.stdout:325: oid 39 updating version 0 to 24 2026-02-25T08:43:09.819 INFO:tasks.rados.rados.0.trial023.stdout:325: oid 39 updating version 24 to 26 2026-02-25T08:43:09.819 INFO:tasks.rados.rados.0.trial023.stdout:325: oid 39 version 26 is already newer than 26 2026-02-25T08:43:09.819 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 39 v 26 (ObjNum 175 snap 40 seq_num 175) dirty exists 2026-02-25T08:43:09.820 INFO:tasks.rados.rados.0.trial023.stdout:325: left oid 39 (ObjNum 175 snap 40 seq_num 175) 2026-02-25T08:43:09.820 INFO:tasks.rados.rados.0.trial023.stdout:329: expect (ObjNum 151 snap 26 seq_num 151) 2026-02-25T08:43:09.820 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 761387 2026-02-25T08:43:09.893 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:43:09.933 INFO:tasks.rados.rados.0.trial023.stdout:331: expect (ObjNum 152 snap 26 seq_num 152) 2026-02-25T08:43:09.997 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1882251 2026-02-25T08:43:10.822 INFO:tasks.rados.rados.0.trial023.stdout:322: done (4 left) 2026-02-25T08:43:10.822 INFO:tasks.rados.rados.0.trial023.stdout:325: done (3 left) 2026-02-25T08:43:10.822 INFO:tasks.rados.rados.0.trial023.stdout:329: done (2 left) 2026-02-25T08:43:10.822 INFO:tasks.rados.rados.0.trial023.stdout:331: done (1 left) 2026-02-25T08:43:10.822 INFO:tasks.rados.rados.0.trial023.stdout:333: done (0 left) 2026-02-25T08:43:10.822 INFO:tasks.rados.rados.0.trial023.stdout:334: snap_create 2026-02-25T08:43:10.889 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:10.967 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:10.965+0000 7f53245fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:10.969 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:10.968+0000 7f53245fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:10.986 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:43:11.014 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:11.015 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:11.825 INFO:tasks.rados.rados.0.trial023.stdout:334: done (0 left) 2026-02-25T08:43:11.825 INFO:tasks.rados.rados.0.trial023.stdout:335: write (excl) oid 14 current snap is 42 2026-02-25T08:43:11.826 INFO:tasks.rados.rados.0.trial023.stdout:335: seq_num 177 ranges {574839=795192,1907901=452059} 2026-02-25T08:43:11.830 INFO:tasks.rados.rados.0.trial023.stdout:335: writing trial02342524-14 from 574839 to 1370031 tid 1 2026-02-25T08:43:11.834 INFO:tasks.rados.rados.0.trial023.stdout:335: writing trial02342524-14 from 1907901 to 2359960 tid 2 2026-02-25T08:43:11.834 INFO:tasks.rados.rados.0.trial023.stdout:336: write (excl) oid 17 current snap is 42 2026-02-25T08:43:11.834 INFO:tasks.rados.rados.0.trial023.stdout:336: seq_num 178 ranges {591646=470441,1566383=595717,2888822=102824} 2026-02-25T08:43:11.838 INFO:tasks.rados.rados.0.trial023.stdout:336: writing trial02342524-17 from 591646 to 1062087 tid 1 2026-02-25T08:43:11.842 INFO:tasks.rados.rados.0.trial023.stdout:336: writing trial02342524-17 from 1566383 to 2162100 tid 2 2026-02-25T08:43:11.845 INFO:tasks.rados.rados.0.trial023.stdout:336: writing trial02342524-17 from 2888822 to 2991646 tid 3 2026-02-25T08:43:11.845 INFO:tasks.rados.rados.0.trial023.stdout:337: write oid 24 current snap is 42 2026-02-25T08:43:11.845 INFO:tasks.rados.rados.0.trial023.stdout:337: seq_num 179 ranges {608453=545690,1624865=455728,2632598=375855} 2026-02-25T08:43:11.849 INFO:tasks.rados.rados.0.trial023.stdout:337: writing trial02342524-24 from 608453 to 1154143 tid 1 2026-02-25T08:43:11.853 INFO:tasks.rados.rados.0.trial023.stdout:337: writing trial02342524-24 from 1624865 to 2080593 tid 2 2026-02-25T08:43:11.856 INFO:tasks.rados.rados.0.trial023.stdout:337: writing trial02342524-24 from 2632598 to 3008453 tid 3 2026-02-25T08:43:11.857 INFO:tasks.rados.rados.0.trial023.stdout:338: snap_remove snap 34 2026-02-25T08:43:12.829 INFO:tasks.rados.rados.0.trial023.stdout:338: done (3 left) 2026-02-25T08:43:12.829 INFO:tasks.rados.rados.0.trial023.stdout:339: snap_create 2026-02-25T08:43:12.829 INFO:tasks.rados.rados.0.trial023.stdout:337: finishing write tid 1 to trial02342524-24 2026-02-25T08:43:12.829 INFO:tasks.rados.rados.0.trial023.stdout:336: finishing write tid 1 to trial02342524-17 2026-02-25T08:43:12.829 INFO:tasks.rados.rados.0.trial023.stdout:337: finishing write tid 2 to trial02342524-24 2026-02-25T08:43:12.829 INFO:tasks.rados.rados.0.trial023.stdout:337: finishing write tid 3 to trial02342524-24 2026-02-25T08:43:12.830 INFO:tasks.rados.rados.0.trial023.stdout:336: finishing write tid 2 to trial02342524-17 2026-02-25T08:43:12.830 INFO:tasks.rados.rados.0.trial023.stdout:336: finishing write tid 3 to trial02342524-17 2026-02-25T08:43:12.830 INFO:tasks.rados.rados.0.trial023.stdout:336: finishing write tid 4 to trial02342524-17 2026-02-25T08:43:12.830 INFO:tasks.rados.rados.0.trial023.stdout:336: finishing write tid 5 to trial02342524-17 2026-02-25T08:43:12.830 INFO:tasks.rados.rados.0.trial023.stdout:336: oid 17 updating version 0 to 9 2026-02-25T08:43:12.830 INFO:tasks.rados.rados.0.trial023.stdout:336: oid 17 updating version 9 to 12 2026-02-25T08:43:12.830 INFO:tasks.rados.rados.0.trial023.stdout:336: oid 17 version 12 is already newer than 12 2026-02-25T08:43:12.830 INFO:tasks.rados.rados.0.trial023.stdout:336: oid 17 version 12 is already newer than 12 2026-02-25T08:43:12.830 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 17 v 12 (ObjNum 178 snap 42 seq_num 178) dirty exists 2026-02-25T08:43:12.830 INFO:tasks.rados.rados.0.trial023.stdout:336: left oid 17 (ObjNum 178 snap 42 seq_num 178) 2026-02-25T08:43:12.830 INFO:tasks.rados.rados.0.trial023.stdout:337: finishing write tid 4 to trial02342524-24 2026-02-25T08:43:12.830 INFO:tasks.rados.rados.0.trial023.stdout:337: finishing write tid 5 to trial02342524-24 2026-02-25T08:43:12.831 INFO:tasks.rados.rados.0.trial023.stdout:337: oid 24 updating version 0 to 54 2026-02-25T08:43:12.831 INFO:tasks.rados.rados.0.trial023.stdout:337: oid 24 version 54 is already newer than 54 2026-02-25T08:43:12.831 INFO:tasks.rados.rados.0.trial023.stdout:337: oid 24 version 54 is already newer than 54 2026-02-25T08:43:12.831 INFO:tasks.rados.rados.0.trial023.stdout:337: oid 24 version 54 is already newer than 54 2026-02-25T08:43:12.831 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 24 v 54 (ObjNum 179 snap 42 seq_num 179) dirty exists 2026-02-25T08:43:12.831 INFO:tasks.rados.rados.0.trial023.stdout:337: left oid 24 (ObjNum 179 snap 42 seq_num 179) 2026-02-25T08:43:12.831 INFO:tasks.rados.rados.0.trial023.stdout:335: finishing write tid 1 to trial02342524-14 2026-02-25T08:43:12.831 INFO:tasks.rados.rados.0.trial023.stdout:335: finishing write tid 2 to trial02342524-14 2026-02-25T08:43:12.831 INFO:tasks.rados.rados.0.trial023.stdout:335: finishing write tid 3 to trial02342524-14 2026-02-25T08:43:12.831 INFO:tasks.rados.rados.0.trial023.stdout:335: finishing write tid 4 to trial02342524-14 2026-02-25T08:43:12.831 INFO:tasks.rados.rados.0.trial023.stdout:335: oid 14 updating version 0 to 23 2026-02-25T08:43:12.831 INFO:tasks.rados.rados.0.trial023.stdout:335: oid 14 version 23 is already newer than 23 2026-02-25T08:43:12.831 INFO:tasks.rados.rados.0.trial023.stdout:335: oid 14 version 23 is already newer than 21 2026-02-25T08:43:12.832 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 14 v 23 (ObjNum 177 snap 42 seq_num 177) dirty exists 2026-02-25T08:43:12.832 INFO:tasks.rados.rados.0.trial023.stdout:335: left oid 14 (ObjNum 177 snap 42 seq_num 177) 2026-02-25T08:43:12.995 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:43:13.015 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:13.066 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:43:13.069 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:43:13.070 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:43:13.112 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:13.109+0000 7f3884d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:13.114 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:13.113+0000 7f3884d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:13.128 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:43:13.155 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:13.156 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:13.831 INFO:tasks.rados.rados.0.trial023.stdout:335: done (3 left) 2026-02-25T08:43:13.832 INFO:tasks.rados.rados.0.trial023.stdout:336: done (2 left) 2026-02-25T08:43:13.832 INFO:tasks.rados.rados.0.trial023.stdout:337: done (1 left) 2026-02-25T08:43:13.832 INFO:tasks.rados.rados.0.trial023.stdout:339: done (0 left) 2026-02-25T08:43:13.832 INFO:tasks.rados.rados.0.trial023.stdout:340: delete oid 12 current snap is 43 2026-02-25T08:43:13.881 INFO:tasks.rados.rados.0.trial023.stdout:340: done (0 left) 2026-02-25T08:43:13.881 INFO:tasks.rados.rados.0.trial023.stdout:341: write (excl) oid 43 current snap is 43 2026-02-25T08:43:13.881 INFO:tasks.rados.rados.0.trial023.stdout:341: seq_num 181 ranges {642067=696188,2025476=459397,2971091=70976} 2026-02-25T08:43:13.886 INFO:tasks.rados.rados.0.trial023.stdout:341: writing trial02342524-43 from 642067 to 1338255 tid 1 2026-02-25T08:43:13.890 INFO:tasks.rados.rados.0.trial023.stdout:341: writing trial02342524-43 from 2025476 to 2484873 tid 2 2026-02-25T08:43:13.892 INFO:tasks.rados.rados.0.trial023.stdout:341: writing trial02342524-43 from 2971091 to 3042067 tid 3 2026-02-25T08:43:13.892 INFO:tasks.rados.rados.0.trial023.stdout:342: snap_create 2026-02-25T08:43:13.912 INFO:tasks.rados.rados.0.trial023.stdout:341: finishing write tid 1 to trial02342524-43 2026-02-25T08:43:13.930 INFO:tasks.rados.rados.0.trial023.stdout:341: finishing write tid 2 to trial02342524-43 2026-02-25T08:43:13.935 INFO:tasks.rados.rados.0.trial023.stdout:341: finishing write tid 3 to trial02342524-43 2026-02-25T08:43:13.940 INFO:tasks.rados.rados.0.trial023.stdout:341: finishing write tid 4 to trial02342524-43 2026-02-25T08:43:13.942 INFO:tasks.rados.rados.0.trial023.stdout:341: finishing write tid 5 to trial02342524-43 2026-02-25T08:43:13.942 INFO:tasks.rados.rados.0.trial023.stdout:341: oid 43 updating version 0 to 12 2026-02-25T08:43:13.942 INFO:tasks.rados.rados.0.trial023.stdout:341: oid 43 version 12 is already newer than 12 2026-02-25T08:43:13.942 INFO:tasks.rados.rados.0.trial023.stdout:341: oid 43 version 12 is already newer than 9 2026-02-25T08:43:13.942 INFO:tasks.rados.rados.0.trial023.stdout:341: oid 43 version 12 is already newer than 12 2026-02-25T08:43:13.943 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 43 v 12 (ObjNum 181 snap 43 seq_num 181) dirty exists 2026-02-25T08:43:13.943 INFO:tasks.rados.rados.0.trial023.stdout:341: left oid 43 (ObjNum 181 snap 43 seq_num 181) 2026-02-25T08:43:13.983 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~94s 2026-02-25T08:43:14.807 INFO:tasks.rados.rados.0.trial023.stdout:341: done (1 left) 2026-02-25T08:43:14.807 INFO:tasks.rados.rados.0.trial023.stdout:342: done (0 left) 2026-02-25T08:43:14.807 INFO:tasks.rados.rados.0.trial023.stdout:343: snap_create 2026-02-25T08:43:15.157 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:15.234 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:15.231+0000 7f4512611640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:15.236 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:15.235+0000 7f4512611640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:15.257 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:43:15.285 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:15.285 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:15.808 INFO:tasks.rados.rados.0.trial023.stdout:343: done (0 left) 2026-02-25T08:43:15.809 INFO:tasks.rados.rados.0.trial023.stdout:344: write oid 45 current snap is 45 2026-02-25T08:43:15.809 INFO:tasks.rados.rados.0.trial023.stdout:344: seq_num 182 ranges {658874=771437,2083958=603055} 2026-02-25T08:43:15.813 INFO:tasks.rados.rados.0.trial023.stdout:344: writing trial02342524-45 from 658874 to 1430311 tid 1 2026-02-25T08:43:15.818 INFO:tasks.rados.rados.0.trial023.stdout:344: writing trial02342524-45 from 2083958 to 2687013 tid 2 2026-02-25T08:43:15.818 INFO:tasks.rados.rados.0.trial023.stdout:345: snap_create 2026-02-25T08:43:15.840 INFO:tasks.rados.rados.0.trial023.stdout:344: finishing write tid 1 to trial02342524-45 2026-02-25T08:43:15.846 INFO:tasks.rados.rados.0.trial023.stdout:344: finishing write tid 2 to trial02342524-45 2026-02-25T08:43:15.851 INFO:tasks.rados.rados.0.trial023.stdout:344: finishing write tid 3 to trial02342524-45 2026-02-25T08:43:15.853 INFO:tasks.rados.rados.0.trial023.stdout:344: finishing write tid 4 to trial02342524-45 2026-02-25T08:43:15.853 INFO:tasks.rados.rados.0.trial023.stdout:344: oid 45 updating version 0 to 100 2026-02-25T08:43:15.853 INFO:tasks.rados.rados.0.trial023.stdout:344: oid 45 version 100 is already newer than 100 2026-02-25T08:43:15.853 INFO:tasks.rados.rados.0.trial023.stdout:344: oid 45 version 100 is already newer than 100 2026-02-25T08:43:15.853 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 45 v 100 (ObjNum 182 snap 45 seq_num 182) dirty exists 2026-02-25T08:43:15.853 INFO:tasks.rados.rados.0.trial023.stdout:344: left oid 45 (ObjNum 182 snap 45 seq_num 182) 2026-02-25T08:43:16.809 INFO:tasks.rados.rados.0.trial023.stdout:344: done (1 left) 2026-02-25T08:43:16.809 INFO:tasks.rados.rados.0.trial023.stdout:345: done (0 left) 2026-02-25T08:43:16.809 INFO:tasks.rados.rados.0.trial023.stdout:346: snap_create 2026-02-25T08:43:17.286 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:17.364 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:17.361+0000 7f581881a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:17.366 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:17.365+0000 7f581881a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:17.381 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:43:17.407 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:17.408 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:17.810 INFO:tasks.rados.rados.0.trial023.stdout:346: done (0 left) 2026-02-25T08:43:17.810 INFO:tasks.rados.rados.0.trial023.stdout:347: delete oid 13 current snap is 47 2026-02-25T08:43:17.849 INFO:tasks.rados.rados.0.trial023.stdout:347: done (0 left) 2026-02-25T08:43:17.849 INFO:tasks.rados.rados.0.trial023.stdout:348: write (excl) oid 36 current snap is 47 2026-02-25T08:43:17.849 INFO:tasks.rados.rados.0.trial023.stdout:348: seq_num 184 ranges {692488=638288,1917275=606724} 2026-02-25T08:43:17.854 INFO:tasks.rados.rados.0.trial023.stdout:348: writing trial02342524-36 from 692488 to 1330776 tid 1 2026-02-25T08:43:17.858 INFO:tasks.rados.rados.0.trial023.stdout:348: writing trial02342524-36 from 1917275 to 2523999 tid 2 2026-02-25T08:43:17.859 INFO:tasks.rados.rados.0.trial023.stdout:349: snap_create 2026-02-25T08:43:17.880 INFO:tasks.rados.rados.0.trial023.stdout:348: finishing write tid 1 to trial02342524-36 2026-02-25T08:43:17.895 INFO:tasks.rados.rados.0.trial023.stdout:348: finishing write tid 2 to trial02342524-36 2026-02-25T08:43:17.899 INFO:tasks.rados.rados.0.trial023.stdout:348: finishing write tid 3 to trial02342524-36 2026-02-25T08:43:17.901 INFO:tasks.rados.rados.0.trial023.stdout:348: finishing write tid 4 to trial02342524-36 2026-02-25T08:43:17.901 INFO:tasks.rados.rados.0.trial023.stdout:348: oid 36 updating version 0 to 45 2026-02-25T08:43:17.902 INFO:tasks.rados.rados.0.trial023.stdout:348: oid 36 updating version 45 to 47 2026-02-25T08:43:17.902 INFO:tasks.rados.rados.0.trial023.stdout:348: oid 36 version 47 is already newer than 47 2026-02-25T08:43:17.902 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 36 v 47 (ObjNum 184 snap 47 seq_num 184) dirty exists 2026-02-25T08:43:17.902 INFO:tasks.rados.rados.0.trial023.stdout:348: left oid 36 (ObjNum 184 snap 47 seq_num 184) 2026-02-25T08:43:18.071 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:43:18.140 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:43:18.143 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:43:18.143 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:43:18.816 INFO:tasks.rados.rados.0.trial023.stdout:348: done (1 left) 2026-02-25T08:43:18.816 INFO:tasks.rados.rados.0.trial023.stdout:349: done (0 left) 2026-02-25T08:43:18.816 INFO:tasks.rados.rados.0.trial023.stdout:350: read oid 48 snap -1 2026-02-25T08:43:18.816 INFO:tasks.rados.rados.0.trial023.stdout:350: expect deleted 2026-02-25T08:43:18.816 INFO:tasks.rados.rados.0.trial023.stdout:351: write (excl) oid 36 current snap is 48 2026-02-25T08:43:18.817 INFO:tasks.rados.rados.0.trial023.stdout:351: seq_num 185 ranges {709295=713537,1859404=750382} 2026-02-25T08:43:18.821 INFO:tasks.rados.rados.0.trial023.stdout:351: writing trial02342524-36 from 709295 to 1422832 tid 1 2026-02-25T08:43:18.826 INFO:tasks.rados.rados.0.trial023.stdout:351: writing trial02342524-36 from 1859404 to 2609786 tid 2 2026-02-25T08:43:18.826 INFO:tasks.rados.rados.0.trial023.stdout:352: snap_remove snap 42 2026-02-25T08:43:19.409 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:19.487 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~99s 2026-02-25T08:43:19.489 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:19.485+0000 7f8e3bd9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:19.491 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:19.489+0000 7f8e3bd9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:19.506 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:43:19.533 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:19.533 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:19.817 INFO:tasks.rados.rados.0.trial023.stdout:352: done (2 left) 2026-02-25T08:43:19.817 INFO:tasks.rados.rados.0.trial023.stdout:353: write oid 50 current snap is 48 2026-02-25T08:43:19.817 INFO:tasks.rados.rados.0.trial023.stdout:353: seq_num 186 ranges {726102=788786,1917886=610393,3108206=17896} 2026-02-25T08:43:19.823 INFO:tasks.rados.rados.0.trial023.stdout:353: writing trial02342524-50 from 726102 to 1514888 tid 1 2026-02-25T08:43:19.827 INFO:tasks.rados.rados.0.trial023.stdout:353: writing trial02342524-50 from 1917886 to 2528279 tid 2 2026-02-25T08:43:19.829 INFO:tasks.rados.rados.0.trial023.stdout:353: writing trial02342524-50 from 3108206 to 3126102 tid 3 2026-02-25T08:43:19.830 INFO:tasks.rados.rados.0.trial023.stdout:354: read oid 1 snap -1 2026-02-25T08:43:19.830 INFO:tasks.rados.rados.0.trial023.stdout:354: expect (ObjNum 152 snap 26 seq_num 152) 2026-02-25T08:43:19.830 INFO:tasks.rados.rados.0.trial023.stdout:355: read oid 45 snap 27 2026-02-25T08:43:19.830 INFO:tasks.rados.rados.0.trial023.stdout:355: expect (ObjNum 138 snap 19 seq_num 138) 2026-02-25T08:43:19.830 INFO:tasks.rados.rados.0.trial023.stdout:356: snap_remove snap 45 2026-02-25T08:43:20.822 INFO:tasks.rados.rados.0.trial023.stdout:356: done (5 left) 2026-02-25T08:43:20.822 INFO:tasks.rados.rados.0.trial023.stdout:357: delete oid 6 current snap is 48 2026-02-25T08:43:20.823 INFO:tasks.rados.rados.0.trial023.stdout:351: finishing write tid 1 to trial02342524-36 2026-02-25T08:43:20.823 INFO:tasks.rados.rados.0.trial023.stdout:351: finishing write tid 2 to trial02342524-36 2026-02-25T08:43:20.823 INFO:tasks.rados.rados.0.trial023.stdout:351: finishing write tid 3 to trial02342524-36 2026-02-25T08:43:20.823 INFO:tasks.rados.rados.0.trial023.stdout:351: finishing write tid 4 to trial02342524-36 2026-02-25T08:43:20.823 INFO:tasks.rados.rados.0.trial023.stdout:351: oid 36 updating version 0 to 51 2026-02-25T08:43:20.823 INFO:tasks.rados.rados.0.trial023.stdout:351: oid 36 version 51 is already newer than 49 2026-02-25T08:43:20.823 INFO:tasks.rados.rados.0.trial023.stdout:351: oid 36 version 51 is already newer than 51 2026-02-25T08:43:20.823 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 36 v 51 (ObjNum 185 snap 48 seq_num 185) dirty exists 2026-02-25T08:43:20.823 INFO:tasks.rados.rados.0.trial023.stdout:351: left oid 36 (ObjNum 185 snap 48 seq_num 185) 2026-02-25T08:43:20.823 INFO:tasks.rados.rados.0.trial023.stdout:354: expect (ObjNum 152 snap 26 seq_num 152) 2026-02-25T08:43:20.823 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:43:20.887 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1746538 2026-02-25T08:43:20.898 INFO:tasks.rados.rados.0.trial023.stdout:353: finishing write tid 1 to trial02342524-50 2026-02-25T08:43:20.898 INFO:tasks.rados.rados.0.trial023.stdout:355: expect (ObjNum 138 snap 19 seq_num 138) 2026-02-25T08:43:20.952 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 879383 2026-02-25T08:43:20.970 INFO:tasks.rados.rados.0.trial023.stdout:353: finishing write tid 2 to trial02342524-50 2026-02-25T08:43:20.970 INFO:tasks.rados.rados.0.trial023.stdout:350: done (5 left) 2026-02-25T08:43:20.970 INFO:tasks.rados.rados.0.trial023.stdout:351: done (4 left) 2026-02-25T08:43:20.970 INFO:tasks.rados.rados.0.trial023.stdout:354: done (3 left) 2026-02-25T08:43:20.970 INFO:tasks.rados.rados.0.trial023.stdout:355: done (2 left) 2026-02-25T08:43:20.970 INFO:tasks.rados.rados.0.trial023.stdout:357: done (1 left) 2026-02-25T08:43:20.970 INFO:tasks.rados.rados.0.trial023.stdout:358: read oid 44 snap 37 2026-02-25T08:43:20.971 INFO:tasks.rados.rados.0.trial023.stdout:358: expect (ObjNum 142 snap 20 seq_num 142) 2026-02-25T08:43:20.971 INFO:tasks.rados.rados.0.trial023.stdout:359: write (excl) oid 9 current snap is 48 2026-02-25T08:43:20.971 INFO:tasks.rados.rados.0.trial023.stdout:353: finishing write tid 3 to trial02342524-50 2026-02-25T08:43:20.971 INFO:tasks.rados.rados.0.trial023.stdout:353: finishing write tid 4 to trial02342524-50 2026-02-25T08:43:20.971 INFO:tasks.rados.rados.0.trial023.stdout:353: finishing write tid 5 to trial02342524-50 2026-02-25T08:43:20.971 INFO:tasks.rados.rados.0.trial023.stdout:353: oid 50 updating version 0 to 105 2026-02-25T08:43:20.971 INFO:tasks.rados.rados.0.trial023.stdout:353: oid 50 version 105 is already newer than 105 2026-02-25T08:43:20.971 INFO:tasks.rados.rados.0.trial023.stdout:353: oid 50 version 105 is already newer than 105 2026-02-25T08:43:20.971 INFO:tasks.rados.rados.0.trial023.stdout:353: oid 50 version 105 is already newer than 105 2026-02-25T08:43:20.971 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 50 v 105 (ObjNum 186 snap 48 seq_num 186) dirty exists 2026-02-25T08:43:20.971 INFO:tasks.rados.rados.0.trial023.stdout:353: left oid 50 (ObjNum 186 snap 48 seq_num 186) 2026-02-25T08:43:20.971 INFO:tasks.rados.rados.0.trial023.stdout:359: seq_num 188 ranges {759716=539284,2034850=614062} 2026-02-25T08:43:20.974 INFO:tasks.rados.rados.0.trial023.stdout:359: writing trial02342524-9 from 759716 to 1299000 tid 1 2026-02-25T08:43:20.980 INFO:tasks.rados.rados.0.trial023.stdout:359: writing trial02342524-9 from 2034850 to 2648912 tid 2 2026-02-25T08:43:20.980 INFO:tasks.rados.rados.0.trial023.stdout:353: done (2 left) 2026-02-25T08:43:20.980 INFO:tasks.rados.rados.0.trial023.stdout:360: read oid 50 snap 33 2026-02-25T08:43:20.980 INFO:tasks.rados.rados.0.trial023.stdout:360: expect (ObjNum 49 snap 0 seq_num 49) 2026-02-25T08:43:20.981 INFO:tasks.rados.rados.0.trial023.stdout:361: read oid 16 snap 40 2026-02-25T08:43:20.981 INFO:tasks.rados.rados.0.trial023.stdout:361: expect (ObjNum 90 snap 9 seq_num 90) 2026-02-25T08:43:20.981 INFO:tasks.rados.rados.0.trial023.stdout:362: snap_remove snap 46 2026-02-25T08:43:21.535 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:21.611 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:21.608+0000 7f927376f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:21.613 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:21.611+0000 7f927376f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:21.631 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:43:21.658 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:21.659 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:21.825 INFO:tasks.rados.rados.0.trial023.stdout:362: done (4 left) 2026-02-25T08:43:21.825 INFO:tasks.rados.rados.0.trial023.stdout:363: snap_remove snap 27 2026-02-25T08:43:22.828 INFO:tasks.rados.rados.0.trial023.stdout:363: done (4 left) 2026-02-25T08:43:22.828 INFO:tasks.rados.rados.0.trial023.stdout:364: read oid 26 snap -1 2026-02-25T08:43:22.828 INFO:tasks.rados.rados.0.trial023.stdout:364: expect (ObjNum 148 snap 25 seq_num 148) 2026-02-25T08:43:22.829 INFO:tasks.rados.rados.0.trial023.stdout:365: write oid 11 current snap is 48 2026-02-25T08:43:22.829 INFO:tasks.rados.rados.0.trial023.stdout:365: seq_num 189 ranges {776523=614533,1976979=757720} 2026-02-25T08:43:22.833 INFO:tasks.rados.rados.0.trial023.stdout:365: writing trial02342524-11 from 776523 to 1391056 tid 1 2026-02-25T08:43:22.839 INFO:tasks.rados.rados.0.trial023.stdout:365: writing trial02342524-11 from 1976979 to 2734699 tid 2 2026-02-25T08:43:22.839 INFO:tasks.rados.rados.0.trial023.stdout:366: write (excl) oid 49 current snap is 48 2026-02-25T08:43:22.839 INFO:tasks.rados.rados.0.trial023.stdout:366: seq_num 190 ranges {793330=689782,2035461=617731,3101545=91785} 2026-02-25T08:43:22.845 INFO:tasks.rados.rados.0.trial023.stdout:366: writing trial02342524-49 from 793330 to 1483112 tid 1 2026-02-25T08:43:22.849 INFO:tasks.rados.rados.0.trial023.stdout:366: writing trial02342524-49 from 2035461 to 2653192 tid 2 2026-02-25T08:43:22.851 INFO:tasks.rados.rados.0.trial023.stdout:366: writing trial02342524-49 from 3101545 to 3193330 tid 3 2026-02-25T08:43:22.852 INFO:tasks.rados.rados.0.trial023.stdout:367: read oid 12 snap -1 2026-02-25T08:43:22.852 INFO:tasks.rados.rados.0.trial023.stdout:367: expect deleted 2026-02-25T08:43:22.852 INFO:tasks.rados.rados.0.trial023.stdout:368: snap_remove snap 43 2026-02-25T08:43:23.144 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:43:23.216 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:43:23.219 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:43:23.220 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:43:23.660 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:23.738 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:23.735+0000 7f04ac974640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:23.741 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:23.738+0000 7f04ac974640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:23.756 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:43:23.783 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:23.784 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:23.832 INFO:tasks.rados.rados.0.trial023.stdout:368: done (8 left) 2026-02-25T08:43:23.832 INFO:tasks.rados.rados.0.trial023.stdout:369: read oid 39 snap 41 2026-02-25T08:43:23.832 INFO:tasks.rados.rados.0.trial023.stdout:369: expect (ObjNum 175 snap 40 seq_num 175) 2026-02-25T08:43:23.832 INFO:tasks.rados.rados.0.trial023.stdout:370: write oid 29 current snap is 48 2026-02-25T08:43:23.832 INFO:tasks.rados.rados.0.trial023.stdout:370: seq_num 191 ranges {410137=481384,1410296=761389,2728968=481169} 2026-02-25T08:43:23.835 INFO:tasks.rados.rados.0.trial023.stdout:370: writing trial02342524-29 from 410137 to 891521 tid 1 2026-02-25T08:43:23.840 INFO:tasks.rados.rados.0.trial023.stdout:370: writing trial02342524-29 from 1410296 to 2171685 tid 2 2026-02-25T08:43:23.844 INFO:tasks.rados.rados.0.trial023.stdout:370: writing trial02342524-29 from 2728968 to 3210137 tid 3 2026-02-25T08:43:23.845 INFO:tasks.rados.rados.0.trial023.stdout:371: snap_remove snap 44 2026-02-25T08:43:24.809 INFO:tasks.rados.rados.0.trial023.stdout:371: done (10 left) 2026-02-25T08:43:24.810 INFO:tasks.rados.rados.0.trial023.stdout:372: read oid 5 snap -1 2026-02-25T08:43:24.810 INFO:tasks.rados.rados.0.trial023.stdout:372: expect (ObjNum 4 snap 0 seq_num 4) 2026-02-25T08:43:24.810 INFO:tasks.rados.rados.0.trial023.stdout:373: snap_remove snap 47 2026-02-25T08:43:24.991 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~105s 2026-02-25T08:43:25.786 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:25.810 INFO:tasks.rados.rados.0.trial023.stdout:373: done (11 left) 2026-02-25T08:43:25.810 INFO:tasks.rados.rados.0.trial023.stdout:374: read oid 34 snap -1 2026-02-25T08:43:25.810 INFO:tasks.rados.rados.0.trial023.stdout:374: expect (ObjNum 161 snap 34 seq_num 161) 2026-02-25T08:43:25.810 INFO:tasks.rados.rados.0.trial023.stdout:375: snap_create 2026-02-25T08:43:25.811 INFO:tasks.rados.rados.0.trial023.stdout:358: expect (ObjNum 142 snap 20 seq_num 142) 2026-02-25T08:43:25.811 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:43:25.867 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:25.863+0000 7f396345a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:25.867 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:43:25.869 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:25.867+0000 7f396345a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:25.884 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:43:25.895 INFO:tasks.rados.rados.0.trial023.stdout:360: expect (ObjNum 49 snap 0 seq_num 49) 2026-02-25T08:43:25.904 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2477848 2026-02-25T08:43:25.907 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:43:25.921 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:25.922 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:25.931 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:43:25.954 INFO:tasks.rados.rados.0.trial023.stdout:361: expect (ObjNum 90 snap 9 seq_num 90) 2026-02-25T08:43:26.098 INFO:tasks.rados.rados.0.trial023.stdout:359: finishing write tid 1 to trial02342524-9 2026-02-25T08:43:26.098 INFO:tasks.rados.rados.0.trial023.stdout:359: finishing write tid 2 to trial02342524-9 2026-02-25T08:43:26.098 INFO:tasks.rados.rados.0.trial023.stdout:359: finishing write tid 3 to trial02342524-9 2026-02-25T08:43:26.098 INFO:tasks.rados.rados.0.trial023.stdout:359: finishing write tid 4 to trial02342524-9 2026-02-25T08:43:26.098 INFO:tasks.rados.rados.0.trial023.stdout:359: oid 9 updating version 0 to 91 2026-02-25T08:43:26.098 INFO:tasks.rados.rados.0.trial023.stdout:359: oid 9 version 91 is already newer than 89 2026-02-25T08:43:26.098 INFO:tasks.rados.rados.0.trial023.stdout:359: oid 9 version 91 is already newer than 91 2026-02-25T08:43:26.098 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 9 v 91 (ObjNum 188 snap 48 seq_num 188) dirty exists 2026-02-25T08:43:26.099 INFO:tasks.rados.rados.0.trial023.stdout:359: left oid 9 (ObjNum 188 snap 48 seq_num 188) 2026-02-25T08:43:26.099 INFO:tasks.rados.rados.0.trial023.stdout:364: expect (ObjNum 148 snap 25 seq_num 148) 2026-02-25T08:43:26.138 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:43:26.198 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2131121 2026-02-25T08:43:26.210 INFO:tasks.rados.rados.0.trial023.stdout:365: finishing write tid 1 to trial02342524-11 2026-02-25T08:43:26.210 INFO:tasks.rados.rados.0.trial023.stdout:365: finishing write tid 2 to trial02342524-11 2026-02-25T08:43:26.210 INFO:tasks.rados.rados.0.trial023.stdout:365: finishing write tid 3 to trial02342524-11 2026-02-25T08:43:26.210 INFO:tasks.rados.rados.0.trial023.stdout:365: finishing write tid 4 to trial02342524-11 2026-02-25T08:43:26.210 INFO:tasks.rados.rados.0.trial023.stdout:365: oid 11 updating version 0 to 108 2026-02-25T08:43:26.211 INFO:tasks.rados.rados.0.trial023.stdout:365: oid 11 version 108 is already newer than 108 2026-02-25T08:43:26.211 INFO:tasks.rados.rados.0.trial023.stdout:365: oid 11 version 108 is already newer than 108 2026-02-25T08:43:26.211 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 11 v 108 (ObjNum 189 snap 48 seq_num 189) dirty exists 2026-02-25T08:43:26.211 INFO:tasks.rados.rados.0.trial023.stdout:365: left oid 11 (ObjNum 189 snap 48 seq_num 189) 2026-02-25T08:43:26.211 INFO:tasks.rados.rados.0.trial023.stdout:366: finishing write tid 1 to trial02342524-49 2026-02-25T08:43:26.211 INFO:tasks.rados.rados.0.trial023.stdout:366: finishing write tid 2 to trial02342524-49 2026-02-25T08:43:26.211 INFO:tasks.rados.rados.0.trial023.stdout:366: finishing write tid 3 to trial02342524-49 2026-02-25T08:43:26.211 INFO:tasks.rados.rados.0.trial023.stdout:366: finishing write tid 4 to trial02342524-49 2026-02-25T08:43:26.211 INFO:tasks.rados.rados.0.trial023.stdout:366: finishing write tid 5 to trial02342524-49 2026-02-25T08:43:26.211 INFO:tasks.rados.rados.0.trial023.stdout:366: oid 49 updating version 0 to 51 2026-02-25T08:43:26.211 INFO:tasks.rados.rados.0.trial023.stdout:366: oid 49 version 51 is already newer than 51 2026-02-25T08:43:26.211 INFO:tasks.rados.rados.0.trial023.stdout:366: oid 49 version 51 is already newer than 51 2026-02-25T08:43:26.211 INFO:tasks.rados.rados.0.trial023.stdout:366: oid 49 version 51 is already newer than 48 2026-02-25T08:43:26.212 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 49 v 51 (ObjNum 190 snap 48 seq_num 190) dirty exists 2026-02-25T08:43:26.212 INFO:tasks.rados.rados.0.trial023.stdout:366: left oid 49 (ObjNum 190 snap 48 seq_num 190) 2026-02-25T08:43:26.212 INFO:tasks.rados.rados.0.trial023.stdout:369: expect (ObjNum 175 snap 40 seq_num 175) 2026-02-25T08:43:26.247 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1775986 2026-02-25T08:43:26.288 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2846236 2026-02-25T08:43:26.293 INFO:tasks.rados.rados.0.trial023.stdout:370: finishing write tid 1 to trial02342524-29 2026-02-25T08:43:26.293 INFO:tasks.rados.rados.0.trial023.stdout:370: finishing write tid 2 to trial02342524-29 2026-02-25T08:43:26.293 INFO:tasks.rados.rados.0.trial023.stdout:370: finishing write tid 3 to trial02342524-29 2026-02-25T08:43:26.293 INFO:tasks.rados.rados.0.trial023.stdout:370: finishing write tid 4 to trial02342524-29 2026-02-25T08:43:26.293 INFO:tasks.rados.rados.0.trial023.stdout:370: finishing write tid 5 to trial02342524-29 2026-02-25T08:43:26.293 INFO:tasks.rados.rados.0.trial023.stdout:370: oid 29 updating version 0 to 99 2026-02-25T08:43:26.293 INFO:tasks.rados.rados.0.trial023.stdout:370: oid 29 version 99 is already newer than 99 2026-02-25T08:43:26.294 INFO:tasks.rados.rados.0.trial023.stdout:370: oid 29 version 99 is already newer than 99 2026-02-25T08:43:26.294 INFO:tasks.rados.rados.0.trial023.stdout:370: oid 29 version 99 is already newer than 99 2026-02-25T08:43:26.294 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 29 v 99 (ObjNum 191 snap 48 seq_num 191) dirty exists 2026-02-25T08:43:26.294 INFO:tasks.rados.rados.0.trial023.stdout:370: left oid 29 (ObjNum 191 snap 48 seq_num 191) 2026-02-25T08:43:26.294 INFO:tasks.rados.rados.0.trial023.stdout:372: expect (ObjNum 4 snap 0 seq_num 4) 2026-02-25T08:43:26.314 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 915405 2026-02-25T08:43:26.342 INFO:tasks.rados.rados.0.trial023.stdout:374: expect (ObjNum 161 snap 34 seq_num 161) 2026-02-25T08:43:26.342 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:43:26.444 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1571315 2026-02-25T08:43:26.812 INFO:tasks.rados.rados.0.trial023.stdout:358: done (12 left) 2026-02-25T08:43:26.812 INFO:tasks.rados.rados.0.trial023.stdout:359: done (11 left) 2026-02-25T08:43:26.812 INFO:tasks.rados.rados.0.trial023.stdout:360: done (10 left) 2026-02-25T08:43:26.813 INFO:tasks.rados.rados.0.trial023.stdout:361: done (9 left) 2026-02-25T08:43:26.813 INFO:tasks.rados.rados.0.trial023.stdout:364: done (8 left) 2026-02-25T08:43:26.813 INFO:tasks.rados.rados.0.trial023.stdout:365: done (7 left) 2026-02-25T08:43:26.813 INFO:tasks.rados.rados.0.trial023.stdout:366: done (6 left) 2026-02-25T08:43:26.813 INFO:tasks.rados.rados.0.trial023.stdout:367: done (5 left) 2026-02-25T08:43:26.813 INFO:tasks.rados.rados.0.trial023.stdout:369: done (4 left) 2026-02-25T08:43:26.813 INFO:tasks.rados.rados.0.trial023.stdout:370: done (3 left) 2026-02-25T08:43:26.813 INFO:tasks.rados.rados.0.trial023.stdout:372: done (2 left) 2026-02-25T08:43:26.813 INFO:tasks.rados.rados.0.trial023.stdout:374: done (1 left) 2026-02-25T08:43:26.813 INFO:tasks.rados.rados.0.trial023.stdout:375: done (0 left) 2026-02-25T08:43:26.813 INFO:tasks.rados.rados.0.trial023.stdout:376: snap_remove snap 37 2026-02-25T08:43:27.817 INFO:tasks.rados.rados.0.trial023.stdout:376: done (0 left) 2026-02-25T08:43:27.817 INFO:tasks.rados.rados.0.trial023.stdout:377: snap_create 2026-02-25T08:43:27.923 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:27.997 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:27.994+0000 7ff965ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:27.999 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:27.997+0000 7ff965ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:28.014 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:43:28.041 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:28.042 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:28.221 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:43:28.293 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:43:28.295 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:43:28.296 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:43:28.820 INFO:tasks.rados.rados.0.trial023.stdout:377: done (0 left) 2026-02-25T08:43:28.821 INFO:tasks.rados.rados.0.trial023.stdout:378: snap_remove snap 49 2026-02-25T08:43:29.825 INFO:tasks.rados.rados.0.trial023.stdout:378: done (0 left) 2026-02-25T08:43:29.826 INFO:tasks.rados.rados.0.trial023.stdout:379: delete oid 48 current snap is 50 2026-02-25T08:43:29.860 INFO:tasks.rados.rados.0.trial023.stdout:379: done (0 left) 2026-02-25T08:43:29.860 INFO:tasks.rados.rados.0.trial023.stdout:380: read oid 46 snap -1 2026-02-25T08:43:29.860 INFO:tasks.rados.rados.0.trial023.stdout:380: expect deleted 2026-02-25T08:43:29.861 INFO:tasks.rados.rados.0.trial023.stdout:381: snap_create 2026-02-25T08:43:30.043 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:30.118 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:30.115+0000 7efcefa8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:30.120 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:30.118+0000 7efcefa8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:30.135 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:43:30.161 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:30.162 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:30.494 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~110s 2026-02-25T08:43:30.830 INFO:tasks.rados.rados.0.trial023.stdout:380: done (1 left) 2026-02-25T08:43:30.830 INFO:tasks.rados.rados.0.trial023.stdout:381: done (0 left) 2026-02-25T08:43:30.831 INFO:tasks.rados.rados.0.trial023.stdout:382: write (excl) oid 9 current snap is 51 2026-02-25T08:43:30.831 INFO:tasks.rados.rados.0.trial023.stdout:382: seq_num 193 ranges {443751=631882,1810907=765058,3067461=176290} 2026-02-25T08:43:30.834 INFO:tasks.rados.rados.0.trial023.stdout:382: writing trial02342524-9 from 443751 to 1075633 tid 1 2026-02-25T08:43:30.840 INFO:tasks.rados.rados.0.trial023.stdout:382: writing trial02342524-9 from 1810907 to 2575965 tid 2 2026-02-25T08:43:30.842 INFO:tasks.rados.rados.0.trial023.stdout:382: writing trial02342524-9 from 3067461 to 3243751 tid 3 2026-02-25T08:43:30.843 INFO:tasks.rados.rados.0.trial023.stdout:383: write oid 41 current snap is 51 2026-02-25T08:43:30.843 INFO:tasks.rados.rados.0.trial023.stdout:383: seq_num 194 ranges {460558=707131,1869389=508716,3094884=165674} 2026-02-25T08:43:30.847 INFO:tasks.rados.rados.0.trial023.stdout:383: writing trial02342524-41 from 460558 to 1167689 tid 1 2026-02-25T08:43:30.851 INFO:tasks.rados.rados.0.trial023.stdout:383: writing trial02342524-41 from 1869389 to 2378105 tid 2 2026-02-25T08:43:30.855 INFO:tasks.rados.rados.0.trial023.stdout:383: writing trial02342524-41 from 3094884 to 3260558 tid 3 2026-02-25T08:43:30.855 INFO:tasks.rados.rados.0.trial023.stdout:384: snap_create 2026-02-25T08:43:30.875 INFO:tasks.rados.rados.0.trial023.stdout:382: finishing write tid 1 to trial02342524-9 2026-02-25T08:43:30.881 INFO:tasks.rados.rados.0.trial023.stdout:383: finishing write tid 1 to trial02342524-41 2026-02-25T08:43:30.889 INFO:tasks.rados.rados.0.trial023.stdout:383: finishing write tid 2 to trial02342524-41 2026-02-25T08:43:30.895 INFO:tasks.rados.rados.0.trial023.stdout:383: finishing write tid 3 to trial02342524-41 2026-02-25T08:43:30.897 INFO:tasks.rados.rados.0.trial023.stdout:382: finishing write tid 2 to trial02342524-9 2026-02-25T08:43:30.898 INFO:tasks.rados.rados.0.trial023.stdout:383: finishing write tid 4 to trial02342524-41 2026-02-25T08:43:30.900 INFO:tasks.rados.rados.0.trial023.stdout:383: finishing write tid 5 to trial02342524-41 2026-02-25T08:43:30.900 INFO:tasks.rados.rados.0.trial023.stdout:383: oid 41 updating version 0 to 59 2026-02-25T08:43:30.900 INFO:tasks.rados.rados.0.trial023.stdout:383: oid 41 version 59 is already newer than 59 2026-02-25T08:43:30.901 INFO:tasks.rados.rados.0.trial023.stdout:383: oid 41 version 59 is already newer than 59 2026-02-25T08:43:30.901 INFO:tasks.rados.rados.0.trial023.stdout:383: oid 41 version 59 is already newer than 59 2026-02-25T08:43:30.901 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 41 v 59 (ObjNum 194 snap 51 seq_num 194) dirty exists 2026-02-25T08:43:30.901 INFO:tasks.rados.rados.0.trial023.stdout:383: left oid 41 (ObjNum 194 snap 51 seq_num 194) 2026-02-25T08:43:30.904 INFO:tasks.rados.rados.0.trial023.stdout:382: finishing write tid 3 to trial02342524-9 2026-02-25T08:43:30.908 INFO:tasks.rados.rados.0.trial023.stdout:382: finishing write tid 4 to trial02342524-9 2026-02-25T08:43:30.910 INFO:tasks.rados.rados.0.trial023.stdout:382: finishing write tid 5 to trial02342524-9 2026-02-25T08:43:30.911 INFO:tasks.rados.rados.0.trial023.stdout:382: oid 9 updating version 0 to 104 2026-02-25T08:43:30.911 INFO:tasks.rados.rados.0.trial023.stdout:382: oid 9 version 104 is already newer than 101 2026-02-25T08:43:30.911 INFO:tasks.rados.rados.0.trial023.stdout:382: oid 9 version 104 is already newer than 104 2026-02-25T08:43:30.911 INFO:tasks.rados.rados.0.trial023.stdout:382: oid 9 version 104 is already newer than 104 2026-02-25T08:43:30.911 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 9 v 104 (ObjNum 193 snap 51 seq_num 193) dirty exists 2026-02-25T08:43:30.911 INFO:tasks.rados.rados.0.trial023.stdout:382: left oid 9 (ObjNum 193 snap 51 seq_num 193) 2026-02-25T08:43:31.833 INFO:tasks.rados.rados.0.trial023.stdout:382: done (2 left) 2026-02-25T08:43:31.833 INFO:tasks.rados.rados.0.trial023.stdout:383: done (1 left) 2026-02-25T08:43:31.833 INFO:tasks.rados.rados.0.trial023.stdout:384: done (0 left) 2026-02-25T08:43:31.833 INFO:tasks.rados.rados.0.trial023.stdout:385: snap_remove snap 48 2026-02-25T08:43:32.163 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:32.239 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:32.236+0000 7fc9f6ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:32.242 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:32.240+0000 7fc9f6ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:32.256 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:43:32.282 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:32.282 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:32.836 INFO:tasks.rados.rados.0.trial023.stdout:385: done (0 left) 2026-02-25T08:43:32.836 INFO:tasks.rados.rados.0.trial023.stdout:386: snap_create 2026-02-25T08:43:33.297 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:43:33.368 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:43:33.371 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:43:33.371 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:43:33.841 INFO:tasks.rados.rados.0.trial023.stdout:386: done (0 left) 2026-02-25T08:43:33.842 INFO:tasks.rados.rados.0.trial023.stdout:387: read oid 32 snap -1 2026-02-25T08:43:33.842 INFO:tasks.rados.rados.0.trial023.stdout:387: expect (ObjNum 174 snap 40 seq_num 174) 2026-02-25T08:43:33.842 INFO:tasks.rados.rados.0.trial023.stdout:388: snap_remove snap 50 2026-02-25T08:43:34.283 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:34.359 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:34.355+0000 7fc6a43b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:34.361 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:34.359+0000 7fc6a43b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:34.378 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:43:34.405 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:34.406 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:34.811 INFO:tasks.rados.rados.0.trial023.stdout:388: done (1 left) 2026-02-25T08:43:34.812 INFO:tasks.rados.rados.0.trial023.stdout:389: snap_create 2026-02-25T08:43:34.812 INFO:tasks.rados.rados.0.trial023.stdout:387: expect (ObjNum 174 snap 40 seq_num 174) 2026-02-25T08:43:34.812 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:43:34.845 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 138272 2026-02-25T08:43:34.878 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:43:35.812 INFO:tasks.rados.rados.0.trial023.stdout:387: done (1 left) 2026-02-25T08:43:35.812 INFO:tasks.rados.rados.0.trial023.stdout:389: done (0 left) 2026-02-25T08:43:35.812 INFO:tasks.rados.rados.0.trial023.stdout:390: read oid 5 snap -1 2026-02-25T08:43:35.812 INFO:tasks.rados.rados.0.trial023.stdout:390: expect (ObjNum 4 snap 0 seq_num 4) 2026-02-25T08:43:35.812 INFO:tasks.rados.rados.0.trial023.stdout:391: write oid 48 current snap is 54 2026-02-25T08:43:35.812 INFO:tasks.rados.rados.0.trial023.stdout:391: seq_num 195 ranges {477365=782380,1927871=768727,3122307=155058} 2026-02-25T08:43:35.817 INFO:tasks.rados.rados.0.trial023.stdout:391: writing trial02342524-48 from 477365 to 1259745 tid 1 2026-02-25T08:43:35.822 INFO:tasks.rados.rados.0.trial023.stdout:391: writing trial02342524-48 from 1927871 to 2696598 tid 2 2026-02-25T08:43:35.825 INFO:tasks.rados.rados.0.trial023.stdout:391: writing trial02342524-48 from 3122307 to 3277365 tid 3 2026-02-25T08:43:35.825 INFO:tasks.rados.rados.0.trial023.stdout:392: read oid 17 snap -1 2026-02-25T08:43:35.825 INFO:tasks.rados.rados.0.trial023.stdout:392: expect (ObjNum 178 snap 42 seq_num 178) 2026-02-25T08:43:35.825 INFO:tasks.rados.rados.0.trial023.stdout:393: snap_create 2026-02-25T08:43:35.842 INFO:tasks.rados.rados.0.trial023.stdout:390: expect (ObjNum 4 snap 0 seq_num 4) 2026-02-25T08:43:35.849 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1778509 2026-02-25T08:43:35.851 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:43:35.880 INFO:tasks.rados.rados.0.trial023.stdout:391: finishing write tid 1 to trial02342524-48 2026-02-25T08:43:35.880 INFO:tasks.rados.rados.0.trial023.stdout:392: expect (ObjNum 178 snap 42 seq_num 178) 2026-02-25T08:43:35.920 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:43:35.957 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:43:35.994 INFO:tasks.rados.rados.0.trial023.stdout:391: finishing write tid 2 to trial02342524-48 2026-02-25T08:43:35.995 INFO:tasks.rados.rados.0.trial023.stdout:391: finishing write tid 3 to trial02342524-48 2026-02-25T08:43:35.995 INFO:tasks.rados.rados.0.trial023.stdout:391: finishing write tid 4 to trial02342524-48 2026-02-25T08:43:35.995 INFO:tasks.rados.rados.0.trial023.stdout:391: finishing write tid 5 to trial02342524-48 2026-02-25T08:43:35.995 INFO:tasks.rados.rados.0.trial023.stdout:391: oid 48 updating version 0 to 41 2026-02-25T08:43:35.995 INFO:tasks.rados.rados.0.trial023.stdout:391: oid 48 version 41 is already newer than 41 2026-02-25T08:43:35.995 INFO:tasks.rados.rados.0.trial023.stdout:391: oid 48 version 41 is already newer than 41 2026-02-25T08:43:35.995 INFO:tasks.rados.rados.0.trial023.stdout:391: oid 48 version 41 is already newer than 41 2026-02-25T08:43:35.995 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 48 v 41 (ObjNum 195 snap 54 seq_num 195) dirty exists 2026-02-25T08:43:35.995 INFO:tasks.rados.rados.0.trial023.stdout:391: left oid 48 (ObjNum 195 snap 54 seq_num 195) 2026-02-25T08:43:35.999 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~116s 2026-02-25T08:43:36.407 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:36.484 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:36.481+0000 7f690181a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:36.487 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:36.485+0000 7f690181a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:36.502 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:43:36.527 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:36.528 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:36.814 INFO:tasks.rados.rados.0.trial023.stdout:390: done (3 left) 2026-02-25T08:43:36.814 INFO:tasks.rados.rados.0.trial023.stdout:391: done (2 left) 2026-02-25T08:43:36.814 INFO:tasks.rados.rados.0.trial023.stdout:392: done (1 left) 2026-02-25T08:43:36.814 INFO:tasks.rados.rados.0.trial023.stdout:393: done (0 left) 2026-02-25T08:43:36.814 INFO:tasks.rados.rados.0.trial023.stdout:394: write (excl) oid 36 current snap is 55 2026-02-25T08:43:36.814 INFO:tasks.rados.rados.0.trial023.stdout:394: seq_num 196 ranges {494172=457629,1586353=512385,2749730=544442} 2026-02-25T08:43:36.818 INFO:tasks.rados.rados.0.trial023.stdout:394: writing trial02342524-36 from 494172 to 951801 tid 1 2026-02-25T08:43:36.822 INFO:tasks.rados.rados.0.trial023.stdout:394: writing trial02342524-36 from 1586353 to 2098738 tid 2 2026-02-25T08:43:36.827 INFO:tasks.rados.rados.0.trial023.stdout:394: writing trial02342524-36 from 2749730 to 3294172 tid 3 2026-02-25T08:43:36.828 INFO:tasks.rados.rados.0.trial023.stdout:395: read oid 12 snap -1 2026-02-25T08:43:36.828 INFO:tasks.rados.rados.0.trial023.stdout:395: expect deleted 2026-02-25T08:43:36.828 INFO:tasks.rados.rados.0.trial023.stdout:396: write (excl) oid 28 current snap is 55 2026-02-25T08:43:36.828 INFO:tasks.rados.rados.0.trial023.stdout:396: seq_num 197 ranges {510979=532878,1528482=772396,3060800=250179} 2026-02-25T08:43:36.831 INFO:tasks.rados.rados.0.trial023.stdout:396: writing trial02342524-28 from 510979 to 1043857 tid 1 2026-02-25T08:43:36.836 INFO:tasks.rados.rados.0.trial023.stdout:396: writing trial02342524-28 from 1528482 to 2300878 tid 2 2026-02-25T08:43:36.840 INFO:tasks.rados.rados.0.trial023.stdout:396: writing trial02342524-28 from 3060800 to 3310979 tid 3 2026-02-25T08:43:36.840 INFO:tasks.rados.rados.0.trial023.stdout:397: write (excl) oid 50 current snap is 55 2026-02-25T08:43:36.840 INFO:tasks.rados.rados.0.trial023.stdout:397: seq_num 198 ranges {527786=724480,1703317=516054,2804576=435035} 2026-02-25T08:43:36.844 INFO:tasks.rados.rados.0.trial023.stdout:397: writing trial02342524-50 from 527786 to 1252266 tid 1 2026-02-25T08:43:36.848 INFO:tasks.rados.rados.0.trial023.stdout:397: writing trial02342524-50 from 1703317 to 2219371 tid 2 2026-02-25T08:43:36.852 INFO:tasks.rados.rados.0.trial023.stdout:397: writing trial02342524-50 from 2804576 to 3239611 tid 3 2026-02-25T08:43:36.852 INFO:tasks.rados.rados.0.trial023.stdout:398: snap_create 2026-02-25T08:43:36.865 INFO:tasks.rados.rados.0.trial023.stdout:396: finishing write tid 1 to trial02342524-28 2026-02-25T08:43:36.867 INFO:tasks.rados.rados.0.trial023.stdout:394: finishing write tid 1 to trial02342524-36 2026-02-25T08:43:36.885 INFO:tasks.rados.rados.0.trial023.stdout:397: finishing write tid 1 to trial02342524-50 2026-02-25T08:43:36.892 INFO:tasks.rados.rados.0.trial023.stdout:394: finishing write tid 2 to trial02342524-36 2026-02-25T08:43:36.892 INFO:tasks.rados.rados.0.trial023.stdout:396: finishing write tid 2 to trial02342524-28 2026-02-25T08:43:36.902 INFO:tasks.rados.rados.0.trial023.stdout:394: finishing write tid 3 to trial02342524-36 2026-02-25T08:43:36.905 INFO:tasks.rados.rados.0.trial023.stdout:396: finishing write tid 3 to trial02342524-28 2026-02-25T08:43:36.907 INFO:tasks.rados.rados.0.trial023.stdout:394: finishing write tid 4 to trial02342524-36 2026-02-25T08:43:36.909 INFO:tasks.rados.rados.0.trial023.stdout:394: finishing write tid 5 to trial02342524-36 2026-02-25T08:43:36.909 INFO:tasks.rados.rados.0.trial023.stdout:394: oid 36 updating version 0 to 66 2026-02-25T08:43:36.910 INFO:tasks.rados.rados.0.trial023.stdout:394: oid 36 version 66 is already newer than 66 2026-02-25T08:43:36.910 INFO:tasks.rados.rados.0.trial023.stdout:394: oid 36 version 66 is already newer than 63 2026-02-25T08:43:36.910 INFO:tasks.rados.rados.0.trial023.stdout:394: oid 36 version 66 is already newer than 66 2026-02-25T08:43:36.910 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 36 v 66 (ObjNum 196 snap 55 seq_num 196) dirty exists 2026-02-25T08:43:36.910 INFO:tasks.rados.rados.0.trial023.stdout:394: left oid 36 (ObjNum 196 snap 55 seq_num 196) 2026-02-25T08:43:36.914 INFO:tasks.rados.rados.0.trial023.stdout:396: finishing write tid 4 to trial02342524-28 2026-02-25T08:43:36.915 INFO:tasks.rados.rados.0.trial023.stdout:396: finishing write tid 5 to trial02342524-28 2026-02-25T08:43:36.915 INFO:tasks.rados.rados.0.trial023.stdout:396: oid 28 updating version 0 to 46 2026-02-25T08:43:36.915 INFO:tasks.rados.rados.0.trial023.stdout:396: oid 28 version 46 is already newer than 46 2026-02-25T08:43:36.916 INFO:tasks.rados.rados.0.trial023.stdout:396: oid 28 version 46 is already newer than 46 2026-02-25T08:43:36.916 INFO:tasks.rados.rados.0.trial023.stdout:396: oid 28 version 46 is already newer than 43 2026-02-25T08:43:36.916 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 28 v 46 (ObjNum 197 snap 55 seq_num 197) dirty exists 2026-02-25T08:43:36.916 INFO:tasks.rados.rados.0.trial023.stdout:396: left oid 28 (ObjNum 197 snap 55 seq_num 197) 2026-02-25T08:43:36.921 INFO:tasks.rados.rados.0.trial023.stdout:397: finishing write tid 2 to trial02342524-50 2026-02-25T08:43:36.927 INFO:tasks.rados.rados.0.trial023.stdout:397: finishing write tid 3 to trial02342524-50 2026-02-25T08:43:36.931 INFO:tasks.rados.rados.0.trial023.stdout:397: finishing write tid 4 to trial02342524-50 2026-02-25T08:43:36.933 INFO:tasks.rados.rados.0.trial023.stdout:397: finishing write tid 5 to trial02342524-50 2026-02-25T08:43:36.933 INFO:tasks.rados.rados.0.trial023.stdout:397: oid 50 updating version 0 to 113 2026-02-25T08:43:36.933 INFO:tasks.rados.rados.0.trial023.stdout:397: oid 50 version 113 is already newer than 113 2026-02-25T08:43:36.933 INFO:tasks.rados.rados.0.trial023.stdout:397: oid 50 version 113 is already newer than 113 2026-02-25T08:43:36.933 INFO:tasks.rados.rados.0.trial023.stdout:397: oid 50 version 113 is already newer than 110 2026-02-25T08:43:36.934 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 50 v 113 (ObjNum 198 snap 55 seq_num 198) dirty exists 2026-02-25T08:43:36.934 INFO:tasks.rados.rados.0.trial023.stdout:397: left oid 50 (ObjNum 198 snap 55 seq_num 198) 2026-02-25T08:43:37.818 INFO:tasks.rados.rados.0.trial023.stdout:394: done (4 left) 2026-02-25T08:43:37.818 INFO:tasks.rados.rados.0.trial023.stdout:395: done (3 left) 2026-02-25T08:43:37.818 INFO:tasks.rados.rados.0.trial023.stdout:396: done (2 left) 2026-02-25T08:43:37.818 INFO:tasks.rados.rados.0.trial023.stdout:397: done (1 left) 2026-02-25T08:43:37.818 INFO:tasks.rados.rados.0.trial023.stdout:398: done (0 left) 2026-02-25T08:43:37.818 INFO:tasks.rados.rados.0.trial023.stdout:399: write oid 18 current snap is 56 2026-02-25T08:43:37.818 INFO:tasks.rados.rados.0.trial023.stdout:399: seq_num 199 ranges {544593=799729,1761799=776065,2948352=396241} 2026-02-25T08:43:37.823 INFO:tasks.rados.rados.0.trial023.stdout:399: writing trial02342524-18 from 544593 to 1344322 tid 1 2026-02-25T08:43:37.827 INFO:tasks.rados.rados.0.trial023.stdout:399: writing trial02342524-18 from 1761799 to 2537864 tid 2 2026-02-25T08:43:37.831 INFO:tasks.rados.rados.0.trial023.stdout:399: writing trial02342524-18 from 2948352 to 3344593 tid 3 2026-02-25T08:43:37.831 INFO:tasks.rados.rados.0.trial023.stdout:400: read oid 47 snap -1 2026-02-25T08:43:37.831 INFO:tasks.rados.rados.0.trial023.stdout:400: expect (ObjNum 46 snap 0 seq_num 46) 2026-02-25T08:43:37.831 INFO:tasks.rados.rados.0.trial023.stdout:401: write (excl) oid 14 current snap is 56 2026-02-25T08:43:37.831 INFO:tasks.rados.rados.0.trial023.stdout:401: seq_num 200 ranges {561400=474978,1820281=519723,2859422=457579} 2026-02-25T08:43:37.834 INFO:tasks.rados.rados.0.trial023.stdout:401: writing trial02342524-14 from 561400 to 1036378 tid 1 2026-02-25T08:43:37.839 INFO:tasks.rados.rados.0.trial023.stdout:401: writing trial02342524-14 from 1820281 to 2340004 tid 2 2026-02-25T08:43:37.843 INFO:tasks.rados.rados.0.trial023.stdout:401: writing trial02342524-14 from 2859422 to 3317001 tid 3 2026-02-25T08:43:37.843 INFO:tasks.rados.rados.0.trial023.stdout:402: delete oid 44 current snap is 56 2026-02-25T08:43:37.850 INFO:tasks.rados.rados.0.trial023.stdout:400: expect (ObjNum 46 snap 0 seq_num 46) 2026-02-25T08:43:37.883 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2434945 2026-02-25T08:43:37.893 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2705692 2026-02-25T08:43:37.896 INFO:tasks.rados.rados.0.trial023.stdout:401: finishing write tid 1 to trial02342524-14 2026-02-25T08:43:37.897 INFO:tasks.rados.rados.0.trial023.stdout:399: finishing write tid 1 to trial02342524-18 2026-02-25T08:43:37.897 INFO:tasks.rados.rados.0.trial023.stdout:399: finishing write tid 2 to trial02342524-18 2026-02-25T08:43:37.897 INFO:tasks.rados.rados.0.trial023.stdout:401: finishing write tid 2 to trial02342524-14 2026-02-25T08:43:37.905 INFO:tasks.rados.rados.0.trial023.stdout:401: finishing write tid 3 to trial02342524-14 2026-02-25T08:43:37.905 INFO:tasks.rados.rados.0.trial023.stdout:399: finishing write tid 3 to trial02342524-18 2026-02-25T08:43:37.911 INFO:tasks.rados.rados.0.trial023.stdout:401: finishing write tid 4 to trial02342524-14 2026-02-25T08:43:37.913 INFO:tasks.rados.rados.0.trial023.stdout:401: finishing write tid 5 to trial02342524-14 2026-02-25T08:43:37.913 INFO:tasks.rados.rados.0.trial023.stdout:401: oid 14 updating version 0 to 28 2026-02-25T08:43:37.913 INFO:tasks.rados.rados.0.trial023.stdout:401: oid 14 version 28 is already newer than 28 2026-02-25T08:43:37.913 INFO:tasks.rados.rados.0.trial023.stdout:401: oid 14 version 28 is already newer than 25 2026-02-25T08:43:37.914 INFO:tasks.rados.rados.0.trial023.stdout:401: oid 14 version 28 is already newer than 28 2026-02-25T08:43:37.914 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 14 v 28 (ObjNum 200 snap 56 seq_num 200) dirty exists 2026-02-25T08:43:37.914 INFO:tasks.rados.rados.0.trial023.stdout:401: left oid 14 (ObjNum 200 snap 56 seq_num 200) 2026-02-25T08:43:37.914 INFO:tasks.rados.rados.0.trial023.stdout:399: finishing write tid 4 to trial02342524-18 2026-02-25T08:43:37.917 INFO:tasks.rados.rados.0.trial023.stdout:399: finishing write tid 5 to trial02342524-18 2026-02-25T08:43:37.917 INFO:tasks.rados.rados.0.trial023.stdout:399: oid 18 updating version 0 to 66 2026-02-25T08:43:37.917 INFO:tasks.rados.rados.0.trial023.stdout:399: oid 18 version 66 is already newer than 66 2026-02-25T08:43:37.917 INFO:tasks.rados.rados.0.trial023.stdout:399: oid 18 version 66 is already newer than 66 2026-02-25T08:43:37.917 INFO:tasks.rados.rados.0.trial023.stdout:399: oid 18 version 66 is already newer than 66 2026-02-25T08:43:37.917 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 18 v 66 (ObjNum 199 snap 56 seq_num 199) dirty exists 2026-02-25T08:43:37.917 INFO:tasks.rados.rados.0.trial023.stdout:399: left oid 18 (ObjNum 199 snap 56 seq_num 199) 2026-02-25T08:43:37.932 INFO:tasks.rados.rados.0.trial023.stdout:399: done (3 left) 2026-02-25T08:43:37.933 INFO:tasks.rados.rados.0.trial023.stdout:400: done (2 left) 2026-02-25T08:43:37.933 INFO:tasks.rados.rados.0.trial023.stdout:401: done (1 left) 2026-02-25T08:43:37.933 INFO:tasks.rados.rados.0.trial023.stdout:402: done (0 left) 2026-02-25T08:43:37.933 INFO:tasks.rados.rados.0.trial023.stdout:403: read oid 40 snap 53 2026-02-25T08:43:37.933 INFO:tasks.rados.rados.0.trial023.stdout:403: expect (ObjNum 103 snap 12 seq_num 103) 2026-02-25T08:43:37.933 INFO:tasks.rados.rados.0.trial023.stdout:404: read oid 23 snap 55 2026-02-25T08:43:37.933 INFO:tasks.rados.rados.0.trial023.stdout:404: expect (ObjNum 170 snap 39 seq_num 170) 2026-02-25T08:43:37.933 INFO:tasks.rados.rados.0.trial023.stdout:405: snap_create 2026-02-25T08:43:37.949 INFO:tasks.rados.rados.0.trial023.stdout:403: expect (ObjNum 103 snap 12 seq_num 103) 2026-02-25T08:43:37.949 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:43:37.987 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2581042 2026-02-25T08:43:38.001 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1483850 2026-02-25T08:43:38.023 INFO:tasks.rados.rados.0.trial023.stdout:404: expect (ObjNum 170 snap 39 seq_num 170) 2026-02-25T08:43:38.023 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 260903 2026-02-25T08:43:38.096 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:43:38.373 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:43:38.443 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:43:38.446 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:43:38.446 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:43:38.529 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:38.606 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:38.602+0000 7f18dc5b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:38.608 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:38.606+0000 7f18dc5b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:38.623 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:43:38.650 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:38.651 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:38.822 INFO:tasks.rados.rados.0.trial023.stdout:403: done (2 left) 2026-02-25T08:43:38.822 INFO:tasks.rados.rados.0.trial023.stdout:404: done (1 left) 2026-02-25T08:43:38.823 INFO:tasks.rados.rados.0.trial023.stdout:405: done (0 left) 2026-02-25T08:43:38.823 INFO:tasks.rados.rados.0.trial023.stdout:406: snap_create 2026-02-25T08:43:39.822 INFO:tasks.rados.rados.0.trial023.stdout:406: done (0 left) 2026-02-25T08:43:39.823 INFO:tasks.rados.rados.0.trial023.stdout:407: write (excl) oid 33 current snap is 58 2026-02-25T08:43:39.823 INFO:tasks.rados.rados.0.trial023.stdout:407: seq_num 202 ranges {595014=625476,1820892=523392,2797915=596476} 2026-02-25T08:43:39.827 INFO:tasks.rados.rados.0.trial023.stdout:407: writing trial02342524-33 from 595014 to 1220490 tid 1 2026-02-25T08:43:39.831 INFO:tasks.rados.rados.0.trial023.stdout:407: writing trial02342524-33 from 1820892 to 2344284 tid 2 2026-02-25T08:43:39.835 INFO:tasks.rados.rados.0.trial023.stdout:407: writing trial02342524-33 from 2797915 to 3394391 tid 3 2026-02-25T08:43:39.835 INFO:tasks.rados.rados.0.trial023.stdout:408: write oid 9 current snap is 58 2026-02-25T08:43:39.835 INFO:tasks.rados.rados.0.trial023.stdout:408: seq_num 203 ranges {611821=700725,1879374=783403,3341691=70130} 2026-02-25T08:43:39.840 INFO:tasks.rados.rados.0.trial023.stdout:408: writing trial02342524-9 from 611821 to 1312546 tid 1 2026-02-25T08:43:39.845 INFO:tasks.rados.rados.0.trial023.stdout:408: writing trial02342524-9 from 1879374 to 2662777 tid 2 2026-02-25T08:43:39.848 INFO:tasks.rados.rados.0.trial023.stdout:408: writing trial02342524-9 from 3341691 to 3411821 tid 3 2026-02-25T08:43:39.848 INFO:tasks.rados.rados.0.trial023.stdout:409: write (excl) oid 10 current snap is 58 2026-02-25T08:43:39.849 INFO:tasks.rados.rados.0.trial023.stdout:409: seq_num 204 ranges {628628=775974,1937856=527061,3252761=175867} 2026-02-25T08:43:39.853 INFO:tasks.rados.rados.0.trial023.stdout:409: writing trial02342524-10 from 628628 to 1404602 tid 1 2026-02-25T08:43:39.857 INFO:tasks.rados.rados.0.trial023.stdout:409: writing trial02342524-10 from 1937856 to 2464917 tid 2 2026-02-25T08:43:39.861 INFO:tasks.rados.rados.0.trial023.stdout:409: writing trial02342524-10 from 3252761 to 3428628 tid 3 2026-02-25T08:43:39.861 INFO:tasks.rados.rados.0.trial023.stdout:410: write (excl) oid 18 current snap is 58 2026-02-25T08:43:39.861 INFO:tasks.rados.rados.0.trial023.stdout:410: seq_num 205 ranges {645435=451223,1879985=787072,3280184=165251} 2026-02-25T08:43:39.865 INFO:tasks.rados.rados.0.trial023.stdout:410: writing trial02342524-18 from 645435 to 1096658 tid 1 2026-02-25T08:43:39.871 INFO:tasks.rados.rados.0.trial023.stdout:410: writing trial02342524-18 from 1879985 to 2667057 tid 2 2026-02-25T08:43:39.874 INFO:tasks.rados.rados.0.trial023.stdout:410: writing trial02342524-18 from 3280184 to 3445435 tid 3 2026-02-25T08:43:39.874 INFO:tasks.rados.rados.0.trial023.stdout:411: snap_remove snap 53 2026-02-25T08:43:40.652 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:40.728 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:40.725+0000 7f0f9a45a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:40.731 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:40.728+0000 7f0f9a45a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:40.746 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:43:40.772 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:40.772 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:40.828 INFO:tasks.rados.rados.0.trial023.stdout:411: done (4 left) 2026-02-25T08:43:40.828 INFO:tasks.rados.rados.0.trial023.stdout:412: read oid 30 snap 56 2026-02-25T08:43:40.828 INFO:tasks.rados.rados.0.trial023.stdout:412: expect deleted 2026-02-25T08:43:40.828 INFO:tasks.rados.rados.0.trial023.stdout:413: snap_create 2026-02-25T08:43:40.829 INFO:tasks.rados.rados.0.trial023.stdout:407: finishing write tid 1 to trial02342524-33 2026-02-25T08:43:40.829 INFO:tasks.rados.rados.0.trial023.stdout:408: finishing write tid 1 to trial02342524-9 2026-02-25T08:43:40.829 INFO:tasks.rados.rados.0.trial023.stdout:408: finishing write tid 2 to trial02342524-9 2026-02-25T08:43:40.829 INFO:tasks.rados.rados.0.trial023.stdout:409: finishing write tid 1 to trial02342524-10 2026-02-25T08:43:40.829 INFO:tasks.rados.rados.0.trial023.stdout:408: finishing write tid 3 to trial02342524-9 2026-02-25T08:43:40.829 INFO:tasks.rados.rados.0.trial023.stdout:407: finishing write tid 2 to trial02342524-33 2026-02-25T08:43:40.829 INFO:tasks.rados.rados.0.trial023.stdout:408: finishing write tid 4 to trial02342524-9 2026-02-25T08:43:40.829 INFO:tasks.rados.rados.0.trial023.stdout:408: finishing write tid 5 to trial02342524-9 2026-02-25T08:43:40.829 INFO:tasks.rados.rados.0.trial023.stdout:408: oid 9 updating version 0 to 113 2026-02-25T08:43:40.829 INFO:tasks.rados.rados.0.trial023.stdout:408: oid 9 version 113 is already newer than 113 2026-02-25T08:43:40.829 INFO:tasks.rados.rados.0.trial023.stdout:408: oid 9 version 113 is already newer than 113 2026-02-25T08:43:40.829 INFO:tasks.rados.rados.0.trial023.stdout:408: oid 9 version 113 is already newer than 113 2026-02-25T08:43:40.830 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 9 v 113 (ObjNum 203 snap 58 seq_num 203) dirty exists 2026-02-25T08:43:40.830 INFO:tasks.rados.rados.0.trial023.stdout:408: left oid 9 (ObjNum 203 snap 58 seq_num 203) 2026-02-25T08:43:40.830 INFO:tasks.rados.rados.0.trial023.stdout:407: finishing write tid 3 to trial02342524-33 2026-02-25T08:43:40.830 INFO:tasks.rados.rados.0.trial023.stdout:410: finishing write tid 1 to trial02342524-18 2026-02-25T08:43:40.830 INFO:tasks.rados.rados.0.trial023.stdout:409: finishing write tid 2 to trial02342524-10 2026-02-25T08:43:40.830 INFO:tasks.rados.rados.0.trial023.stdout:409: finishing write tid 3 to trial02342524-10 2026-02-25T08:43:40.830 INFO:tasks.rados.rados.0.trial023.stdout:410: finishing write tid 2 to trial02342524-18 2026-02-25T08:43:40.830 INFO:tasks.rados.rados.0.trial023.stdout:410: finishing write tid 3 to trial02342524-18 2026-02-25T08:43:40.830 INFO:tasks.rados.rados.0.trial023.stdout:409: finishing write tid 4 to trial02342524-10 2026-02-25T08:43:40.830 INFO:tasks.rados.rados.0.trial023.stdout:410: finishing write tid 4 to trial02342524-18 2026-02-25T08:43:40.830 INFO:tasks.rados.rados.0.trial023.stdout:409: finishing write tid 5 to trial02342524-10 2026-02-25T08:43:40.830 INFO:tasks.rados.rados.0.trial023.stdout:409: oid 10 updating version 0 to 103 2026-02-25T08:43:40.830 INFO:tasks.rados.rados.0.trial023.stdout:409: oid 10 version 103 is already newer than 100 2026-02-25T08:43:40.831 INFO:tasks.rados.rados.0.trial023.stdout:409: oid 10 version 103 is already newer than 103 2026-02-25T08:43:40.831 INFO:tasks.rados.rados.0.trial023.stdout:409: oid 10 version 103 is already newer than 103 2026-02-25T08:43:40.831 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 10 v 103 (ObjNum 204 snap 58 seq_num 204) dirty exists 2026-02-25T08:43:40.831 INFO:tasks.rados.rados.0.trial023.stdout:409: left oid 10 (ObjNum 204 snap 58 seq_num 204) 2026-02-25T08:43:40.831 INFO:tasks.rados.rados.0.trial023.stdout:410: finishing write tid 5 to trial02342524-18 2026-02-25T08:43:40.831 INFO:tasks.rados.rados.0.trial023.stdout:410: oid 18 updating version 0 to 71 2026-02-25T08:43:40.831 INFO:tasks.rados.rados.0.trial023.stdout:410: oid 18 version 71 is already newer than 71 2026-02-25T08:43:40.831 INFO:tasks.rados.rados.0.trial023.stdout:410: oid 18 version 71 is already newer than 68 2026-02-25T08:43:40.831 INFO:tasks.rados.rados.0.trial023.stdout:410: oid 18 version 71 is already newer than 71 2026-02-25T08:43:40.831 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 18 v 71 (ObjNum 205 snap 58 seq_num 205) dirty exists 2026-02-25T08:43:40.831 INFO:tasks.rados.rados.0.trial023.stdout:410: left oid 18 (ObjNum 205 snap 58 seq_num 205) 2026-02-25T08:43:40.831 INFO:tasks.rados.rados.0.trial023.stdout:407: finishing write tid 4 to trial02342524-33 2026-02-25T08:43:40.831 INFO:tasks.rados.rados.0.trial023.stdout:407: finishing write tid 5 to trial02342524-33 2026-02-25T08:43:40.832 INFO:tasks.rados.rados.0.trial023.stdout:407: oid 33 updating version 0 to 22 2026-02-25T08:43:40.832 INFO:tasks.rados.rados.0.trial023.stdout:407: oid 33 version 22 is already newer than 22 2026-02-25T08:43:40.832 INFO:tasks.rados.rados.0.trial023.stdout:407: oid 33 version 22 is already newer than 22 2026-02-25T08:43:40.832 INFO:tasks.rados.rados.0.trial023.stdout:407: oid 33 version 22 is already newer than 19 2026-02-25T08:43:40.832 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 33 v 22 (ObjNum 202 snap 58 seq_num 202) dirty exists 2026-02-25T08:43:40.832 INFO:tasks.rados.rados.0.trial023.stdout:407: left oid 33 (ObjNum 202 snap 58 seq_num 202) 2026-02-25T08:43:41.503 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~121s 2026-02-25T08:43:41.829 INFO:tasks.rados.rados.0.trial023.stdout:407: done (5 left) 2026-02-25T08:43:41.829 INFO:tasks.rados.rados.0.trial023.stdout:408: done (4 left) 2026-02-25T08:43:41.829 INFO:tasks.rados.rados.0.trial023.stdout:409: done (3 left) 2026-02-25T08:43:41.829 INFO:tasks.rados.rados.0.trial023.stdout:410: done (2 left) 2026-02-25T08:43:41.829 INFO:tasks.rados.rados.0.trial023.stdout:412: done (1 left) 2026-02-25T08:43:41.829 INFO:tasks.rados.rados.0.trial023.stdout:413: done (0 left) 2026-02-25T08:43:41.829 INFO:tasks.rados.rados.0.trial023.stdout:414: write (excl) oid 1 current snap is 59 2026-02-25T08:43:41.829 INFO:tasks.rados.rados.0.trial023.stdout:414: seq_num 206 ranges {662242=642825,2054820=530730,3307607=154635} 2026-02-25T08:43:41.833 INFO:tasks.rados.rados.0.trial023.stdout:414: writing trial02342524-1 from 662242 to 1305067 tid 1 2026-02-25T08:43:41.838 INFO:tasks.rados.rados.0.trial023.stdout:414: writing trial02342524-1 from 2054820 to 2585550 tid 2 2026-02-25T08:43:41.842 INFO:tasks.rados.rados.0.trial023.stdout:414: writing trial02342524-1 from 3307607 to 3462242 tid 3 2026-02-25T08:43:41.842 INFO:tasks.rados.rados.0.trial023.stdout:415: read oid 10 snap 58 2026-02-25T08:43:41.842 INFO:tasks.rados.rados.0.trial023.stdout:415: expect (ObjNum 204 snap 58 seq_num 204) 2026-02-25T08:43:41.842 INFO:tasks.rados.rados.0.trial023.stdout:416: snap_create 2026-02-25T08:43:41.862 INFO:tasks.rados.rados.0.trial023.stdout:415: expect (ObjNum 204 snap 58 seq_num 204) 2026-02-25T08:43:41.934 INFO:tasks.rados.rados.0.trial023.stdout:414: finishing write tid 1 to trial02342524-1 2026-02-25T08:43:41.934 INFO:tasks.rados.rados.0.trial023.stdout:414: finishing write tid 2 to trial02342524-1 2026-02-25T08:43:41.934 INFO:tasks.rados.rados.0.trial023.stdout:414: finishing write tid 3 to trial02342524-1 2026-02-25T08:43:41.935 INFO:tasks.rados.rados.0.trial023.stdout:414: finishing write tid 4 to trial02342524-1 2026-02-25T08:43:41.935 INFO:tasks.rados.rados.0.trial023.stdout:414: finishing write tid 5 to trial02342524-1 2026-02-25T08:43:41.935 INFO:tasks.rados.rados.0.trial023.stdout:414: oid 1 updating version 0 to 64 2026-02-25T08:43:41.935 INFO:tasks.rados.rados.0.trial023.stdout:414: oid 1 version 64 is already newer than 64 2026-02-25T08:43:41.935 INFO:tasks.rados.rados.0.trial023.stdout:414: oid 1 version 64 is already newer than 61 2026-02-25T08:43:41.935 INFO:tasks.rados.rados.0.trial023.stdout:414: oid 1 version 64 is already newer than 64 2026-02-25T08:43:41.935 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 1 v 64 (ObjNum 206 snap 59 seq_num 206) dirty exists 2026-02-25T08:43:41.935 INFO:tasks.rados.rados.0.trial023.stdout:414: left oid 1 (ObjNum 206 snap 59 seq_num 206) 2026-02-25T08:43:42.774 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:42.833 INFO:tasks.rados.rados.0.trial023.stdout:414: done (2 left) 2026-02-25T08:43:42.833 INFO:tasks.rados.rados.0.trial023.stdout:415: done (1 left) 2026-02-25T08:43:42.833 INFO:tasks.rados.rados.0.trial023.stdout:416: done (0 left) 2026-02-25T08:43:42.833 INFO:tasks.rados.rados.0.trial023.stdout:417: read oid 6 snap 51 2026-02-25T08:43:42.833 INFO:tasks.rados.rados.0.trial023.stdout:417: expect deleted 2026-02-25T08:43:42.833 INFO:tasks.rados.rados.0.trial023.stdout:418: write (excl) oid 4 current snap is 60 2026-02-25T08:43:42.833 INFO:tasks.rados.rados.0.trial023.stdout:418: seq_num 207 ranges {679049=718074,2113302=674388,3335030=144019} 2026-02-25T08:43:42.838 INFO:tasks.rados.rados.0.trial023.stdout:418: writing trial02342524-4 from 679049 to 1397123 tid 1 2026-02-25T08:43:42.844 INFO:tasks.rados.rados.0.trial023.stdout:418: writing trial02342524-4 from 2113302 to 2787690 tid 2 2026-02-25T08:43:42.846 INFO:tasks.rados.rados.0.trial023.stdout:418: writing trial02342524-4 from 3335030 to 3479049 tid 3 2026-02-25T08:43:42.847 INFO:tasks.rados.rados.0.trial023.stdout:419: write (excl) oid 44 current snap is 60 2026-02-25T08:43:42.847 INFO:tasks.rados.rados.0.trial023.stdout:419: seq_num 208 ranges {695856=793323,2171784=534399,3362453=133403} 2026-02-25T08:43:42.852 INFO:tasks.rados.rados.0.trial023.stdout:419: writing trial02342524-44 from 695856 to 1489179 tid 1 2026-02-25T08:43:42.857 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:42.854+0000 7f5251eb7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:42.857 INFO:tasks.rados.rados.0.trial023.stdout:419: writing trial02342524-44 from 2171784 to 2706183 tid 2 2026-02-25T08:43:42.859 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:42.857+0000 7f5251eb7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:42.860 INFO:tasks.rados.rados.0.trial023.stdout:419: writing trial02342524-44 from 3362453 to 3495856 tid 3 2026-02-25T08:43:42.860 INFO:tasks.rados.rados.0.trial023.stdout:420: delete oid 26 current snap is 60 2026-02-25T08:43:42.872 INFO:tasks.rados.rados.0.trial023.stdout:419: finishing write tid 1 to trial02342524-44 2026-02-25T08:43:42.884 INFO:tasks.rados.rados.0.trial023.stdout:418: finishing write tid 1 to trial02342524-4 2026-02-25T08:43:42.885 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:43:42.887 INFO:tasks.rados.rados.0.trial023.stdout:419: finishing write tid 2 to trial02342524-44 2026-02-25T08:43:42.897 INFO:tasks.rados.rados.0.trial023.stdout:419: finishing write tid 3 to trial02342524-44 2026-02-25T08:43:42.902 INFO:tasks.rados.rados.0.trial023.stdout:417: done (3 left) 2026-02-25T08:43:42.902 INFO:tasks.rados.rados.0.trial023.stdout:420: done (2 left) 2026-02-25T08:43:42.902 INFO:tasks.rados.rados.0.trial023.stdout:421: read oid 13 snap -1 2026-02-25T08:43:42.902 INFO:tasks.rados.rados.0.trial023.stdout:421: expect deleted 2026-02-25T08:43:42.902 INFO:tasks.rados.rados.0.trial023.stdout:422: delete oid 7 current snap is 60 2026-02-25T08:43:42.905 INFO:tasks.rados.rados.0.trial023.stdout:419: finishing write tid 4 to trial02342524-44 2026-02-25T08:43:42.907 INFO:tasks.rados.rados.0.trial023.stdout:419: finishing write tid 5 to trial02342524-44 2026-02-25T08:43:42.908 INFO:tasks.rados.rados.0.trial023.stdout:419: oid 44 updating version 0 to 16 2026-02-25T08:43:42.908 INFO:tasks.rados.rados.0.trial023.stdout:419: oid 44 version 16 is already newer than 16 2026-02-25T08:43:42.908 INFO:tasks.rados.rados.0.trial023.stdout:419: oid 44 version 16 is already newer than 13 2026-02-25T08:43:42.908 INFO:tasks.rados.rados.0.trial023.stdout:419: oid 44 version 16 is already newer than 16 2026-02-25T08:43:42.908 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 44 v 16 (ObjNum 208 snap 60 seq_num 208) dirty exists 2026-02-25T08:43:42.908 INFO:tasks.rados.rados.0.trial023.stdout:419: left oid 44 (ObjNum 208 snap 60 seq_num 208) 2026-02-25T08:43:42.908 INFO:tasks.rados.rados.0.trial023.stdout:418: finishing write tid 2 to trial02342524-4 2026-02-25T08:43:42.918 INFO:tasks.rados.rados.0.trial023.stdout:418: finishing write tid 3 to trial02342524-4 2026-02-25T08:43:42.920 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:42.920 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:42.921 INFO:tasks.rados.rados.0.trial023.stdout:418: finishing write tid 4 to trial02342524-4 2026-02-25T08:43:42.922 INFO:tasks.rados.rados.0.trial023.stdout:418: finishing write tid 5 to trial02342524-4 2026-02-25T08:43:42.922 INFO:tasks.rados.rados.0.trial023.stdout:418: oid 4 updating version 0 to 115 2026-02-25T08:43:42.923 INFO:tasks.rados.rados.0.trial023.stdout:418: oid 4 updating version 115 to 118 2026-02-25T08:43:42.923 INFO:tasks.rados.rados.0.trial023.stdout:418: oid 4 version 118 is already newer than 118 2026-02-25T08:43:42.923 INFO:tasks.rados.rados.0.trial023.stdout:418: oid 4 version 118 is already newer than 118 2026-02-25T08:43:42.923 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 4 v 118 (ObjNum 207 snap 60 seq_num 207) dirty exists 2026-02-25T08:43:42.923 INFO:tasks.rados.rados.0.trial023.stdout:418: left oid 4 (ObjNum 207 snap 60 seq_num 207) 2026-02-25T08:43:42.934 INFO:tasks.rados.rados.0.trial023.stdout:418: done (3 left) 2026-02-25T08:43:42.934 INFO:tasks.rados.rados.0.trial023.stdout:419: done (2 left) 2026-02-25T08:43:42.934 INFO:tasks.rados.rados.0.trial023.stdout:421: done (1 left) 2026-02-25T08:43:42.935 INFO:tasks.rados.rados.0.trial023.stdout:422: done (0 left) 2026-02-25T08:43:42.935 INFO:tasks.rados.rados.0.trial023.stdout:423: read oid 19 snap 33 2026-02-25T08:43:42.935 INFO:tasks.rados.rados.0.trial023.stdout:423: expect (ObjNum 151 snap 26 seq_num 151) 2026-02-25T08:43:42.935 INFO:tasks.rados.rados.0.trial023.stdout:424: read oid 13 snap -1 2026-02-25T08:43:42.935 INFO:tasks.rados.rados.0.trial023.stdout:424: expect deleted 2026-02-25T08:43:42.935 INFO:tasks.rados.rados.0.trial023.stdout:425: read oid 42 snap -1 2026-02-25T08:43:42.935 INFO:tasks.rados.rados.0.trial023.stdout:425: expect deleted 2026-02-25T08:43:42.935 INFO:tasks.rados.rados.0.trial023.stdout:426: snap_create 2026-02-25T08:43:42.948 INFO:tasks.rados.rados.0.trial023.stdout:423: expect (ObjNum 151 snap 26 seq_num 151) 2026-02-25T08:43:42.948 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 40384 2026-02-25T08:43:42.992 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1275933 2026-02-25T08:43:43.447 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:43:43.519 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:43:43.522 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:43:43.523 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:43:43.834 INFO:tasks.rados.rados.0.trial023.stdout:423: done (3 left) 2026-02-25T08:43:43.834 INFO:tasks.rados.rados.0.trial023.stdout:424: done (2 left) 2026-02-25T08:43:43.835 INFO:tasks.rados.rados.0.trial023.stdout:425: done (1 left) 2026-02-25T08:43:43.835 INFO:tasks.rados.rados.0.trial023.stdout:426: done (0 left) 2026-02-25T08:43:43.835 INFO:tasks.rados.rados.0.trial023.stdout:427: read oid 39 snap 56 2026-02-25T08:43:43.835 INFO:tasks.rados.rados.0.trial023.stdout:427: expect (ObjNum 175 snap 40 seq_num 175) 2026-02-25T08:43:43.835 INFO:tasks.rados.rados.0.trial023.stdout:428: delete oid 31 current snap is 61 2026-02-25T08:43:43.869 INFO:tasks.rados.rados.0.trial023.stdout:427: expect (ObjNum 175 snap 40 seq_num 175) 2026-02-25T08:43:43.869 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 268463 2026-02-25T08:43:43.902 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:43:43.962 INFO:tasks.rados.rados.0.trial023.stdout:427: done (1 left) 2026-02-25T08:43:43.962 INFO:tasks.rados.rados.0.trial023.stdout:428: done (0 left) 2026-02-25T08:43:43.962 INFO:tasks.rados.rados.0.trial023.stdout:429: snap_create 2026-02-25T08:43:44.813 INFO:tasks.rados.rados.0.trial023.stdout:429: done (0 left) 2026-02-25T08:43:44.813 INFO:tasks.rados.rados.0.trial023.stdout:430: snap_remove snap 57 2026-02-25T08:43:44.921 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:44.998 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:44.994+0000 7fa447bec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:45.000 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:44.998+0000 7fa447bec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:45.015 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:43:45.042 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:45.043 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:45.814 INFO:tasks.rados.rados.0.trial023.stdout:430: done (0 left) 2026-02-25T08:43:45.815 INFO:tasks.rados.rados.0.trial023.stdout:431: read oid 22 snap -1 2026-02-25T08:43:45.815 INFO:tasks.rados.rados.0.trial023.stdout:431: expect (ObjNum 160 snap 34 seq_num 160) 2026-02-25T08:43:45.815 INFO:tasks.rados.rados.0.trial023.stdout:432: write oid 9 current snap is 62 2026-02-25T08:43:45.815 INFO:tasks.rados.rados.0.trial023.stdout:432: seq_num 212 ranges {763084=694319,1889359=541737,2955792=541902} 2026-02-25T08:43:45.819 INFO:tasks.rados.rados.0.trial023.stdout:432: writing trial02342524-9 from 763084 to 1457403 tid 1 2026-02-25T08:43:45.823 INFO:tasks.rados.rados.0.trial023.stdout:432: writing trial02342524-9 from 1889359 to 2431096 tid 2 2026-02-25T08:43:45.827 INFO:tasks.rados.rados.0.trial023.stdout:432: writing trial02342524-9 from 2955792 to 3497694 tid 3 2026-02-25T08:43:45.827 INFO:tasks.rados.rados.0.trial023.stdout:433: delete oid 8 current snap is 62 2026-02-25T08:43:45.863 INFO:tasks.rados.rados.0.trial023.stdout:431: expect (ObjNum 160 snap 34 seq_num 160) 2026-02-25T08:43:45.945 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 33903 2026-02-25T08:43:45.989 INFO:tasks.rados.rados.0.trial023.stdout:431: done (2 left) 2026-02-25T08:43:45.989 INFO:tasks.rados.rados.0.trial023.stdout:433: done (1 left) 2026-02-25T08:43:45.989 INFO:tasks.rados.rados.0.trial023.stdout:434: snap_remove snap 52 2026-02-25T08:43:46.843 INFO:tasks.rados.rados.0.trial023.stdout:434: done (1 left) 2026-02-25T08:43:46.843 INFO:tasks.rados.rados.0.trial023.stdout:435: write oid 14 current snap is 62 2026-02-25T08:43:46.844 INFO:tasks.rados.rados.0.trial023.stdout:432: finishing write tid 1 to trial02342524-9 2026-02-25T08:43:46.844 INFO:tasks.rados.rados.0.trial023.stdout:432: finishing write tid 2 to trial02342524-9 2026-02-25T08:43:46.844 INFO:tasks.rados.rados.0.trial023.stdout:432: finishing write tid 3 to trial02342524-9 2026-02-25T08:43:46.844 INFO:tasks.rados.rados.0.trial023.stdout:435: seq_num 214 ranges {796698=561170,2006323=545406,3126991=469707} 2026-02-25T08:43:46.848 INFO:tasks.rados.rados.0.trial023.stdout:435: writing trial02342524-14 from 796698 to 1357868 tid 1 2026-02-25T08:43:46.853 INFO:tasks.rados.rados.0.trial023.stdout:435: writing trial02342524-14 from 2006323 to 2551729 tid 2 2026-02-25T08:43:46.857 INFO:tasks.rados.rados.0.trial023.stdout:435: writing trial02342524-14 from 3126991 to 3596698 tid 3 2026-02-25T08:43:46.857 INFO:tasks.rados.rados.0.trial023.stdout:436: write oid 43 current snap is 62 2026-02-25T08:43:46.857 INFO:tasks.rados.rados.0.trial023.stdout:436: seq_num 215 ranges {413505=636419,1664805=689064,3038061=492071} 2026-02-25T08:43:46.861 INFO:tasks.rados.rados.0.trial023.stdout:436: writing trial02342524-43 from 413505 to 1049924 tid 1 2026-02-25T08:43:46.866 INFO:tasks.rados.rados.0.trial023.stdout:436: writing trial02342524-43 from 1664805 to 2353869 tid 2 2026-02-25T08:43:46.871 INFO:tasks.rados.rados.0.trial023.stdout:436: writing trial02342524-43 from 3038061 to 3530132 tid 3 2026-02-25T08:43:46.871 INFO:tasks.rados.rados.0.trial023.stdout:437: snap_create 2026-02-25T08:43:46.871 INFO:tasks.rados.rados.0.trial023.stdout:432: finishing write tid 4 to trial02342524-9 2026-02-25T08:43:46.872 INFO:tasks.rados.rados.0.trial023.stdout:432: finishing write tid 5 to trial02342524-9 2026-02-25T08:43:46.872 INFO:tasks.rados.rados.0.trial023.stdout:432: oid 9 updating version 0 to 122 2026-02-25T08:43:46.872 INFO:tasks.rados.rados.0.trial023.stdout:432: oid 9 version 122 is already newer than 122 2026-02-25T08:43:46.872 INFO:tasks.rados.rados.0.trial023.stdout:432: oid 9 version 122 is already newer than 122 2026-02-25T08:43:46.872 INFO:tasks.rados.rados.0.trial023.stdout:432: oid 9 version 122 is already newer than 122 2026-02-25T08:43:46.872 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 9 v 122 (ObjNum 212 snap 62 seq_num 212) dirty exists 2026-02-25T08:43:46.872 INFO:tasks.rados.rados.0.trial023.stdout:432: left oid 9 (ObjNum 212 snap 62 seq_num 212) 2026-02-25T08:43:46.898 INFO:tasks.rados.rados.0.trial023.stdout:435: finishing write tid 1 to trial02342524-14 2026-02-25T08:43:46.902 INFO:tasks.rados.rados.0.trial023.stdout:435: finishing write tid 2 to trial02342524-14 2026-02-25T08:43:46.906 INFO:tasks.rados.rados.0.trial023.stdout:436: finishing write tid 1 to trial02342524-43 2026-02-25T08:43:46.909 INFO:tasks.rados.rados.0.trial023.stdout:435: finishing write tid 3 to trial02342524-14 2026-02-25T08:43:46.913 INFO:tasks.rados.rados.0.trial023.stdout:435: finishing write tid 4 to trial02342524-14 2026-02-25T08:43:46.913 INFO:tasks.rados.rados.0.trial023.stdout:436: finishing write tid 2 to trial02342524-43 2026-02-25T08:43:46.915 INFO:tasks.rados.rados.0.trial023.stdout:435: finishing write tid 5 to trial02342524-14 2026-02-25T08:43:46.915 INFO:tasks.rados.rados.0.trial023.stdout:435: oid 14 updating version 0 to 33 2026-02-25T08:43:46.915 INFO:tasks.rados.rados.0.trial023.stdout:435: oid 14 version 33 is already newer than 33 2026-02-25T08:43:46.915 INFO:tasks.rados.rados.0.trial023.stdout:435: oid 14 version 33 is already newer than 33 2026-02-25T08:43:46.915 INFO:tasks.rados.rados.0.trial023.stdout:435: oid 14 version 33 is already newer than 33 2026-02-25T08:43:46.915 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 14 v 33 (ObjNum 214 snap 62 seq_num 214) dirty exists 2026-02-25T08:43:46.915 INFO:tasks.rados.rados.0.trial023.stdout:435: left oid 14 (ObjNum 214 snap 62 seq_num 214) 2026-02-25T08:43:46.919 INFO:tasks.rados.rados.0.trial023.stdout:436: finishing write tid 3 to trial02342524-43 2026-02-25T08:43:46.922 INFO:tasks.rados.rados.0.trial023.stdout:436: finishing write tid 4 to trial02342524-43 2026-02-25T08:43:46.924 INFO:tasks.rados.rados.0.trial023.stdout:436: finishing write tid 5 to trial02342524-43 2026-02-25T08:43:46.924 INFO:tasks.rados.rados.0.trial023.stdout:436: oid 43 updating version 0 to 17 2026-02-25T08:43:46.924 INFO:tasks.rados.rados.0.trial023.stdout:436: oid 43 version 17 is already newer than 17 2026-02-25T08:43:46.924 INFO:tasks.rados.rados.0.trial023.stdout:436: oid 43 version 17 is already newer than 17 2026-02-25T08:43:46.925 INFO:tasks.rados.rados.0.trial023.stdout:436: oid 43 version 17 is already newer than 17 2026-02-25T08:43:46.925 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 43 v 17 (ObjNum 215 snap 62 seq_num 215) dirty exists 2026-02-25T08:43:46.925 INFO:tasks.rados.rados.0.trial023.stdout:436: left oid 43 (ObjNum 215 snap 62 seq_num 215) 2026-02-25T08:43:47.006 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~127s 2026-02-25T08:43:47.044 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:47.120 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:47.117+0000 7f7977eb7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:47.122 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:47.121+0000 7f7977eb7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:47.138 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:43:47.165 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:47.166 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:47.847 INFO:tasks.rados.rados.0.trial023.stdout:432: done (3 left) 2026-02-25T08:43:47.847 INFO:tasks.rados.rados.0.trial023.stdout:435: done (2 left) 2026-02-25T08:43:47.847 INFO:tasks.rados.rados.0.trial023.stdout:436: done (1 left) 2026-02-25T08:43:47.847 INFO:tasks.rados.rados.0.trial023.stdout:437: done (0 left) 2026-02-25T08:43:47.847 INFO:tasks.rados.rados.0.trial023.stdout:438: write oid 22 current snap is 63 2026-02-25T08:43:47.847 INFO:tasks.rados.rados.0.trial023.stdout:438: seq_num 216 ranges {430312=711668,1723287=549075,2781837=703343} 2026-02-25T08:43:47.851 INFO:tasks.rados.rados.0.trial023.stdout:438: writing trial02342524-22 from 430312 to 1141980 tid 1 2026-02-25T08:43:47.855 INFO:tasks.rados.rados.0.trial023.stdout:438: writing trial02342524-22 from 1723287 to 2272362 tid 2 2026-02-25T08:43:47.860 INFO:tasks.rados.rados.0.trial023.stdout:438: writing trial02342524-22 from 2781837 to 3485180 tid 3 2026-02-25T08:43:47.860 INFO:tasks.rados.rados.0.trial023.stdout:439: snap_remove snap 60 2026-02-25T08:43:48.524 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:43:48.595 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:43:48.597 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:43:48.598 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:43:48.851 INFO:tasks.rados.rados.0.trial023.stdout:439: done (1 left) 2026-02-25T08:43:48.851 INFO:tasks.rados.rados.0.trial023.stdout:440: snap_create 2026-02-25T08:43:48.851 INFO:tasks.rados.rados.0.trial023.stdout:438: finishing write tid 1 to trial02342524-22 2026-02-25T08:43:48.851 INFO:tasks.rados.rados.0.trial023.stdout:438: finishing write tid 2 to trial02342524-22 2026-02-25T08:43:48.851 INFO:tasks.rados.rados.0.trial023.stdout:438: finishing write tid 3 to trial02342524-22 2026-02-25T08:43:48.851 INFO:tasks.rados.rados.0.trial023.stdout:438: finishing write tid 4 to trial02342524-22 2026-02-25T08:43:48.851 INFO:tasks.rados.rados.0.trial023.stdout:438: finishing write tid 5 to trial02342524-22 2026-02-25T08:43:48.851 INFO:tasks.rados.rados.0.trial023.stdout:438: oid 22 updating version 0 to 36 2026-02-25T08:43:48.852 INFO:tasks.rados.rados.0.trial023.stdout:438: oid 22 version 36 is already newer than 36 2026-02-25T08:43:48.852 INFO:tasks.rados.rados.0.trial023.stdout:438: oid 22 version 36 is already newer than 36 2026-02-25T08:43:48.852 INFO:tasks.rados.rados.0.trial023.stdout:438: oid 22 version 36 is already newer than 36 2026-02-25T08:43:48.852 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 22 v 36 (ObjNum 216 snap 63 seq_num 216) dirty exists 2026-02-25T08:43:48.852 INFO:tasks.rados.rados.0.trial023.stdout:438: left oid 22 (ObjNum 216 snap 63 seq_num 216) 2026-02-25T08:43:49.167 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:49.245 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:49.242+0000 7f0a4a8cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:49.247 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:49.246+0000 7f0a4a8cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:49.262 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:43:49.288 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:49.289 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:49.852 INFO:tasks.rados.rados.0.trial023.stdout:438: done (1 left) 2026-02-25T08:43:49.853 INFO:tasks.rados.rados.0.trial023.stdout:440: done (0 left) 2026-02-25T08:43:49.853 INFO:tasks.rados.rados.0.trial023.stdout:441: snap_create 2026-02-25T08:43:50.858 INFO:tasks.rados.rados.0.trial023.stdout:441: done (0 left) 2026-02-25T08:43:50.858 INFO:tasks.rados.rados.0.trial023.stdout:442: snap_remove snap 24 2026-02-25T08:43:51.290 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:51.366 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:51.364+0000 7f384a811640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:51.369 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:51.367+0000 7f384a811640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:51.384 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:43:51.412 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:51.412 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:51.860 INFO:tasks.rados.rados.0.trial023.stdout:442: done (0 left) 2026-02-25T08:43:51.860 INFO:tasks.rados.rados.0.trial023.stdout:443: write oid 46 current snap is 65 2026-02-25T08:43:51.860 INFO:tasks.rados.rados.0.trial023.stdout:443: seq_num 217 ranges {447119=786917,1665416=692733,2976554=514615} 2026-02-25T08:43:51.865 INFO:tasks.rados.rados.0.trial023.stdout:443: writing trial02342524-46 from 447119 to 1234036 tid 1 2026-02-25T08:43:51.869 INFO:tasks.rados.rados.0.trial023.stdout:443: writing trial02342524-46 from 1665416 to 2358149 tid 2 2026-02-25T08:43:51.874 INFO:tasks.rados.rados.0.trial023.stdout:443: writing trial02342524-46 from 2976554 to 3491169 tid 3 2026-02-25T08:43:51.874 INFO:tasks.rados.rados.0.trial023.stdout:444: snap_remove snap 63 2026-02-25T08:43:52.510 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~132s 2026-02-25T08:43:52.867 INFO:tasks.rados.rados.0.trial023.stdout:444: done (1 left) 2026-02-25T08:43:52.867 INFO:tasks.rados.rados.0.trial023.stdout:445: delete oid 37 current snap is 65 2026-02-25T08:43:52.867 INFO:tasks.rados.rados.0.trial023.stdout:443: finishing write tid 1 to trial02342524-46 2026-02-25T08:43:52.868 INFO:tasks.rados.rados.0.trial023.stdout:443: finishing write tid 2 to trial02342524-46 2026-02-25T08:43:52.868 INFO:tasks.rados.rados.0.trial023.stdout:443: finishing write tid 3 to trial02342524-46 2026-02-25T08:43:52.868 INFO:tasks.rados.rados.0.trial023.stdout:443: finishing write tid 4 to trial02342524-46 2026-02-25T08:43:52.868 INFO:tasks.rados.rados.0.trial023.stdout:443: finishing write tid 5 to trial02342524-46 2026-02-25T08:43:52.868 INFO:tasks.rados.rados.0.trial023.stdout:443: oid 46 updating version 0 to 130 2026-02-25T08:43:52.868 INFO:tasks.rados.rados.0.trial023.stdout:443: oid 46 version 130 is already newer than 130 2026-02-25T08:43:52.868 INFO:tasks.rados.rados.0.trial023.stdout:443: oid 46 version 130 is already newer than 130 2026-02-25T08:43:52.868 INFO:tasks.rados.rados.0.trial023.stdout:443: oid 46 version 130 is already newer than 130 2026-02-25T08:43:52.868 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 46 v 130 (ObjNum 217 snap 65 seq_num 217) dirty exists 2026-02-25T08:43:52.868 INFO:tasks.rados.rados.0.trial023.stdout:443: left oid 46 (ObjNum 217 snap 65 seq_num 217) 2026-02-25T08:43:52.907 INFO:tasks.rados.rados.0.trial023.stdout:443: done (1 left) 2026-02-25T08:43:52.908 INFO:tasks.rados.rados.0.trial023.stdout:445: done (0 left) 2026-02-25T08:43:52.908 INFO:tasks.rados.rados.0.trial023.stdout:446: delete oid 48 current snap is 65 2026-02-25T08:43:52.936 INFO:tasks.rados.rados.0.trial023.stdout:446: done (0 left) 2026-02-25T08:43:52.936 INFO:tasks.rados.rados.0.trial023.stdout:447: read oid 9 snap -1 2026-02-25T08:43:52.936 INFO:tasks.rados.rados.0.trial023.stdout:447: expect (ObjNum 212 snap 62 seq_num 212) 2026-02-25T08:43:52.937 INFO:tasks.rados.rados.0.trial023.stdout:448: read oid 33 snap 33 2026-02-25T08:43:52.937 INFO:tasks.rados.rados.0.trial023.stdout:448: expect deleted 2026-02-25T08:43:52.937 INFO:tasks.rados.rados.0.trial023.stdout:449: write oid 19 current snap is 65 2026-02-25T08:43:52.937 INFO:tasks.rados.rados.0.trial023.stdout:449: seq_num 220 ranges {497540=612664,1840862=440060,3058823=464784} 2026-02-25T08:43:52.940 INFO:tasks.rados.rados.0.trial023.stdout:449: writing trial02342524-19 from 497540 to 1110204 tid 1 2026-02-25T08:43:52.944 INFO:tasks.rados.rados.0.trial023.stdout:449: writing trial02342524-19 from 1840862 to 2280922 tid 2 2026-02-25T08:43:52.949 INFO:tasks.rados.rados.0.trial023.stdout:449: writing trial02342524-19 from 3058823 to 3523607 tid 3 2026-02-25T08:43:52.949 INFO:tasks.rados.rados.0.trial023.stdout:450: write oid 16 current snap is 65 2026-02-25T08:43:52.949 INFO:tasks.rados.rados.0.trial023.stdout:450: seq_num 221 ranges {514347=404266,1499344=700071,2686246=676056} 2026-02-25T08:43:52.952 INFO:tasks.rados.rados.0.trial023.stdout:450: writing trial02342524-16 from 514347 to 918613 tid 1 2026-02-25T08:43:52.957 INFO:tasks.rados.rados.0.trial023.stdout:450: writing trial02342524-16 from 1499344 to 2199415 tid 2 2026-02-25T08:43:52.962 INFO:tasks.rados.rados.0.trial023.stdout:450: writing trial02342524-16 from 2686246 to 3362302 tid 3 2026-02-25T08:43:52.963 INFO:tasks.rados.rados.0.trial023.stdout:451: read oid 18 snap 62 2026-02-25T08:43:52.963 INFO:tasks.rados.rados.0.trial023.stdout:451: expect (ObjNum 205 snap 58 seq_num 205) 2026-02-25T08:43:52.963 INFO:tasks.rados.rados.0.trial023.stdout:452: snap_create 2026-02-25T08:43:52.994 INFO:tasks.rados.rados.0.trial023.stdout:447: expect (ObjNum 212 snap 62 seq_num 212) 2026-02-25T08:43:53.075 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 681751 2026-02-25T08:43:53.142 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:43:53.217 INFO:tasks.rados.rados.0.trial023.stdout:449: finishing write tid 1 to trial02342524-19 2026-02-25T08:43:53.217 INFO:tasks.rados.rados.0.trial023.stdout:449: finishing write tid 2 to trial02342524-19 2026-02-25T08:43:53.217 INFO:tasks.rados.rados.0.trial023.stdout:449: finishing write tid 3 to trial02342524-19 2026-02-25T08:43:53.217 INFO:tasks.rados.rados.0.trial023.stdout:449: finishing write tid 4 to trial02342524-19 2026-02-25T08:43:53.217 INFO:tasks.rados.rados.0.trial023.stdout:449: finishing write tid 5 to trial02342524-19 2026-02-25T08:43:53.217 INFO:tasks.rados.rados.0.trial023.stdout:449: oid 19 updating version 0 to 76 2026-02-25T08:43:53.217 INFO:tasks.rados.rados.0.trial023.stdout:449: oid 19 version 76 is already newer than 76 2026-02-25T08:43:53.218 INFO:tasks.rados.rados.0.trial023.stdout:449: oid 19 version 76 is already newer than 76 2026-02-25T08:43:53.218 INFO:tasks.rados.rados.0.trial023.stdout:449: oid 19 version 76 is already newer than 76 2026-02-25T08:43:53.218 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 19 v 76 (ObjNum 220 snap 65 seq_num 220) dirty exists 2026-02-25T08:43:53.218 INFO:tasks.rados.rados.0.trial023.stdout:449: left oid 19 (ObjNum 220 snap 65 seq_num 220) 2026-02-25T08:43:53.218 INFO:tasks.rados.rados.0.trial023.stdout:451: expect (ObjNum 205 snap 58 seq_num 205) 2026-02-25T08:43:53.218 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:43:53.348 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:43:53.413 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:53.414 INFO:tasks.rados.rados.0.trial023.stdout:450: finishing write tid 1 to trial02342524-16 2026-02-25T08:43:53.414 INFO:tasks.rados.rados.0.trial023.stdout:450: finishing write tid 2 to trial02342524-16 2026-02-25T08:43:53.415 INFO:tasks.rados.rados.0.trial023.stdout:450: finishing write tid 3 to trial02342524-16 2026-02-25T08:43:53.415 INFO:tasks.rados.rados.0.trial023.stdout:450: finishing write tid 4 to trial02342524-16 2026-02-25T08:43:53.415 INFO:tasks.rados.rados.0.trial023.stdout:450: finishing write tid 5 to trial02342524-16 2026-02-25T08:43:53.415 INFO:tasks.rados.rados.0.trial023.stdout:450: oid 16 updating version 0 to 81 2026-02-25T08:43:53.415 INFO:tasks.rados.rados.0.trial023.stdout:450: oid 16 version 81 is already newer than 81 2026-02-25T08:43:53.415 INFO:tasks.rados.rados.0.trial023.stdout:450: oid 16 version 81 is already newer than 81 2026-02-25T08:43:53.415 INFO:tasks.rados.rados.0.trial023.stdout:450: oid 16 version 81 is already newer than 81 2026-02-25T08:43:53.415 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 16 v 81 (ObjNum 221 snap 65 seq_num 221) dirty exists 2026-02-25T08:43:53.415 INFO:tasks.rados.rados.0.trial023.stdout:450: left oid 16 (ObjNum 221 snap 65 seq_num 221) 2026-02-25T08:43:53.489 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:53.487+0000 7f80c42b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:53.492 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:53.491+0000 7f80c42b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:53.506 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:43:53.532 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:53.533 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:53.599 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:43:53.667 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:43:53.670 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:43:53.670 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:43:53.868 INFO:tasks.rados.rados.0.trial023.stdout:447: done (5 left) 2026-02-25T08:43:53.868 INFO:tasks.rados.rados.0.trial023.stdout:448: done (4 left) 2026-02-25T08:43:53.868 INFO:tasks.rados.rados.0.trial023.stdout:449: done (3 left) 2026-02-25T08:43:53.868 INFO:tasks.rados.rados.0.trial023.stdout:450: done (2 left) 2026-02-25T08:43:53.868 INFO:tasks.rados.rados.0.trial023.stdout:451: done (1 left) 2026-02-25T08:43:53.868 INFO:tasks.rados.rados.0.trial023.stdout:452: done (0 left) 2026-02-25T08:43:53.868 INFO:tasks.rados.rados.0.trial023.stdout:453: snap_remove snap 33 2026-02-25T08:43:54.815 INFO:tasks.rados.rados.0.trial023.stdout:453: done (0 left) 2026-02-25T08:43:54.815 INFO:tasks.rados.rados.0.trial023.stdout:454: write (excl) oid 30 current snap is 66 2026-02-25T08:43:54.815 INFO:tasks.rados.rados.0.trial023.stdout:454: seq_num 222 ranges {531154=479515,1557826=443729,2713669=487328,3679295=51859} 2026-02-25T08:43:54.818 INFO:tasks.rados.rados.0.trial023.stdout:454: writing trial02342524-30 from 531154 to 1010669 tid 1 2026-02-25T08:43:54.822 INFO:tasks.rados.rados.0.trial023.stdout:454: writing trial02342524-30 from 1557826 to 2001555 tid 2 2026-02-25T08:43:54.827 INFO:tasks.rados.rados.0.trial023.stdout:454: writing trial02342524-30 from 2713669 to 3200997 tid 3 2026-02-25T08:43:54.829 INFO:tasks.rados.rados.0.trial023.stdout:454: writing trial02342524-30 from 3679295 to 3731154 tid 4 2026-02-25T08:43:54.829 INFO:tasks.rados.rados.0.trial023.stdout:455: snap_create 2026-02-25T08:43:54.865 INFO:tasks.rados.rados.0.trial023.stdout:454: finishing write tid 1 to trial02342524-30 2026-02-25T08:43:54.879 INFO:tasks.rados.rados.0.trial023.stdout:454: finishing write tid 2 to trial02342524-30 2026-02-25T08:43:54.885 INFO:tasks.rados.rados.0.trial023.stdout:454: finishing write tid 3 to trial02342524-30 2026-02-25T08:43:54.890 INFO:tasks.rados.rados.0.trial023.stdout:454: finishing write tid 4 to trial02342524-30 2026-02-25T08:43:54.896 INFO:tasks.rados.rados.0.trial023.stdout:454: finishing write tid 5 to trial02342524-30 2026-02-25T08:43:54.898 INFO:tasks.rados.rados.0.trial023.stdout:454: finishing write tid 6 to trial02342524-30 2026-02-25T08:43:54.898 INFO:tasks.rados.rados.0.trial023.stdout:454: oid 30 updating version 0 to 66 2026-02-25T08:43:54.898 INFO:tasks.rados.rados.0.trial023.stdout:454: oid 30 version 66 is already newer than 66 2026-02-25T08:43:54.898 INFO:tasks.rados.rados.0.trial023.stdout:454: oid 30 version 66 is already newer than 66 2026-02-25T08:43:54.898 INFO:tasks.rados.rados.0.trial023.stdout:454: oid 30 version 66 is already newer than 66 2026-02-25T08:43:54.898 INFO:tasks.rados.rados.0.trial023.stdout:454: oid 30 version 66 is already newer than 62 2026-02-25T08:43:54.898 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 30 v 66 (ObjNum 222 snap 66 seq_num 222) dirty exists 2026-02-25T08:43:54.898 INFO:tasks.rados.rados.0.trial023.stdout:454: left oid 30 (ObjNum 222 snap 66 seq_num 222) 2026-02-25T08:43:55.534 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:55.610 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:55.608+0000 7f357a0cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:55.613 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:55.612+0000 7f357a0cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:55.628 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:43:55.660 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:55.661 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:55.817 INFO:tasks.rados.rados.0.trial023.stdout:454: done (1 left) 2026-02-25T08:43:55.817 INFO:tasks.rados.rados.0.trial023.stdout:455: done (0 left) 2026-02-25T08:43:55.817 INFO:tasks.rados.rados.0.trial023.stdout:456: write (excl) oid 2 current snap is 67 2026-02-25T08:43:55.817 INFO:tasks.rados.rados.0.trial023.stdout:456: seq_num 223 ranges {547961=554764,1616308=703740,2741092=698600} 2026-02-25T08:43:55.821 INFO:tasks.rados.rados.0.trial023.stdout:456: writing trial02342524-2 from 547961 to 1102725 tid 1 2026-02-25T08:43:55.825 INFO:tasks.rados.rados.0.trial023.stdout:456: writing trial02342524-2 from 1616308 to 2320048 tid 2 2026-02-25T08:43:55.830 INFO:tasks.rados.rados.0.trial023.stdout:456: writing trial02342524-2 from 2741092 to 3439692 tid 3 2026-02-25T08:43:55.831 INFO:tasks.rados.rados.0.trial023.stdout:457: read oid 35 snap -1 2026-02-25T08:43:55.831 INFO:tasks.rados.rados.0.trial023.stdout:457: expect deleted 2026-02-25T08:43:55.831 INFO:tasks.rados.rados.0.trial023.stdout:458: write (excl) oid 32 current snap is 67 2026-02-25T08:43:55.831 INFO:tasks.rados.rados.0.trial023.stdout:458: seq_num 224 ranges {564768=630013,1674790=447398,2768515=626225} 2026-02-25T08:43:55.834 INFO:tasks.rados.rados.0.trial023.stdout:458: writing trial02342524-32 from 564768 to 1194781 tid 1 2026-02-25T08:43:55.838 INFO:tasks.rados.rados.0.trial023.stdout:458: writing trial02342524-32 from 1674790 to 2122188 tid 2 2026-02-25T08:43:55.844 INFO:tasks.rados.rados.0.trial023.stdout:458: writing trial02342524-32 from 2768515 to 3394740 tid 3 2026-02-25T08:43:55.844 INFO:tasks.rados.rados.0.trial023.stdout:459: read oid 26 snap -1 2026-02-25T08:43:55.844 INFO:tasks.rados.rados.0.trial023.stdout:459: expect deleted 2026-02-25T08:43:55.844 INFO:tasks.rados.rados.0.trial023.stdout:460: read oid 19 snap 59 2026-02-25T08:43:55.844 INFO:tasks.rados.rados.0.trial023.stdout:460: expect (ObjNum 151 snap 26 seq_num 151) 2026-02-25T08:43:55.844 INFO:tasks.rados.rados.0.trial023.stdout:461: snap_remove snap 66 2026-02-25T08:43:56.815 INFO:tasks.rados.rados.0.trial023.stdout:461: done (5 left) 2026-02-25T08:43:56.815 INFO:tasks.rados.rados.0.trial023.stdout:462: snap_remove snap 54 2026-02-25T08:43:57.662 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:57.739 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:57.737+0000 7f2f4f05a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:57.742 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:57.741+0000 7f2f4f05a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:57.757 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:43:57.785 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:57.785 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:57.817 INFO:tasks.rados.rados.0.trial023.stdout:462: done (5 left) 2026-02-25T08:43:57.817 INFO:tasks.rados.rados.0.trial023.stdout:463: snap_create 2026-02-25T08:43:57.817 INFO:tasks.rados.rados.0.trial023.stdout:456: finishing write tid 1 to trial02342524-2 2026-02-25T08:43:57.817 INFO:tasks.rados.rados.0.trial023.stdout:456: finishing write tid 2 to trial02342524-2 2026-02-25T08:43:57.818 INFO:tasks.rados.rados.0.trial023.stdout:460: expect (ObjNum 151 snap 26 seq_num 151) 2026-02-25T08:43:57.863 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1187618 2026-02-25T08:43:57.933 INFO:tasks.rados.rados.0.trial023.stdout:456: finishing write tid 3 to trial02342524-2 2026-02-25T08:43:57.933 INFO:tasks.rados.rados.0.trial023.stdout:456: finishing write tid 4 to trial02342524-2 2026-02-25T08:43:57.934 INFO:tasks.rados.rados.0.trial023.stdout:456: finishing write tid 5 to trial02342524-2 2026-02-25T08:43:57.934 INFO:tasks.rados.rados.0.trial023.stdout:456: oid 2 updating version 0 to 37 2026-02-25T08:43:57.934 INFO:tasks.rados.rados.0.trial023.stdout:456: oid 2 version 37 is already newer than 37 2026-02-25T08:43:57.934 INFO:tasks.rados.rados.0.trial023.stdout:456: oid 2 version 37 is already newer than 37 2026-02-25T08:43:57.934 INFO:tasks.rados.rados.0.trial023.stdout:456: oid 2 version 37 is already newer than 34 2026-02-25T08:43:57.934 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 2 v 37 (ObjNum 223 snap 67 seq_num 223) dirty exists 2026-02-25T08:43:57.934 INFO:tasks.rados.rados.0.trial023.stdout:456: left oid 2 (ObjNum 223 snap 67 seq_num 223) 2026-02-25T08:43:57.934 INFO:tasks.rados.rados.0.trial023.stdout:458: finishing write tid 1 to trial02342524-32 2026-02-25T08:43:57.934 INFO:tasks.rados.rados.0.trial023.stdout:458: finishing write tid 2 to trial02342524-32 2026-02-25T08:43:57.934 INFO:tasks.rados.rados.0.trial023.stdout:458: finishing write tid 3 to trial02342524-32 2026-02-25T08:43:57.934 INFO:tasks.rados.rados.0.trial023.stdout:458: finishing write tid 4 to trial02342524-32 2026-02-25T08:43:57.934 INFO:tasks.rados.rados.0.trial023.stdout:458: finishing write tid 5 to trial02342524-32 2026-02-25T08:43:57.934 INFO:tasks.rados.rados.0.trial023.stdout:458: oid 32 updating version 0 to 71 2026-02-25T08:43:57.935 INFO:tasks.rados.rados.0.trial023.stdout:458: oid 32 version 71 is already newer than 71 2026-02-25T08:43:57.935 INFO:tasks.rados.rados.0.trial023.stdout:458: oid 32 version 71 is already newer than 68 2026-02-25T08:43:57.935 INFO:tasks.rados.rados.0.trial023.stdout:458: oid 32 version 71 is already newer than 71 2026-02-25T08:43:57.935 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 32 v 71 (ObjNum 224 snap 67 seq_num 224) dirty exists 2026-02-25T08:43:57.935 INFO:tasks.rados.rados.0.trial023.stdout:458: left oid 32 (ObjNum 224 snap 67 seq_num 224) 2026-02-25T08:43:58.014 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~138s 2026-02-25T08:43:58.671 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:43:58.741 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:43:58.744 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:43:58.744 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:43:58.819 INFO:tasks.rados.rados.0.trial023.stdout:456: done (5 left) 2026-02-25T08:43:58.819 INFO:tasks.rados.rados.0.trial023.stdout:457: done (4 left) 2026-02-25T08:43:58.819 INFO:tasks.rados.rados.0.trial023.stdout:458: done (3 left) 2026-02-25T08:43:58.819 INFO:tasks.rados.rados.0.trial023.stdout:459: done (2 left) 2026-02-25T08:43:58.819 INFO:tasks.rados.rados.0.trial023.stdout:460: done (1 left) 2026-02-25T08:43:58.819 INFO:tasks.rados.rados.0.trial023.stdout:463: done (0 left) 2026-02-25T08:43:58.819 INFO:tasks.rados.rados.0.trial023.stdout:464: snap_create 2026-02-25T08:43:59.786 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:59.822 INFO:tasks.rados.rados.0.trial023.stdout:464: done (0 left) 2026-02-25T08:43:59.822 INFO:tasks.rados.rados.0.trial023.stdout:465: write oid 29 current snap is 69 2026-02-25T08:43:59.822 INFO:tasks.rados.rados.0.trial023.stdout:465: seq_num 225 ranges {581575=705262,1733272=707409,3195938=437497} 2026-02-25T08:43:59.827 INFO:tasks.rados.rados.0.trial023.stdout:465: writing trial02342524-29 from 581575 to 1286837 tid 1 2026-02-25T08:43:59.832 INFO:tasks.rados.rados.0.trial023.stdout:465: writing trial02342524-29 from 1733272 to 2440681 tid 2 2026-02-25T08:43:59.836 INFO:tasks.rados.rados.0.trial023.stdout:465: writing trial02342524-29 from 3195938 to 3633435 tid 3 2026-02-25T08:43:59.837 INFO:tasks.rados.rados.0.trial023.stdout:466: write (excl) oid 19 current snap is 69 2026-02-25T08:43:59.837 INFO:tasks.rados.rados.0.trial023.stdout:466: seq_num 226 ranges {598382=780511,2075401=451067,3107008=648769} 2026-02-25T08:43:59.841 INFO:tasks.rados.rados.0.trial023.stdout:466: writing trial02342524-19 from 598382 to 1378893 tid 1 2026-02-25T08:43:59.846 INFO:tasks.rados.rados.0.trial023.stdout:466: writing trial02342524-19 from 2075401 to 2526468 tid 2 2026-02-25T08:43:59.851 INFO:tasks.rados.rados.0.trial023.stdout:466: writing trial02342524-19 from 3107008 to 3755777 tid 3 2026-02-25T08:43:59.851 INFO:tasks.rados.rados.0.trial023.stdout:467: write (excl) oid 31 current snap is 69 2026-02-25T08:43:59.851 INFO:tasks.rados.rados.0.trial023.stdout:467: seq_num 227 ranges {615189=455760,1733883=711078,3134431=460041} 2026-02-25T08:43:59.855 INFO:tasks.rados.rados.0.trial023.stdout:467: writing trial02342524-31 from 615189 to 1070949 tid 1 2026-02-25T08:43:59.860 INFO:tasks.rados.rados.0.trial023.stdout:467: writing trial02342524-31 from 1733883 to 2444961 tid 2 2026-02-25T08:43:59.864 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:59.862+0000 7fa28430b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:59.865 INFO:tasks.rados.rados.0.trial023.stdout:467: writing trial02342524-31 from 3134431 to 3594472 tid 3 2026-02-25T08:43:59.865 INFO:tasks.rados.rados.0.trial023.stdout:468: read oid 22 snap -1 2026-02-25T08:43:59.865 INFO:tasks.rados.rados.0.trial023.stdout:468: expect (ObjNum 216 snap 63 seq_num 216) 2026-02-25T08:43:59.865 INFO:tasks.rados.rados.0.trial023.stdout:469: read oid 1 snap 58 2026-02-25T08:43:59.865 INFO:tasks.rados.rados.0.trial023.stdout:469: expect (ObjNum 152 snap 26 seq_num 152) 2026-02-25T08:43:59.866 INFO:tasks.rados.rados.0.trial023.stdout:470: read oid 42 snap -1 2026-02-25T08:43:59.866 INFO:tasks.rados.rados.0.trial023.stdout:470: expect deleted 2026-02-25T08:43:59.866 INFO:tasks.rados.rados.0.trial023.stdout:471: read oid 28 snap 68 2026-02-25T08:43:59.866 INFO:tasks.rados.rados.0.trial023.stdout:471: expect (ObjNum 197 snap 55 seq_num 197) 2026-02-25T08:43:59.866 INFO:tasks.rados.rados.0.trial023.stdout:472: read oid 16 snap 59 2026-02-25T08:43:59.866 INFO:tasks.rados.rados.0.trial023.stdout:472: expect (ObjNum 90 snap 9 seq_num 90) 2026-02-25T08:43:59.866 INFO:tasks.rados.rados.0.trial023.stdout:473: delete oid 7 current snap is 69 2026-02-25T08:43:59.866 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:43:59.865+0000 7fa28430b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:59.876 INFO:tasks.rados.rados.0.trial023.stdout:466: finishing write tid 1 to trial02342524-19 2026-02-25T08:43:59.882 INFO:tasks.rados.rados.0.trial023.stdout:467: finishing write tid 1 to trial02342524-31 2026-02-25T08:43:59.884 INFO:tasks.rados.rados.0.trial023.stdout:470: done (8 left) 2026-02-25T08:43:59.884 INFO:tasks.rados.rados.0.trial023.stdout:473: done (7 left) 2026-02-25T08:43:59.884 INFO:tasks.rados.rados.0.trial023.stdout:474: read oid 46 snap -1 2026-02-25T08:43:59.884 INFO:tasks.rados.rados.0.trial023.stdout:474: expect (ObjNum 217 snap 65 seq_num 217) 2026-02-25T08:43:59.884 INFO:tasks.rados.rados.0.trial023.stdout:475: write oid 4 current snap is 69 2026-02-25T08:43:59.884 INFO:tasks.rados.rados.0.trial023.stdout:475: seq_num 229 ranges {648803=722611,1967200=714747,3421983=426820} 2026-02-25T08:43:59.889 INFO:tasks.rados.rados.0.trial023.stdout:475: writing trial02342524-4 from 648803 to 1371414 tid 1 2026-02-25T08:43:59.895 INFO:tasks.rados.rados.0.trial023.stdout:475: writing trial02342524-4 from 1967200 to 2681947 tid 2 2026-02-25T08:43:59.899 INFO:tasks.rados.rados.0.trial023.stdout:475: writing trial02342524-4 from 3421983 to 3848803 tid 3 2026-02-25T08:43:59.899 INFO:tasks.rados.rados.0.trial023.stdout:476: delete oid 25 current snap is 69 2026-02-25T08:43:59.903 INFO:tasks.rados.rados.0.trial023.stdout:468: expect (ObjNum 216 snap 63 seq_num 216) 2026-02-25T08:43:59.925 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:43:59.955 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:59.956 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:59.963 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:00.079 INFO:tasks.rados.rados.0.trial023.stdout:468: done (9 left) 2026-02-25T08:44:00.079 INFO:tasks.rados.rados.0.trial023.stdout:476: done (8 left) 2026-02-25T08:44:00.079 INFO:tasks.rados.rados.0.trial023.stdout:477: delete oid 26 current snap is 69 2026-02-25T08:44:00.079 INFO:tasks.rados.rados.0.trial023.stdout:466: finishing write tid 2 to trial02342524-19 2026-02-25T08:44:00.079 INFO:tasks.rados.rados.0.trial023.stdout:474: expect (ObjNum 217 snap 65 seq_num 217) 2026-02-25T08:44:00.106 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:00.138 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 582672 2026-02-25T08:44:00.164 INFO:tasks.rados.rados.0.trial023.stdout:466: finishing write tid 3 to trial02342524-19 2026-02-25T08:44:00.164 INFO:tasks.rados.rados.0.trial023.stdout:467: finishing write tid 2 to trial02342524-31 2026-02-25T08:44:00.164 INFO:tasks.rados.rados.0.trial023.stdout:475: finishing write tid 1 to trial02342524-4 2026-02-25T08:44:00.164 INFO:tasks.rados.rados.0.trial023.stdout:465: finishing write tid 1 to trial02342524-29 2026-02-25T08:44:00.164 INFO:tasks.rados.rados.0.trial023.stdout:474: done (8 left) 2026-02-25T08:44:00.164 INFO:tasks.rados.rados.0.trial023.stdout:477: done (7 left) 2026-02-25T08:44:00.165 INFO:tasks.rados.rados.0.trial023.stdout:478: write oid 8 current snap is 69 2026-02-25T08:44:00.165 INFO:tasks.rados.rados.0.trial023.stdout:466: finishing write tid 4 to trial02342524-19 2026-02-25T08:44:00.165 INFO:tasks.rados.rados.0.trial023.stdout:467: finishing write tid 3 to trial02342524-31 2026-02-25T08:44:00.165 INFO:tasks.rados.rados.0.trial023.stdout:466: finishing write tid 5 to trial02342524-19 2026-02-25T08:44:00.165 INFO:tasks.rados.rados.0.trial023.stdout:466: oid 19 updating version 0 to 83 2026-02-25T08:44:00.165 INFO:tasks.rados.rados.0.trial023.stdout:466: oid 19 updating version 83 to 86 2026-02-25T08:44:00.165 INFO:tasks.rados.rados.0.trial023.stdout:466: oid 19 version 86 is already newer than 86 2026-02-25T08:44:00.165 INFO:tasks.rados.rados.0.trial023.stdout:466: oid 19 version 86 is already newer than 86 2026-02-25T08:44:00.165 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 19 v 86 (ObjNum 226 snap 69 seq_num 226) dirty exists 2026-02-25T08:44:00.165 INFO:tasks.rados.rados.0.trial023.stdout:466: left oid 19 (ObjNum 226 snap 69 seq_num 226) 2026-02-25T08:44:00.165 INFO:tasks.rados.rados.0.trial023.stdout:475: finishing write tid 2 to trial02342524-4 2026-02-25T08:44:00.166 INFO:tasks.rados.rados.0.trial023.stdout:465: finishing write tid 2 to trial02342524-29 2026-02-25T08:44:00.166 INFO:tasks.rados.rados.0.trial023.stdout:478: seq_num 232 ranges {699224=548358,2026293=462074,3271546=432754} 2026-02-25T08:44:00.169 INFO:tasks.rados.rados.0.trial023.stdout:478: writing trial02342524-8 from 699224 to 1247582 tid 1 2026-02-25T08:44:00.173 INFO:tasks.rados.rados.0.trial023.stdout:478: writing trial02342524-8 from 2026293 to 2488367 tid 2 2026-02-25T08:44:00.179 INFO:tasks.rados.rados.0.trial023.stdout:478: writing trial02342524-8 from 3271546 to 3704300 tid 3 2026-02-25T08:44:00.180 INFO:tasks.rados.rados.0.trial023.stdout:466: done (7 left) 2026-02-25T08:44:00.180 INFO:tasks.rados.rados.0.trial023.stdout:479: read oid 40 snap -1 2026-02-25T08:44:00.180 INFO:tasks.rados.rados.0.trial023.stdout:479: expect (ObjNum 103 snap 12 seq_num 103) 2026-02-25T08:44:00.180 INFO:tasks.rados.rados.0.trial023.stdout:480: read oid 15 snap 55 2026-02-25T08:44:00.180 INFO:tasks.rados.rados.0.trial023.stdout:480: expect (ObjNum 162 snap 34 seq_num 162) 2026-02-25T08:44:00.180 INFO:tasks.rados.rados.0.trial023.stdout:471: expect (ObjNum 197 snap 55 seq_num 197) 2026-02-25T08:44:00.281 INFO:tasks.rados.rados.0.trial023.stdout:472: expect (ObjNum 90 snap 9 seq_num 90) 2026-02-25T08:44:00.369 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:00.418 INFO:tasks.rados.rados.0.trial023.stdout:475: finishing write tid 3 to trial02342524-4 2026-02-25T08:44:00.418 INFO:tasks.rados.rados.0.trial023.stdout:467: finishing write tid 4 to trial02342524-31 2026-02-25T08:44:00.418 INFO:tasks.rados.rados.0.trial023.stdout:481: read oid 25 snap -1 2026-02-25T08:44:00.419 INFO:tasks.rados.rados.0.trial023.stdout:481: expect deleted 2026-02-25T08:44:00.419 INFO:tasks.rados.rados.0.trial023.stdout:471: done (9 left) 2026-02-25T08:44:00.419 INFO:tasks.rados.rados.0.trial023.stdout:472: done (8 left) 2026-02-25T08:44:00.419 INFO:tasks.rados.rados.0.trial023.stdout:482: write oid 43 current snap is 69 2026-02-25T08:44:00.419 INFO:tasks.rados.rados.0.trial023.stdout:465: finishing write tid 3 to trial02342524-29 2026-02-25T08:44:00.419 INFO:tasks.rados.rados.0.trial023.stdout:482: seq_num 233 ranges {716031=623607,2084775=605732,3298969=617062} 2026-02-25T08:44:00.423 INFO:tasks.rados.rados.0.trial023.stdout:482: writing trial02342524-43 from 716031 to 1339638 tid 1 2026-02-25T08:44:00.428 INFO:tasks.rados.rados.0.trial023.stdout:482: writing trial02342524-43 from 2084775 to 2690507 tid 2 2026-02-25T08:44:00.433 INFO:tasks.rados.rados.0.trial023.stdout:482: writing trial02342524-43 from 3298969 to 3916031 tid 3 2026-02-25T08:44:00.433 INFO:tasks.rados.rados.0.trial023.stdout:483: delete oid 10 current snap is 69 2026-02-25T08:44:00.434 INFO:tasks.rados.rados.0.trial023.stdout:469: expect (ObjNum 152 snap 26 seq_num 152) 2026-02-25T08:44:00.435 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 417743 2026-02-25T08:44:00.462 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:00.494 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:00.526 INFO:tasks.rados.rados.0.trial023.stdout:475: finishing write tid 4 to trial02342524-4 2026-02-25T08:44:00.526 INFO:tasks.rados.rados.0.trial023.stdout:467: finishing write tid 5 to trial02342524-31 2026-02-25T08:44:00.526 INFO:tasks.rados.rados.0.trial023.stdout:467: oid 31 updating version 0 to 27 2026-02-25T08:44:00.526 INFO:tasks.rados.rados.0.trial023.stdout:467: oid 31 version 27 is already newer than 27 2026-02-25T08:44:00.527 INFO:tasks.rados.rados.0.trial023.stdout:467: oid 31 version 27 is already newer than 24 2026-02-25T08:44:00.527 INFO:tasks.rados.rados.0.trial023.stdout:467: oid 31 version 27 is already newer than 27 2026-02-25T08:44:00.527 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 31 v 27 (ObjNum 227 snap 69 seq_num 227) dirty exists 2026-02-25T08:44:00.527 INFO:tasks.rados.rados.0.trial023.stdout:467: left oid 31 (ObjNum 227 snap 69 seq_num 227) 2026-02-25T08:44:00.527 INFO:tasks.rados.rados.0.trial023.stdout:475: finishing write tid 5 to trial02342524-4 2026-02-25T08:44:00.527 INFO:tasks.rados.rados.0.trial023.stdout:475: oid 4 updating version 0 to 123 2026-02-25T08:44:00.527 INFO:tasks.rados.rados.0.trial023.stdout:475: oid 4 version 123 is already newer than 123 2026-02-25T08:44:00.527 INFO:tasks.rados.rados.0.trial023.stdout:475: oid 4 version 123 is already newer than 123 2026-02-25T08:44:00.527 INFO:tasks.rados.rados.0.trial023.stdout:475: oid 4 version 123 is already newer than 123 2026-02-25T08:44:00.527 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 4 v 123 (ObjNum 229 snap 69 seq_num 229) dirty exists 2026-02-25T08:44:00.527 INFO:tasks.rados.rados.0.trial023.stdout:475: left oid 4 (ObjNum 229 snap 69 seq_num 229) 2026-02-25T08:44:00.527 INFO:tasks.rados.rados.0.trial023.stdout:465: finishing write tid 4 to trial02342524-29 2026-02-25T08:44:00.527 INFO:tasks.rados.rados.0.trial023.stdout:465: finishing write tid 5 to trial02342524-29 2026-02-25T08:44:00.528 INFO:tasks.rados.rados.0.trial023.stdout:465: oid 29 updating version 0 to 112 2026-02-25T08:44:00.528 INFO:tasks.rados.rados.0.trial023.stdout:465: oid 29 version 112 is already newer than 112 2026-02-25T08:44:00.528 INFO:tasks.rados.rados.0.trial023.stdout:465: oid 29 version 112 is already newer than 112 2026-02-25T08:44:00.528 INFO:tasks.rados.rados.0.trial023.stdout:465: oid 29 version 112 is already newer than 112 2026-02-25T08:44:00.528 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 29 v 112 (ObjNum 225 snap 69 seq_num 225) dirty exists 2026-02-25T08:44:00.528 INFO:tasks.rados.rados.0.trial023.stdout:465: left oid 29 (ObjNum 225 snap 69 seq_num 225) 2026-02-25T08:44:00.528 INFO:tasks.rados.rados.0.trial023.stdout:478: finishing write tid 1 to trial02342524-8 2026-02-25T08:44:00.528 INFO:tasks.rados.rados.0.trial023.stdout:478: finishing write tid 2 to trial02342524-8 2026-02-25T08:44:00.528 INFO:tasks.rados.rados.0.trial023.stdout:478: finishing write tid 3 to trial02342524-8 2026-02-25T08:44:00.528 INFO:tasks.rados.rados.0.trial023.stdout:479: expect (ObjNum 103 snap 12 seq_num 103) 2026-02-25T08:44:00.588 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:00.620 INFO:tasks.rados.rados.0.trial023.stdout:478: finishing write tid 4 to trial02342524-8 2026-02-25T08:44:00.620 INFO:tasks.rados.rados.0.trial023.stdout:465: done (9 left) 2026-02-25T08:44:00.620 INFO:tasks.rados.rados.0.trial023.stdout:467: done (8 left) 2026-02-25T08:44:00.620 INFO:tasks.rados.rados.0.trial023.stdout:469: done (7 left) 2026-02-25T08:44:00.620 INFO:tasks.rados.rados.0.trial023.stdout:475: done (6 left) 2026-02-25T08:44:00.620 INFO:tasks.rados.rados.0.trial023.stdout:479: done (5 left) 2026-02-25T08:44:00.620 INFO:tasks.rados.rados.0.trial023.stdout:483: done (4 left) 2026-02-25T08:44:00.620 INFO:tasks.rados.rados.0.trial023.stdout:484: read oid 2 snap -1 2026-02-25T08:44:00.621 INFO:tasks.rados.rados.0.trial023.stdout:484: expect (ObjNum 223 snap 67 seq_num 223) 2026-02-25T08:44:00.621 INFO:tasks.rados.rados.0.trial023.stdout:485: read oid 1 snap -1 2026-02-25T08:44:00.621 INFO:tasks.rados.rados.0.trial023.stdout:485: expect (ObjNum 206 snap 59 seq_num 206) 2026-02-25T08:44:00.621 INFO:tasks.rados.rados.0.trial023.stdout:486: write (excl) oid 35 current snap is 69 2026-02-25T08:44:00.621 INFO:tasks.rados.rados.0.trial023.stdout:478: finishing write tid 5 to trial02342524-8 2026-02-25T08:44:00.621 INFO:tasks.rados.rados.0.trial023.stdout:478: oid 8 updating version 0 to 24 2026-02-25T08:44:00.621 INFO:tasks.rados.rados.0.trial023.stdout:478: oid 8 version 24 is already newer than 24 2026-02-25T08:44:00.621 INFO:tasks.rados.rados.0.trial023.stdout:478: oid 8 version 24 is already newer than 24 2026-02-25T08:44:00.621 INFO:tasks.rados.rados.0.trial023.stdout:478: oid 8 version 24 is already newer than 24 2026-02-25T08:44:00.621 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 8 v 24 (ObjNum 232 snap 69 seq_num 232) dirty exists 2026-02-25T08:44:00.621 INFO:tasks.rados.rados.0.trial023.stdout:478: left oid 8 (ObjNum 232 snap 69 seq_num 232) 2026-02-25T08:44:00.621 INFO:tasks.rados.rados.0.trial023.stdout:480: expect (ObjNum 162 snap 34 seq_num 162) 2026-02-25T08:44:00.654 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2110961 2026-02-25T08:44:00.659 INFO:tasks.rados.rados.0.trial023.stdout:486: seq_num 235 ranges {749645=774105,2085386=609401,3237462=712183} 2026-02-25T08:44:00.665 INFO:tasks.rados.rados.0.trial023.stdout:486: writing trial02342524-35 from 749645 to 1523750 tid 1 2026-02-25T08:44:00.669 INFO:tasks.rados.rados.0.trial023.stdout:486: writing trial02342524-35 from 2085386 to 2694787 tid 2 2026-02-25T08:44:00.675 INFO:tasks.rados.rados.0.trial023.stdout:486: writing trial02342524-35 from 3237462 to 3949645 tid 3 2026-02-25T08:44:00.675 INFO:tasks.rados.rados.0.trial023.stdout:478: done (6 left) 2026-02-25T08:44:00.675 INFO:tasks.rados.rados.0.trial023.stdout:480: done (5 left) 2026-02-25T08:44:00.675 INFO:tasks.rados.rados.0.trial023.stdout:481: done (4 left) 2026-02-25T08:44:00.675 INFO:tasks.rados.rados.0.trial023.stdout:487: delete oid 49 current snap is 69 2026-02-25T08:44:00.675 INFO:tasks.rados.rados.0.trial023.stdout:482: finishing write tid 1 to trial02342524-43 2026-02-25T08:44:00.675 INFO:tasks.rados.rados.0.trial023.stdout:482: finishing write tid 2 to trial02342524-43 2026-02-25T08:44:00.675 INFO:tasks.rados.rados.0.trial023.stdout:482: finishing write tid 3 to trial02342524-43 2026-02-25T08:44:00.675 INFO:tasks.rados.rados.0.trial023.stdout:482: finishing write tid 4 to trial02342524-43 2026-02-25T08:44:00.676 INFO:tasks.rados.rados.0.trial023.stdout:482: finishing write tid 5 to trial02342524-43 2026-02-25T08:44:00.676 INFO:tasks.rados.rados.0.trial023.stdout:482: oid 43 updating version 0 to 22 2026-02-25T08:44:00.676 INFO:tasks.rados.rados.0.trial023.stdout:482: oid 43 version 22 is already newer than 22 2026-02-25T08:44:00.676 INFO:tasks.rados.rados.0.trial023.stdout:482: oid 43 version 22 is already newer than 22 2026-02-25T08:44:00.676 INFO:tasks.rados.rados.0.trial023.stdout:482: oid 43 version 22 is already newer than 22 2026-02-25T08:44:00.676 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 43 v 22 (ObjNum 233 snap 69 seq_num 233) dirty exists 2026-02-25T08:44:00.676 INFO:tasks.rados.rados.0.trial023.stdout:482: left oid 43 (ObjNum 233 snap 69 seq_num 233) 2026-02-25T08:44:00.676 INFO:tasks.rados.rados.0.trial023.stdout:484: expect (ObjNum 223 snap 67 seq_num 223) 2026-02-25T08:44:00.738 INFO:tasks.rados.rados.0.trial023.stdout:485: expect (ObjNum 206 snap 59 seq_num 206) 2026-02-25T08:44:00.820 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 674824 2026-02-25T08:44:00.851 INFO:tasks.rados.rados.0.trial023.stdout:486: finishing write tid 1 to trial02342524-35 2026-02-25T08:44:00.851 INFO:tasks.rados.rados.0.trial023.stdout:486: finishing write tid 2 to trial02342524-35 2026-02-25T08:44:00.851 INFO:tasks.rados.rados.0.trial023.stdout:486: finishing write tid 3 to trial02342524-35 2026-02-25T08:44:00.852 INFO:tasks.rados.rados.0.trial023.stdout:486: finishing write tid 4 to trial02342524-35 2026-02-25T08:44:00.852 INFO:tasks.rados.rados.0.trial023.stdout:486: finishing write tid 5 to trial02342524-35 2026-02-25T08:44:00.852 INFO:tasks.rados.rados.0.trial023.stdout:486: oid 35 updating version 0 to 139 2026-02-25T08:44:00.852 INFO:tasks.rados.rados.0.trial023.stdout:486: oid 35 updating version 139 to 142 2026-02-25T08:44:00.852 INFO:tasks.rados.rados.0.trial023.stdout:486: oid 35 version 142 is already newer than 142 2026-02-25T08:44:00.852 INFO:tasks.rados.rados.0.trial023.stdout:486: oid 35 version 142 is already newer than 142 2026-02-25T08:44:00.852 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 35 v 142 (ObjNum 235 snap 69 seq_num 235) dirty exists 2026-02-25T08:44:00.852 INFO:tasks.rados.rados.0.trial023.stdout:486: left oid 35 (ObjNum 235 snap 69 seq_num 235) 2026-02-25T08:44:00.852 INFO:tasks.rados.rados.0.trial023.stdout:482: done (4 left) 2026-02-25T08:44:00.852 INFO:tasks.rados.rados.0.trial023.stdout:484: done (3 left) 2026-02-25T08:44:00.852 INFO:tasks.rados.rados.0.trial023.stdout:485: done (2 left) 2026-02-25T08:44:00.852 INFO:tasks.rados.rados.0.trial023.stdout:486: done (1 left) 2026-02-25T08:44:00.852 INFO:tasks.rados.rados.0.trial023.stdout:487: done (0 left) 2026-02-25T08:44:00.853 INFO:tasks.rados.rados.0.trial023.stdout:488: read oid 3 snap 36 2026-02-25T08:44:00.853 INFO:tasks.rados.rados.0.trial023.stdout:488: expect (ObjNum 112 snap 12 seq_num 112) 2026-02-25T08:44:00.853 INFO:tasks.rados.rados.0.trial023.stdout:489: snap_remove snap 65 2026-02-25T08:44:01.828 INFO:tasks.rados.rados.0.trial023.stdout:489: done (1 left) 2026-02-25T08:44:01.828 INFO:tasks.rados.rados.0.trial023.stdout:490: read oid 7 snap -1 2026-02-25T08:44:01.828 INFO:tasks.rados.rados.0.trial023.stdout:490: expect deleted 2026-02-25T08:44:01.828 INFO:tasks.rados.rados.0.trial023.stdout:491: delete oid 27 current snap is 69 2026-02-25T08:44:01.828 INFO:tasks.rados.rados.0.trial023.stdout:488: expect (ObjNum 112 snap 12 seq_num 112) 2026-02-25T08:44:01.873 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:01.956 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:01.963 INFO:tasks.rados.rados.0.trial023.stdout:488: done (2 left) 2026-02-25T08:44:01.963 INFO:tasks.rados.rados.0.trial023.stdout:490: done (1 left) 2026-02-25T08:44:01.963 INFO:tasks.rados.rados.0.trial023.stdout:491: done (0 left) 2026-02-25T08:44:01.963 INFO:tasks.rados.rados.0.trial023.stdout:492: snap_remove snap 61 2026-02-25T08:44:02.031 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:02.029+0000 7fb5a1ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:02.034 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:02.033+0000 7fb5a1ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:02.049 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:44:02.074 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:02.075 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:02.832 INFO:tasks.rados.rados.0.trial023.stdout:492: done (0 left) 2026-02-25T08:44:02.833 INFO:tasks.rados.rados.0.trial023.stdout:493: write oid 19 current snap is 69 2026-02-25T08:44:02.833 INFO:tasks.rados.rados.0.trial023.stdout:493: seq_num 238 ranges {400066=716205,1860832=139234} 2026-02-25T08:44:02.837 INFO:tasks.rados.rados.0.trial023.stdout:493: writing trial02342524-19 from 400066 to 1116271 tid 1 2026-02-25T08:44:02.840 INFO:tasks.rados.rados.0.trial023.stdout:493: writing trial02342524-19 from 1860832 to 2000066 tid 2 2026-02-25T08:44:02.840 INFO:tasks.rados.rados.0.trial023.stdout:494: delete oid 25 current snap is 69 2026-02-25T08:44:02.852 INFO:tasks.rados.rados.0.trial023.stdout:494: done (1 left) 2026-02-25T08:44:02.852 INFO:tasks.rados.rados.0.trial023.stdout:495: read oid 17 snap -1 2026-02-25T08:44:02.852 INFO:tasks.rados.rados.0.trial023.stdout:495: expect (ObjNum 178 snap 42 seq_num 178) 2026-02-25T08:44:02.852 INFO:tasks.rados.rados.0.trial023.stdout:496: read oid 10 snap -1 2026-02-25T08:44:02.853 INFO:tasks.rados.rados.0.trial023.stdout:496: expect deleted 2026-02-25T08:44:02.853 INFO:tasks.rados.rados.0.trial023.stdout:497: read oid 11 snap -1 2026-02-25T08:44:02.853 INFO:tasks.rados.rados.0.trial023.stdout:497: expect (ObjNum 189 snap 48 seq_num 189) 2026-02-25T08:44:02.853 INFO:tasks.rados.rados.0.trial023.stdout:498: snap_remove snap 56 2026-02-25T08:44:03.517 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~143s 2026-02-25T08:44:03.745 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:44:03.814 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:44:03.817 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:44:03.818 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:44:03.834 INFO:tasks.rados.rados.0.trial023.stdout:498: done (4 left) 2026-02-25T08:44:03.835 INFO:tasks.rados.rados.0.trial023.stdout:499: write oid 9 current snap is 69 2026-02-25T08:44:03.835 INFO:tasks.rados.rados.0.trial023.stdout:499: seq_num 240 ranges {433680=466703,1577796=455884} 2026-02-25T08:44:03.838 INFO:tasks.rados.rados.0.trial023.stdout:499: writing trial02342524-9 from 433680 to 900383 tid 1 2026-02-25T08:44:03.843 INFO:tasks.rados.rados.0.trial023.stdout:499: writing trial02342524-9 from 1577796 to 2033680 tid 2 2026-02-25T08:44:03.843 INFO:tasks.rados.rados.0.trial023.stdout:500: read oid 4 snap -1 2026-02-25T08:44:03.843 INFO:tasks.rados.rados.0.trial023.stdout:500: expect (ObjNum 229 snap 69 seq_num 229) 2026-02-25T08:44:03.843 INFO:tasks.rados.rados.0.trial023.stdout:501: read oid 30 snap -1 2026-02-25T08:44:03.843 INFO:tasks.rados.rados.0.trial023.stdout:501: expect (ObjNum 222 snap 66 seq_num 222) 2026-02-25T08:44:03.843 INFO:tasks.rados.rados.0.trial023.stdout:502: snap_remove snap 55 2026-02-25T08:44:04.076 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:04.149 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:04.147+0000 7f930999c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:04.152 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:04.151+0000 7f930999c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:04.167 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:44:04.195 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:04.195 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:04.837 INFO:tasks.rados.rados.0.trial023.stdout:502: done (7 left) 2026-02-25T08:44:04.837 INFO:tasks.rados.rados.0.trial023.stdout:503: delete oid 48 current snap is 69 2026-02-25T08:44:04.837 INFO:tasks.rados.rados.0.trial023.stdout:495: expect (ObjNum 178 snap 42 seq_num 178) 2026-02-25T08:44:04.856 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2518017 2026-02-25T08:44:04.895 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 81295 2026-02-25T08:44:04.932 INFO:tasks.rados.rados.0.trial023.stdout:497: expect (ObjNum 189 snap 48 seq_num 189) 2026-02-25T08:44:04.932 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:04.986 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:05.013 INFO:tasks.rados.rados.0.trial023.stdout:493: finishing write tid 1 to trial02342524-19 2026-02-25T08:44:05.013 INFO:tasks.rados.rados.0.trial023.stdout:493: finishing write tid 2 to trial02342524-19 2026-02-25T08:44:05.013 INFO:tasks.rados.rados.0.trial023.stdout:493: finishing write tid 3 to trial02342524-19 2026-02-25T08:44:05.014 INFO:tasks.rados.rados.0.trial023.stdout:493: finishing write tid 4 to trial02342524-19 2026-02-25T08:44:05.014 INFO:tasks.rados.rados.0.trial023.stdout:493: oid 19 updating version 0 to 89 2026-02-25T08:44:05.014 INFO:tasks.rados.rados.0.trial023.stdout:493: oid 19 version 89 is already newer than 89 2026-02-25T08:44:05.014 INFO:tasks.rados.rados.0.trial023.stdout:493: oid 19 version 89 is already newer than 89 2026-02-25T08:44:05.014 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 19 v 89 (ObjNum 238 snap 69 seq_num 238) dirty exists 2026-02-25T08:44:05.014 INFO:tasks.rados.rados.0.trial023.stdout:493: left oid 19 (ObjNum 238 snap 69 seq_num 238) 2026-02-25T08:44:05.014 INFO:tasks.rados.rados.0.trial023.stdout:493: done (7 left) 2026-02-25T08:44:05.014 INFO:tasks.rados.rados.0.trial023.stdout:495: done (6 left) 2026-02-25T08:44:05.014 INFO:tasks.rados.rados.0.trial023.stdout:496: done (5 left) 2026-02-25T08:44:05.014 INFO:tasks.rados.rados.0.trial023.stdout:497: done (4 left) 2026-02-25T08:44:05.014 INFO:tasks.rados.rados.0.trial023.stdout:503: done (3 left) 2026-02-25T08:44:05.015 INFO:tasks.rados.rados.0.trial023.stdout:504: delete oid 1 current snap is 69 2026-02-25T08:44:05.015 INFO:tasks.rados.rados.0.trial023.stdout:501: expect (ObjNum 222 snap 66 seq_num 222) 2026-02-25T08:44:05.050 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2427137 2026-02-25T08:44:05.074 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 3291247 2026-02-25T08:44:05.081 INFO:tasks.rados.rados.0.trial023.stdout:500: expect (ObjNum 229 snap 69 seq_num 229) 2026-02-25T08:44:05.098 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1731775 2026-02-25T08:44:05.142 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:05.285 INFO:tasks.rados.rados.0.trial023.stdout:499: finishing write tid 1 to trial02342524-9 2026-02-25T08:44:05.285 INFO:tasks.rados.rados.0.trial023.stdout:499: finishing write tid 2 to trial02342524-9 2026-02-25T08:44:05.286 INFO:tasks.rados.rados.0.trial023.stdout:499: finishing write tid 3 to trial02342524-9 2026-02-25T08:44:05.286 INFO:tasks.rados.rados.0.trial023.stdout:499: finishing write tid 4 to trial02342524-9 2026-02-25T08:44:05.286 INFO:tasks.rados.rados.0.trial023.stdout:499: oid 9 updating version 0 to 148 2026-02-25T08:44:05.286 INFO:tasks.rados.rados.0.trial023.stdout:499: oid 9 version 148 is already newer than 148 2026-02-25T08:44:05.286 INFO:tasks.rados.rados.0.trial023.stdout:499: oid 9 version 148 is already newer than 148 2026-02-25T08:44:05.286 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 9 v 148 (ObjNum 240 snap 69 seq_num 240) dirty exists 2026-02-25T08:44:05.286 INFO:tasks.rados.rados.0.trial023.stdout:499: left oid 9 (ObjNum 240 snap 69 seq_num 240) 2026-02-25T08:44:05.286 INFO:tasks.rados.rados.0.trial023.stdout:499: done (3 left) 2026-02-25T08:44:05.286 INFO:tasks.rados.rados.0.trial023.stdout:500: done (2 left) 2026-02-25T08:44:05.286 INFO:tasks.rados.rados.0.trial023.stdout:501: done (1 left) 2026-02-25T08:44:05.286 INFO:tasks.rados.rados.0.trial023.stdout:504: done (0 left) 2026-02-25T08:44:05.286 INFO:tasks.rados.rados.0.trial023.stdout:505: read oid 11 snap 36 2026-02-25T08:44:05.287 INFO:tasks.rados.rados.0.trial023.stdout:505: expect deleted 2026-02-25T08:44:05.287 INFO:tasks.rados.rados.0.trial023.stdout:506: snap_create 2026-02-25T08:44:05.841 INFO:tasks.rados.rados.0.trial023.stdout:505: done (1 left) 2026-02-25T08:44:05.841 INFO:tasks.rados.rados.0.trial023.stdout:506: done (0 left) 2026-02-25T08:44:05.841 INFO:tasks.rados.rados.0.trial023.stdout:507: read oid 27 snap -1 2026-02-25T08:44:05.841 INFO:tasks.rados.rados.0.trial023.stdout:507: expect deleted 2026-02-25T08:44:05.842 INFO:tasks.rados.rados.0.trial023.stdout:508: write (excl) oid 21 current snap is 70 2026-02-25T08:44:05.842 INFO:tasks.rados.rados.0.trial023.stdout:508: seq_num 243 ranges {484101=692450,1636889=447212} 2026-02-25T08:44:05.846 INFO:tasks.rados.rados.0.trial023.stdout:508: writing trial02342524-21 from 484101 to 1176551 tid 1 2026-02-25T08:44:05.850 INFO:tasks.rados.rados.0.trial023.stdout:508: writing trial02342524-21 from 1636889 to 2084101 tid 2 2026-02-25T08:44:05.850 INFO:tasks.rados.rados.0.trial023.stdout:509: read oid 38 snap 59 2026-02-25T08:44:05.850 INFO:tasks.rados.rados.0.trial023.stdout:509: expect (ObjNum 167 snap 36 seq_num 167) 2026-02-25T08:44:05.850 INFO:tasks.rados.rados.0.trial023.stdout:510: write (excl) oid 15 current snap is 70 2026-02-25T08:44:05.851 INFO:tasks.rados.rados.0.trial023.stdout:510: seq_num 244 ranges {500908=484052,1411724=689184} 2026-02-25T08:44:05.854 INFO:tasks.rados.rados.0.trial023.stdout:510: writing trial02342524-15 from 500908 to 984960 tid 1 2026-02-25T08:44:05.858 INFO:tasks.rados.rados.0.trial023.stdout:510: writing trial02342524-15 from 1411724 to 2100908 tid 2 2026-02-25T08:44:05.859 INFO:tasks.rados.rados.0.trial023.stdout:511: snap_remove snap 69 2026-02-25T08:44:06.197 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:06.274 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:06.272+0000 7fd99b30b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:06.277 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:06.275+0000 7fd99b30b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:06.292 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:44:06.319 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:06.319 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:06.846 INFO:tasks.rados.rados.0.trial023.stdout:511: done (4 left) 2026-02-25T08:44:06.846 INFO:tasks.rados.rados.0.trial023.stdout:512: delete oid 50 current snap is 70 2026-02-25T08:44:06.846 INFO:tasks.rados.rados.0.trial023.stdout:508: finishing write tid 1 to trial02342524-21 2026-02-25T08:44:06.846 INFO:tasks.rados.rados.0.trial023.stdout:510: finishing write tid 1 to trial02342524-15 2026-02-25T08:44:06.847 INFO:tasks.rados.rados.0.trial023.stdout:508: finishing write tid 2 to trial02342524-21 2026-02-25T08:44:06.847 INFO:tasks.rados.rados.0.trial023.stdout:508: finishing write tid 3 to trial02342524-21 2026-02-25T08:44:06.847 INFO:tasks.rados.rados.0.trial023.stdout:508: finishing write tid 4 to trial02342524-21 2026-02-25T08:44:06.847 INFO:tasks.rados.rados.0.trial023.stdout:508: oid 21 updating version 0 to 92 2026-02-25T08:44:06.847 INFO:tasks.rados.rados.0.trial023.stdout:508: oid 21 version 92 is already newer than 90 2026-02-25T08:44:06.847 INFO:tasks.rados.rados.0.trial023.stdout:508: oid 21 version 92 is already newer than 92 2026-02-25T08:44:06.847 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 21 v 92 (ObjNum 243 snap 70 seq_num 243) dirty exists 2026-02-25T08:44:06.847 INFO:tasks.rados.rados.0.trial023.stdout:508: left oid 21 (ObjNum 243 snap 70 seq_num 243) 2026-02-25T08:44:06.847 INFO:tasks.rados.rados.0.trial023.stdout:509: expect (ObjNum 167 snap 36 seq_num 167) 2026-02-25T08:44:06.866 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2491471 2026-02-25T08:44:06.899 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:06.935 INFO:tasks.rados.rados.0.trial023.stdout:510: finishing write tid 2 to trial02342524-15 2026-02-25T08:44:06.935 INFO:tasks.rados.rados.0.trial023.stdout:510: finishing write tid 3 to trial02342524-15 2026-02-25T08:44:06.935 INFO:tasks.rados.rados.0.trial023.stdout:510: finishing write tid 4 to trial02342524-15 2026-02-25T08:44:06.936 INFO:tasks.rados.rados.0.trial023.stdout:510: oid 15 updating version 0 to 76 2026-02-25T08:44:06.936 INFO:tasks.rados.rados.0.trial023.stdout:510: oid 15 version 76 is already newer than 76 2026-02-25T08:44:06.936 INFO:tasks.rados.rados.0.trial023.stdout:510: oid 15 version 76 is already newer than 74 2026-02-25T08:44:06.936 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 15 v 76 (ObjNum 244 snap 70 seq_num 244) dirty exists 2026-02-25T08:44:06.936 INFO:tasks.rados.rados.0.trial023.stdout:510: left oid 15 (ObjNum 244 snap 70 seq_num 244) 2026-02-25T08:44:06.936 INFO:tasks.rados.rados.0.trial023.stdout:507: done (4 left) 2026-02-25T08:44:06.936 INFO:tasks.rados.rados.0.trial023.stdout:508: done (3 left) 2026-02-25T08:44:06.936 INFO:tasks.rados.rados.0.trial023.stdout:509: done (2 left) 2026-02-25T08:44:06.936 INFO:tasks.rados.rados.0.trial023.stdout:510: done (1 left) 2026-02-25T08:44:06.936 INFO:tasks.rados.rados.0.trial023.stdout:512: done (0 left) 2026-02-25T08:44:06.936 INFO:tasks.rados.rados.0.trial023.stdout:513: snap_remove snap 59 2026-02-25T08:44:07.848 INFO:tasks.rados.rados.0.trial023.stdout:513: done (0 left) 2026-02-25T08:44:07.849 INFO:tasks.rados.rados.0.trial023.stdout:514: snap_create 2026-02-25T08:44:08.320 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:08.398 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:08.394+0000 7f892805a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:08.400 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:08.398+0000 7f892805a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:08.416 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:44:08.443 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:08.444 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:08.818 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:44:08.850 INFO:tasks.rados.rados.0.trial023.stdout:514: done (0 left) 2026-02-25T08:44:08.850 INFO:tasks.rados.rados.0.trial023.stdout:515: read oid 9 snap -1 2026-02-25T08:44:08.850 INFO:tasks.rados.rados.0.trial023.stdout:515: expect (ObjNum 240 snap 69 seq_num 240) 2026-02-25T08:44:08.850 INFO:tasks.rados.rados.0.trial023.stdout:516: snap_create 2026-02-25T08:44:08.881 INFO:tasks.rados.rados.0.trial023.stdout:515: expect (ObjNum 240 snap 69 seq_num 240) 2026-02-25T08:44:08.891 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:44:08.894 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:44:08.895 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:44:08.924 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:09.021 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~149s 2026-02-25T08:44:09.818 INFO:tasks.rados.rados.0.trial023.stdout:515: done (1 left) 2026-02-25T08:44:09.818 INFO:tasks.rados.rados.0.trial023.stdout:516: done (0 left) 2026-02-25T08:44:09.818 INFO:tasks.rados.rados.0.trial023.stdout:517: write (excl) oid 24 current snap is 72 2026-02-25T08:44:09.819 INFO:tasks.rados.rados.0.trial023.stdout:517: seq_num 246 ranges {534522=634550,1812335=322187} 2026-02-25T08:44:09.822 INFO:tasks.rados.rados.0.trial023.stdout:517: writing trial02342524-24 from 534522 to 1169072 tid 1 2026-02-25T08:44:09.826 INFO:tasks.rados.rados.0.trial023.stdout:517: writing trial02342524-24 from 1812335 to 2134522 tid 2 2026-02-25T08:44:09.826 INFO:tasks.rados.rados.0.trial023.stdout:518: read oid 4 snap -1 2026-02-25T08:44:09.826 INFO:tasks.rados.rados.0.trial023.stdout:518: expect (ObjNum 229 snap 69 seq_num 229) 2026-02-25T08:44:09.826 INFO:tasks.rados.rados.0.trial023.stdout:519: snap_remove snap 40 2026-02-25T08:44:10.445 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:10.522 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:10.518+0000 7f604b574640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:10.524 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:10.522+0000 7f604b574640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:10.539 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:44:10.565 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:10.566 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:10.819 INFO:tasks.rados.rados.0.trial023.stdout:519: done (2 left) 2026-02-25T08:44:10.820 INFO:tasks.rados.rados.0.trial023.stdout:520: snap_create 2026-02-25T08:44:10.820 INFO:tasks.rados.rados.0.trial023.stdout:517: finishing write tid 1 to trial02342524-24 2026-02-25T08:44:10.820 INFO:tasks.rados.rados.0.trial023.stdout:518: expect (ObjNum 229 snap 69 seq_num 229) 2026-02-25T08:44:10.962 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:11.032 INFO:tasks.rados.rados.0.trial023.stdout:517: finishing write tid 2 to trial02342524-24 2026-02-25T08:44:11.033 INFO:tasks.rados.rados.0.trial023.stdout:517: finishing write tid 3 to trial02342524-24 2026-02-25T08:44:11.033 INFO:tasks.rados.rados.0.trial023.stdout:517: finishing write tid 4 to trial02342524-24 2026-02-25T08:44:11.033 INFO:tasks.rados.rados.0.trial023.stdout:517: oid 24 updating version 0 to 75 2026-02-25T08:44:11.033 INFO:tasks.rados.rados.0.trial023.stdout:517: oid 24 version 75 is already newer than 73 2026-02-25T08:44:11.033 INFO:tasks.rados.rados.0.trial023.stdout:517: oid 24 version 75 is already newer than 75 2026-02-25T08:44:11.033 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 24 v 75 (ObjNum 246 snap 72 seq_num 246) dirty exists 2026-02-25T08:44:11.033 INFO:tasks.rados.rados.0.trial023.stdout:517: left oid 24 (ObjNum 246 snap 72 seq_num 246) 2026-02-25T08:44:11.820 INFO:tasks.rados.rados.0.trial023.stdout:517: done (2 left) 2026-02-25T08:44:11.820 INFO:tasks.rados.rados.0.trial023.stdout:518: done (1 left) 2026-02-25T08:44:11.821 INFO:tasks.rados.rados.0.trial023.stdout:520: done (0 left) 2026-02-25T08:44:11.821 INFO:tasks.rados.rados.0.trial023.stdout:521: read oid 46 snap -1 2026-02-25T08:44:11.821 INFO:tasks.rados.rados.0.trial023.stdout:521: expect (ObjNum 217 snap 65 seq_num 217) 2026-02-25T08:44:11.821 INFO:tasks.rados.rados.0.trial023.stdout:522: read oid 12 snap 64 2026-02-25T08:44:11.821 INFO:tasks.rados.rados.0.trial023.stdout:522: expect deleted 2026-02-25T08:44:11.821 INFO:tasks.rados.rados.0.trial023.stdout:523: snap_remove snap 67 2026-02-25T08:44:12.567 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:12.645 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:12.642+0000 7fc615d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:12.647 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:12.645+0000 7fc615d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:12.662 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:44:12.689 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:12.690 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:12.823 INFO:tasks.rados.rados.0.trial023.stdout:523: done (2 left) 2026-02-25T08:44:12.823 INFO:tasks.rados.rados.0.trial023.stdout:524: delete oid 18 current snap is 73 2026-02-25T08:44:12.823 INFO:tasks.rados.rados.0.trial023.stdout:521: expect (ObjNum 217 snap 65 seq_num 217) 2026-02-25T08:44:12.915 INFO:tasks.rados.rados.0.trial023.stdout:521: done (2 left) 2026-02-25T08:44:12.915 INFO:tasks.rados.rados.0.trial023.stdout:522: done (1 left) 2026-02-25T08:44:12.915 INFO:tasks.rados.rados.0.trial023.stdout:524: done (0 left) 2026-02-25T08:44:12.915 INFO:tasks.rados.rados.0.trial023.stdout:525: snap_remove snap 51 2026-02-25T08:44:13.826 INFO:tasks.rados.rados.0.trial023.stdout:525: done (0 left) 2026-02-25T08:44:13.826 INFO:tasks.rados.rados.0.trial023.stdout:526: snap_create 2026-02-25T08:44:13.896 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:44:13.965 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:44:13.968 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:44:13.969 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:44:14.526 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~154s 2026-02-25T08:44:14.691 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:14.770 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:14.767+0000 7fc209d74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:14.772 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:14.770+0000 7fc209d74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:14.788 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:44:14.815 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:14.816 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:14.829 INFO:tasks.rados.rados.0.trial023.stdout:526: done (0 left) 2026-02-25T08:44:14.829 INFO:tasks.rados.rados.0.trial023.stdout:527: read oid 19 snap -1 2026-02-25T08:44:14.829 INFO:tasks.rados.rados.0.trial023.stdout:527: expect (ObjNum 238 snap 69 seq_num 238) 2026-02-25T08:44:14.829 INFO:tasks.rados.rados.0.trial023.stdout:528: read oid 27 snap -1 2026-02-25T08:44:14.829 INFO:tasks.rados.rados.0.trial023.stdout:528: expect deleted 2026-02-25T08:44:14.829 INFO:tasks.rados.rados.0.trial023.stdout:529: read oid 39 snap -1 2026-02-25T08:44:14.829 INFO:tasks.rados.rados.0.trial023.stdout:529: expect (ObjNum 175 snap 40 seq_num 175) 2026-02-25T08:44:14.829 INFO:tasks.rados.rados.0.trial023.stdout:530: read oid 18 snap -1 2026-02-25T08:44:14.829 INFO:tasks.rados.rados.0.trial023.stdout:530: expect deleted 2026-02-25T08:44:14.829 INFO:tasks.rados.rados.0.trial023.stdout:531: delete oid 16 current snap is 74 2026-02-25T08:44:14.871 INFO:tasks.rados.rados.0.trial023.stdout:529: expect (ObjNum 175 snap 40 seq_num 175) 2026-02-25T08:44:14.872 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:14.906 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:14.941 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:14.976 INFO:tasks.rados.rados.0.trial023.stdout:528: done (4 left) 2026-02-25T08:44:14.976 INFO:tasks.rados.rados.0.trial023.stdout:529: done (3 left) 2026-02-25T08:44:14.976 INFO:tasks.rados.rados.0.trial023.stdout:531: done (2 left) 2026-02-25T08:44:14.976 INFO:tasks.rados.rados.0.trial023.stdout:532: snap_remove snap 58 2026-02-25T08:44:15.832 INFO:tasks.rados.rados.0.trial023.stdout:532: done (2 left) 2026-02-25T08:44:15.832 INFO:tasks.rados.rados.0.trial023.stdout:533: read oid 44 snap 71 2026-02-25T08:44:15.832 INFO:tasks.rados.rados.0.trial023.stdout:533: expect (ObjNum 208 snap 60 seq_num 208) 2026-02-25T08:44:15.833 INFO:tasks.rados.rados.0.trial023.stdout:534: snap_remove snap 36 2026-02-25T08:44:16.817 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:16.833 INFO:tasks.rados.rados.0.trial023.stdout:534: done (3 left) 2026-02-25T08:44:16.833 INFO:tasks.rados.rados.0.trial023.stdout:535: read oid 22 snap 41 2026-02-25T08:44:16.833 INFO:tasks.rados.rados.0.trial023.stdout:535: expect (ObjNum 160 snap 34 seq_num 160) 2026-02-25T08:44:16.833 INFO:tasks.rados.rados.0.trial023.stdout:536: read oid 10 snap 64 2026-02-25T08:44:16.834 INFO:tasks.rados.rados.0.trial023.stdout:536: expect (ObjNum 204 snap 58 seq_num 204) 2026-02-25T08:44:16.834 INFO:tasks.rados.rados.0.trial023.stdout:537: write (excl) oid 4 current snap is 74 2026-02-25T08:44:16.834 INFO:tasks.rados.rados.0.trial023.stdout:537: seq_num 249 ranges {584943=460297,1587781=597162} 2026-02-25T08:44:16.837 INFO:tasks.rados.rados.0.trial023.stdout:537: writing trial02342524-4 from 584943 to 1045240 tid 1 2026-02-25T08:44:16.842 INFO:tasks.rados.rados.0.trial023.stdout:537: writing trial02342524-4 from 1587781 to 2184943 tid 2 2026-02-25T08:44:16.842 INFO:tasks.rados.rados.0.trial023.stdout:538: read oid 41 snap -1 2026-02-25T08:44:16.842 INFO:tasks.rados.rados.0.trial023.stdout:538: expect (ObjNum 194 snap 51 seq_num 194) 2026-02-25T08:44:16.842 INFO:tasks.rados.rados.0.trial023.stdout:539: delete oid 23 current snap is 74 2026-02-25T08:44:16.842 INFO:tasks.rados.rados.0.trial023.stdout:527: expect (ObjNum 238 snap 69 seq_num 238) 2026-02-25T08:44:16.843 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 152097 2026-02-25T08:44:16.898 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:16.895+0000 7f3ba8251640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:16.901 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:16.899+0000 7f3ba8251640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:16.920 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:44:16.921 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:16.962 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:16.962 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:16.963 INFO:tasks.rados.rados.0.trial023.stdout:527: done (7 left) 2026-02-25T08:44:16.963 INFO:tasks.rados.rados.0.trial023.stdout:530: done (6 left) 2026-02-25T08:44:16.964 INFO:tasks.rados.rados.0.trial023.stdout:539: done (5 left) 2026-02-25T08:44:16.964 INFO:tasks.rados.rados.0.trial023.stdout:540: read oid 18 snap 73 2026-02-25T08:44:16.964 INFO:tasks.rados.rados.0.trial023.stdout:540: expect deleted 2026-02-25T08:44:16.964 INFO:tasks.rados.rados.0.trial023.stdout:541: read oid 40 snap 64 2026-02-25T08:44:16.964 INFO:tasks.rados.rados.0.trial023.stdout:541: expect (ObjNum 103 snap 12 seq_num 103) 2026-02-25T08:44:16.964 INFO:tasks.rados.rados.0.trial023.stdout:542: delete oid 29 current snap is 74 2026-02-25T08:44:16.964 INFO:tasks.rados.rados.0.trial023.stdout:533: expect (ObjNum 208 snap 60 seq_num 208) 2026-02-25T08:44:16.991 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:17.021 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:17.051 INFO:tasks.rados.rados.0.trial023.stdout:535: expect (ObjNum 160 snap 34 seq_num 160) 2026-02-25T08:44:17.095 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 286935 2026-02-25T08:44:17.175 INFO:tasks.rados.rados.0.trial023.stdout:538: expect (ObjNum 194 snap 51 seq_num 194) 2026-02-25T08:44:17.175 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:17.252 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:17.289 INFO:tasks.rados.rados.0.trial023.stdout:536: expect (ObjNum 204 snap 58 seq_num 204) 2026-02-25T08:44:17.346 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 823753 2026-02-25T08:44:17.369 INFO:tasks.rados.rados.0.trial023.stdout:537: finishing write tid 1 to trial02342524-4 2026-02-25T08:44:17.369 INFO:tasks.rados.rados.0.trial023.stdout:537: finishing write tid 2 to trial02342524-4 2026-02-25T08:44:17.369 INFO:tasks.rados.rados.0.trial023.stdout:537: finishing write tid 3 to trial02342524-4 2026-02-25T08:44:17.369 INFO:tasks.rados.rados.0.trial023.stdout:533: done (7 left) 2026-02-25T08:44:17.369 INFO:tasks.rados.rados.0.trial023.stdout:535: done (6 left) 2026-02-25T08:44:17.369 INFO:tasks.rados.rados.0.trial023.stdout:536: done (5 left) 2026-02-25T08:44:17.370 INFO:tasks.rados.rados.0.trial023.stdout:538: done (4 left) 2026-02-25T08:44:17.370 INFO:tasks.rados.rados.0.trial023.stdout:542: done (3 left) 2026-02-25T08:44:17.370 INFO:tasks.rados.rados.0.trial023.stdout:543: snap_remove snap 68 2026-02-25T08:44:17.839 INFO:tasks.rados.rados.0.trial023.stdout:543: done (3 left) 2026-02-25T08:44:17.839 INFO:tasks.rados.rados.0.trial023.stdout:544: write oid 34 current snap is 74 2026-02-25T08:44:17.839 INFO:tasks.rados.rados.0.trial023.stdout:537: finishing write tid 4 to trial02342524-4 2026-02-25T08:44:17.839 INFO:tasks.rados.rados.0.trial023.stdout:537: oid 4 updating version 0 to 129 2026-02-25T08:44:17.839 INFO:tasks.rados.rados.0.trial023.stdout:537: oid 4 updating version 129 to 131 2026-02-25T08:44:17.839 INFO:tasks.rados.rados.0.trial023.stdout:537: oid 4 version 131 is already newer than 131 2026-02-25T08:44:17.839 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 4 v 131 (ObjNum 249 snap 74 seq_num 249) dirty exists 2026-02-25T08:44:17.840 INFO:tasks.rados.rados.0.trial023.stdout:537: left oid 4 (ObjNum 249 snap 74 seq_num 249) 2026-02-25T08:44:17.840 INFO:tasks.rados.rados.0.trial023.stdout:544: seq_num 252 ranges {635364=402397,1763227=472137} 2026-02-25T08:44:17.842 INFO:tasks.rados.rados.0.trial023.stdout:544: writing trial02342524-34 from 635364 to 1037761 tid 1 2026-02-25T08:44:17.847 INFO:tasks.rados.rados.0.trial023.stdout:544: writing trial02342524-34 from 1763227 to 2235364 tid 2 2026-02-25T08:44:17.847 INFO:tasks.rados.rados.0.trial023.stdout:537: done (3 left) 2026-02-25T08:44:17.848 INFO:tasks.rados.rados.0.trial023.stdout:545: snap_create 2026-02-25T08:44:17.848 INFO:tasks.rados.rados.0.trial023.stdout:541: expect (ObjNum 103 snap 12 seq_num 103) 2026-02-25T08:44:17.848 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:17.887 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2007027 2026-02-25T08:44:17.920 INFO:tasks.rados.rados.0.trial023.stdout:544: finishing write tid 1 to trial02342524-34 2026-02-25T08:44:17.921 INFO:tasks.rados.rados.0.trial023.stdout:544: finishing write tid 2 to trial02342524-34 2026-02-25T08:44:17.921 INFO:tasks.rados.rados.0.trial023.stdout:544: finishing write tid 3 to trial02342524-34 2026-02-25T08:44:17.921 INFO:tasks.rados.rados.0.trial023.stdout:544: finishing write tid 4 to trial02342524-34 2026-02-25T08:44:17.921 INFO:tasks.rados.rados.0.trial023.stdout:544: oid 34 updating version 0 to 40 2026-02-25T08:44:17.921 INFO:tasks.rados.rados.0.trial023.stdout:544: oid 34 version 40 is already newer than 40 2026-02-25T08:44:17.921 INFO:tasks.rados.rados.0.trial023.stdout:544: oid 34 version 40 is already newer than 40 2026-02-25T08:44:17.921 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 34 v 40 (ObjNum 252 snap 74 seq_num 252) dirty exists 2026-02-25T08:44:17.921 INFO:tasks.rados.rados.0.trial023.stdout:544: left oid 34 (ObjNum 252 snap 74 seq_num 252) 2026-02-25T08:44:18.843 INFO:tasks.rados.rados.0.trial023.stdout:540: done (3 left) 2026-02-25T08:44:18.843 INFO:tasks.rados.rados.0.trial023.stdout:541: done (2 left) 2026-02-25T08:44:18.843 INFO:tasks.rados.rados.0.trial023.stdout:544: done (1 left) 2026-02-25T08:44:18.843 INFO:tasks.rados.rados.0.trial023.stdout:545: done (0 left) 2026-02-25T08:44:18.843 INFO:tasks.rados.rados.0.trial023.stdout:546: read oid 15 snap 62 2026-02-25T08:44:18.843 INFO:tasks.rados.rados.0.trial023.stdout:546: expect (ObjNum 162 snap 34 seq_num 162) 2026-02-25T08:44:18.844 INFO:tasks.rados.rados.0.trial023.stdout:547: read oid 50 snap -1 2026-02-25T08:44:18.844 INFO:tasks.rados.rados.0.trial023.stdout:547: expect deleted 2026-02-25T08:44:18.844 INFO:tasks.rados.rados.0.trial023.stdout:548: snap_remove snap 41 2026-02-25T08:44:18.964 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:18.970 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:44:19.038 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:19.035+0000 7f50c2b83640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:19.041 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:19.039+0000 7f50c2b83640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:19.057 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:44:19.079 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:44:19.081 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:44:19.082 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:44:19.083 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:19.083 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:19.820 INFO:tasks.rados.rados.0.trial023.stdout:548: done (2 left) 2026-02-25T08:44:19.820 INFO:tasks.rados.rados.0.trial023.stdout:549: write (excl) oid 30 current snap is 75 2026-02-25T08:44:19.820 INFO:tasks.rados.rados.0.trial023.stdout:549: seq_num 253 ranges {652171=477646,1821709=430462} 2026-02-25T08:44:19.824 INFO:tasks.rados.rados.0.trial023.stdout:549: writing trial02342524-30 from 652171 to 1129817 tid 1 2026-02-25T08:44:19.828 INFO:tasks.rados.rados.0.trial023.stdout:549: writing trial02342524-30 from 1821709 to 2252171 tid 2 2026-02-25T08:44:19.829 INFO:tasks.rados.rados.0.trial023.stdout:550: write oid 37 current snap is 75 2026-02-25T08:44:19.829 INFO:tasks.rados.rados.0.trial023.stdout:550: seq_num 254 ranges {668978=552895,1763838=505140} 2026-02-25T08:44:19.833 INFO:tasks.rados.rados.0.trial023.stdout:550: writing trial02342524-37 from 668978 to 1221873 tid 1 2026-02-25T08:44:19.838 INFO:tasks.rados.rados.0.trial023.stdout:550: writing trial02342524-37 from 1763838 to 2268978 tid 2 2026-02-25T08:44:19.838 INFO:tasks.rados.rados.0.trial023.stdout:551: delete oid 14 current snap is 75 2026-02-25T08:44:19.838 INFO:tasks.rados.rados.0.trial023.stdout:546: expect (ObjNum 162 snap 34 seq_num 162) 2026-02-25T08:44:19.890 INFO:tasks.rados.rados.0.trial023.stdout:550: finishing write tid 1 to trial02342524-37 2026-02-25T08:44:19.890 INFO:tasks.rados.rados.0.trial023.stdout:550: finishing write tid 2 to trial02342524-37 2026-02-25T08:44:19.890 INFO:tasks.rados.rados.0.trial023.stdout:550: finishing write tid 3 to trial02342524-37 2026-02-25T08:44:19.890 INFO:tasks.rados.rados.0.trial023.stdout:550: finishing write tid 4 to trial02342524-37 2026-02-25T08:44:19.890 INFO:tasks.rados.rados.0.trial023.stdout:550: oid 37 updating version 0 to 121 2026-02-25T08:44:19.890 INFO:tasks.rados.rados.0.trial023.stdout:550: oid 37 version 121 is already newer than 121 2026-02-25T08:44:19.891 INFO:tasks.rados.rados.0.trial023.stdout:550: oid 37 version 121 is already newer than 121 2026-02-25T08:44:19.891 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 37 v 121 (ObjNum 254 snap 75 seq_num 254) dirty exists 2026-02-25T08:44:19.891 INFO:tasks.rados.rados.0.trial023.stdout:550: left oid 37 (ObjNum 254 snap 75 seq_num 254) 2026-02-25T08:44:19.891 INFO:tasks.rados.rados.0.trial023.stdout:549: finishing write tid 1 to trial02342524-30 2026-02-25T08:44:19.891 INFO:tasks.rados.rados.0.trial023.stdout:549: finishing write tid 2 to trial02342524-30 2026-02-25T08:44:19.897 INFO:tasks.rados.rados.0.trial023.stdout:549: finishing write tid 3 to trial02342524-30 2026-02-25T08:44:19.899 INFO:tasks.rados.rados.0.trial023.stdout:549: finishing write tid 4 to trial02342524-30 2026-02-25T08:44:19.899 INFO:tasks.rados.rados.0.trial023.stdout:549: oid 30 updating version 0 to 79 2026-02-25T08:44:19.900 INFO:tasks.rados.rados.0.trial023.stdout:549: oid 30 version 79 is already newer than 79 2026-02-25T08:44:19.900 INFO:tasks.rados.rados.0.trial023.stdout:549: oid 30 version 79 is already newer than 77 2026-02-25T08:44:19.900 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 30 v 79 (ObjNum 253 snap 75 seq_num 253) dirty exists 2026-02-25T08:44:19.900 INFO:tasks.rados.rados.0.trial023.stdout:549: left oid 30 (ObjNum 253 snap 75 seq_num 253) 2026-02-25T08:44:19.920 INFO:tasks.rados.rados.0.trial023.stdout:546: done (4 left) 2026-02-25T08:44:19.920 INFO:tasks.rados.rados.0.trial023.stdout:547: done (3 left) 2026-02-25T08:44:19.920 INFO:tasks.rados.rados.0.trial023.stdout:549: done (2 left) 2026-02-25T08:44:19.920 INFO:tasks.rados.rados.0.trial023.stdout:550: done (1 left) 2026-02-25T08:44:19.920 INFO:tasks.rados.rados.0.trial023.stdout:551: done (0 left) 2026-02-25T08:44:19.920 INFO:tasks.rados.rados.0.trial023.stdout:552: snap_remove snap 73 2026-02-25T08:44:20.030 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~160s 2026-02-25T08:44:20.847 INFO:tasks.rados.rados.0.trial023.stdout:552: done (0 left) 2026-02-25T08:44:20.847 INFO:tasks.rados.rados.0.trial023.stdout:553: read oid 32 snap -1 2026-02-25T08:44:20.848 INFO:tasks.rados.rados.0.trial023.stdout:553: expect (ObjNum 224 snap 67 seq_num 224) 2026-02-25T08:44:20.848 INFO:tasks.rados.rados.0.trial023.stdout:554: snap_create 2026-02-25T08:44:20.886 INFO:tasks.rados.rados.0.trial023.stdout:553: expect (ObjNum 224 snap 67 seq_num 224) 2026-02-25T08:44:20.886 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:20.939 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:21.084 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:21.158 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:21.155+0000 7f09fba46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:21.161 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:21.159+0000 7f09fba46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:21.176 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:44:21.202 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:21.202 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:21.850 INFO:tasks.rados.rados.0.trial023.stdout:553: done (1 left) 2026-02-25T08:44:21.850 INFO:tasks.rados.rados.0.trial023.stdout:554: done (0 left) 2026-02-25T08:44:21.850 INFO:tasks.rados.rados.0.trial023.stdout:555: write (excl) oid 25 current snap is 76 2026-02-25T08:44:21.850 INFO:tasks.rados.rados.0.trial023.stdout:555: seq_num 256 ranges {702592=703393,1880802=421790} 2026-02-25T08:44:21.854 INFO:tasks.rados.rados.0.trial023.stdout:555: writing trial02342524-25 from 702592 to 1405985 tid 1 2026-02-25T08:44:21.858 INFO:tasks.rados.rados.0.trial023.stdout:555: writing trial02342524-25 from 1880802 to 2302592 tid 2 2026-02-25T08:44:21.858 INFO:tasks.rados.rados.0.trial023.stdout:556: write oid 45 current snap is 76 2026-02-25T08:44:21.858 INFO:tasks.rados.rados.0.trial023.stdout:556: seq_num 257 ranges {719399=778642,1939284=380115} 2026-02-25T08:44:21.864 INFO:tasks.rados.rados.0.trial023.stdout:556: writing trial02342524-45 from 719399 to 1498041 tid 1 2026-02-25T08:44:21.867 INFO:tasks.rados.rados.0.trial023.stdout:556: writing trial02342524-45 from 1939284 to 2319399 tid 2 2026-02-25T08:44:21.867 INFO:tasks.rados.rados.0.trial023.stdout:557: snap_remove snap 74 2026-02-25T08:44:22.854 INFO:tasks.rados.rados.0.trial023.stdout:557: done (2 left) 2026-02-25T08:44:22.855 INFO:tasks.rados.rados.0.trial023.stdout:558: delete oid 5 current snap is 76 2026-02-25T08:44:22.855 INFO:tasks.rados.rados.0.trial023.stdout:555: finishing write tid 1 to trial02342524-25 2026-02-25T08:44:22.855 INFO:tasks.rados.rados.0.trial023.stdout:555: finishing write tid 2 to trial02342524-25 2026-02-25T08:44:22.855 INFO:tasks.rados.rados.0.trial023.stdout:555: finishing write tid 3 to trial02342524-25 2026-02-25T08:44:22.855 INFO:tasks.rados.rados.0.trial023.stdout:555: finishing write tid 4 to trial02342524-25 2026-02-25T08:44:22.855 INFO:tasks.rados.rados.0.trial023.stdout:555: oid 25 updating version 0 to 132 2026-02-25T08:44:22.855 INFO:tasks.rados.rados.0.trial023.stdout:555: oid 25 updating version 132 to 134 2026-02-25T08:44:22.855 INFO:tasks.rados.rados.0.trial023.stdout:555: oid 25 version 134 is already newer than 134 2026-02-25T08:44:22.855 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 25 v 134 (ObjNum 256 snap 76 seq_num 256) dirty exists 2026-02-25T08:44:22.855 INFO:tasks.rados.rados.0.trial023.stdout:555: left oid 25 (ObjNum 256 snap 76 seq_num 256) 2026-02-25T08:44:22.855 INFO:tasks.rados.rados.0.trial023.stdout:556: finishing write tid 1 to trial02342524-45 2026-02-25T08:44:22.856 INFO:tasks.rados.rados.0.trial023.stdout:556: finishing write tid 2 to trial02342524-45 2026-02-25T08:44:22.856 INFO:tasks.rados.rados.0.trial023.stdout:556: finishing write tid 3 to trial02342524-45 2026-02-25T08:44:22.856 INFO:tasks.rados.rados.0.trial023.stdout:556: finishing write tid 4 to trial02342524-45 2026-02-25T08:44:22.856 INFO:tasks.rados.rados.0.trial023.stdout:556: oid 45 updating version 0 to 138 2026-02-25T08:44:22.856 INFO:tasks.rados.rados.0.trial023.stdout:556: oid 45 version 138 is already newer than 138 2026-02-25T08:44:22.856 INFO:tasks.rados.rados.0.trial023.stdout:556: oid 45 version 138 is already newer than 138 2026-02-25T08:44:22.856 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 45 v 138 (ObjNum 257 snap 76 seq_num 257) dirty exists 2026-02-25T08:44:22.856 INFO:tasks.rados.rados.0.trial023.stdout:556: left oid 45 (ObjNum 257 snap 76 seq_num 257) 2026-02-25T08:44:22.892 INFO:tasks.rados.rados.0.trial023.stdout:555: done (2 left) 2026-02-25T08:44:22.893 INFO:tasks.rados.rados.0.trial023.stdout:556: done (1 left) 2026-02-25T08:44:22.893 INFO:tasks.rados.rados.0.trial023.stdout:558: done (0 left) 2026-02-25T08:44:22.893 INFO:tasks.rados.rados.0.trial023.stdout:559: write (excl) oid 50 current snap is 76 2026-02-25T08:44:22.893 INFO:tasks.rados.rados.0.trial023.stdout:559: seq_num 259 ranges {753013=645493,2056248=296765} 2026-02-25T08:44:22.897 INFO:tasks.rados.rados.0.trial023.stdout:559: writing trial02342524-50 from 753013 to 1398506 tid 1 2026-02-25T08:44:22.900 INFO:tasks.rados.rados.0.trial023.stdout:559: writing trial02342524-50 from 2056248 to 2353013 tid 2 2026-02-25T08:44:22.900 INFO:tasks.rados.rados.0.trial023.stdout:560: snap_create 2026-02-25T08:44:22.912 INFO:tasks.rados.rados.0.trial023.stdout:559: finishing write tid 1 to trial02342524-50 2026-02-25T08:44:22.925 INFO:tasks.rados.rados.0.trial023.stdout:559: finishing write tid 2 to trial02342524-50 2026-02-25T08:44:22.929 INFO:tasks.rados.rados.0.trial023.stdout:559: finishing write tid 3 to trial02342524-50 2026-02-25T08:44:22.931 INFO:tasks.rados.rados.0.trial023.stdout:559: finishing write tid 4 to trial02342524-50 2026-02-25T08:44:22.931 INFO:tasks.rados.rados.0.trial023.stdout:559: oid 50 updating version 0 to 141 2026-02-25T08:44:22.931 INFO:tasks.rados.rados.0.trial023.stdout:559: oid 50 version 141 is already newer than 139 2026-02-25T08:44:22.931 INFO:tasks.rados.rados.0.trial023.stdout:559: oid 50 version 141 is already newer than 141 2026-02-25T08:44:22.931 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 50 v 141 (ObjNum 259 snap 76 seq_num 259) dirty exists 2026-02-25T08:44:22.931 INFO:tasks.rados.rados.0.trial023.stdout:559: left oid 50 (ObjNum 259 snap 76 seq_num 259) 2026-02-25T08:44:23.204 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:23.282 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:23.279+0000 7f400b50b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:23.284 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:23.282+0000 7f400b50b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:23.299 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:44:23.326 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:23.327 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:23.858 INFO:tasks.rados.rados.0.trial023.stdout:559: done (1 left) 2026-02-25T08:44:23.858 INFO:tasks.rados.rados.0.trial023.stdout:560: done (0 left) 2026-02-25T08:44:23.858 INFO:tasks.rados.rados.0.trial023.stdout:561: read oid 17 snap -1 2026-02-25T08:44:23.858 INFO:tasks.rados.rados.0.trial023.stdout:561: expect (ObjNum 178 snap 42 seq_num 178) 2026-02-25T08:44:23.858 INFO:tasks.rados.rados.0.trial023.stdout:562: write oid 34 current snap is 77 2026-02-25T08:44:23.858 INFO:tasks.rados.rados.0.trial023.stdout:562: seq_num 260 ranges {769820=720742,2114730=255090} 2026-02-25T08:44:23.863 INFO:tasks.rados.rados.0.trial023.stdout:562: writing trial02342524-34 from 769820 to 1490562 tid 1 2026-02-25T08:44:23.866 INFO:tasks.rados.rados.0.trial023.stdout:562: writing trial02342524-34 from 2114730 to 2369820 tid 2 2026-02-25T08:44:23.866 INFO:tasks.rados.rados.0.trial023.stdout:563: write oid 33 current snap is 77 2026-02-25T08:44:23.867 INFO:tasks.rados.rados.0.trial023.stdout:563: seq_num 261 ranges {786627=795991,2173212=213415} 2026-02-25T08:44:23.872 INFO:tasks.rados.rados.0.trial023.stdout:563: writing trial02342524-33 from 786627 to 1582618 tid 1 2026-02-25T08:44:23.876 INFO:tasks.rados.rados.0.trial023.stdout:563: writing trial02342524-33 from 2173212 to 2386627 tid 2 2026-02-25T08:44:23.876 INFO:tasks.rados.rados.0.trial023.stdout:564: read oid 28 snap -1 2026-02-25T08:44:23.876 INFO:tasks.rados.rados.0.trial023.stdout:564: expect (ObjNum 197 snap 55 seq_num 197) 2026-02-25T08:44:23.876 INFO:tasks.rados.rados.0.trial023.stdout:565: write (excl) oid 31 current snap is 77 2026-02-25T08:44:23.876 INFO:tasks.rados.rados.0.trial023.stdout:565: seq_num 262 ranges {403434=471240,1315341=400756} 2026-02-25T08:44:23.879 INFO:tasks.rados.rados.0.trial023.stdout:565: writing trial02342524-31 from 403434 to 874674 tid 1 2026-02-25T08:44:23.882 INFO:tasks.rados.rados.0.trial023.stdout:565: writing trial02342524-31 from 1315341 to 1716097 tid 2 2026-02-25T08:44:23.882 INFO:tasks.rados.rados.0.trial023.stdout:566: delete oid 48 current snap is 77 2026-02-25T08:44:23.898 INFO:tasks.rados.rados.0.trial023.stdout:566: done (5 left) 2026-02-25T08:44:23.899 INFO:tasks.rados.rados.0.trial023.stdout:567: write oid 10 current snap is 77 2026-02-25T08:44:23.899 INFO:tasks.rados.rados.0.trial023.stdout:567: seq_num 264 ranges {437048=621738,1832305=404425} 2026-02-25T08:44:23.903 INFO:tasks.rados.rados.0.trial023.stdout:567: writing trial02342524-10 from 437048 to 1058786 tid 1 2026-02-25T08:44:23.908 INFO:tasks.rados.rados.0.trial023.stdout:567: writing trial02342524-10 from 1832305 to 2236730 tid 2 2026-02-25T08:44:23.908 INFO:tasks.rados.rados.0.trial023.stdout:568: read oid 4 snap 72 2026-02-25T08:44:23.908 INFO:tasks.rados.rados.0.trial023.stdout:568: expect (ObjNum 229 snap 69 seq_num 229) 2026-02-25T08:44:23.908 INFO:tasks.rados.rados.0.trial023.stdout:569: snap_remove snap 76 2026-02-25T08:44:24.083 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:44:24.156 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:44:24.158 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:44:24.159 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:44:24.860 INFO:tasks.rados.rados.0.trial023.stdout:569: done (7 left) 2026-02-25T08:44:24.860 INFO:tasks.rados.rados.0.trial023.stdout:570: delete oid 16 current snap is 77 2026-02-25T08:44:24.860 INFO:tasks.rados.rados.0.trial023.stdout:564: expect (ObjNum 197 snap 55 seq_num 197) 2026-02-25T08:44:24.860 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:24.897 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:24.963 INFO:tasks.rados.rados.0.trial023.stdout:565: finishing write tid 1 to trial02342524-31 2026-02-25T08:44:24.963 INFO:tasks.rados.rados.0.trial023.stdout:561: expect (ObjNum 178 snap 42 seq_num 178) 2026-02-25T08:44:24.963 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:25.046 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1691815 2026-02-25T08:44:25.061 INFO:tasks.rados.rados.0.trial023.stdout:563: finishing write tid 1 to trial02342524-33 2026-02-25T08:44:25.061 INFO:tasks.rados.rados.0.trial023.stdout:562: finishing write tid 1 to trial02342524-34 2026-02-25T08:44:25.061 INFO:tasks.rados.rados.0.trial023.stdout:561: done (7 left) 2026-02-25T08:44:25.061 INFO:tasks.rados.rados.0.trial023.stdout:564: done (6 left) 2026-02-25T08:44:25.061 INFO:tasks.rados.rados.0.trial023.stdout:570: done (5 left) 2026-02-25T08:44:25.061 INFO:tasks.rados.rados.0.trial023.stdout:571: snap_create 2026-02-25T08:44:25.061 INFO:tasks.rados.rados.0.trial023.stdout:563: finishing write tid 2 to trial02342524-33 2026-02-25T08:44:25.061 INFO:tasks.rados.rados.0.trial023.stdout:562: finishing write tid 2 to trial02342524-34 2026-02-25T08:44:25.062 INFO:tasks.rados.rados.0.trial023.stdout:567: finishing write tid 1 to trial02342524-10 2026-02-25T08:44:25.062 INFO:tasks.rados.rados.0.trial023.stdout:562: finishing write tid 3 to trial02342524-34 2026-02-25T08:44:25.062 INFO:tasks.rados.rados.0.trial023.stdout:562: finishing write tid 4 to trial02342524-34 2026-02-25T08:44:25.062 INFO:tasks.rados.rados.0.trial023.stdout:562: oid 34 updating version 0 to 44 2026-02-25T08:44:25.062 INFO:tasks.rados.rados.0.trial023.stdout:562: oid 34 version 44 is already newer than 44 2026-02-25T08:44:25.062 INFO:tasks.rados.rados.0.trial023.stdout:562: oid 34 version 44 is already newer than 44 2026-02-25T08:44:25.062 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 34 v 44 (ObjNum 260 snap 77 seq_num 260) dirty exists 2026-02-25T08:44:25.062 INFO:tasks.rados.rados.0.trial023.stdout:562: left oid 34 (ObjNum 260 snap 77 seq_num 260) 2026-02-25T08:44:25.062 INFO:tasks.rados.rados.0.trial023.stdout:568: expect (ObjNum 229 snap 69 seq_num 229) 2026-02-25T08:44:25.089 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1891745 2026-02-25T08:44:25.252 INFO:tasks.rados.rados.0.trial023.stdout:565: finishing write tid 2 to trial02342524-31 2026-02-25T08:44:25.253 INFO:tasks.rados.rados.0.trial023.stdout:567: finishing write tid 2 to trial02342524-10 2026-02-25T08:44:25.253 INFO:tasks.rados.rados.0.trial023.stdout:563: finishing write tid 3 to trial02342524-33 2026-02-25T08:44:25.253 INFO:tasks.rados.rados.0.trial023.stdout:567: finishing write tid 3 to trial02342524-10 2026-02-25T08:44:25.253 INFO:tasks.rados.rados.0.trial023.stdout:567: finishing write tid 4 to trial02342524-10 2026-02-25T08:44:25.253 INFO:tasks.rados.rados.0.trial023.stdout:567: oid 10 updating version 0 to 124 2026-02-25T08:44:25.253 INFO:tasks.rados.rados.0.trial023.stdout:567: oid 10 version 124 is already newer than 124 2026-02-25T08:44:25.253 INFO:tasks.rados.rados.0.trial023.stdout:567: oid 10 version 124 is already newer than 124 2026-02-25T08:44:25.253 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 10 v 124 (ObjNum 264 snap 77 seq_num 264) dirty exists 2026-02-25T08:44:25.253 INFO:tasks.rados.rados.0.trial023.stdout:567: left oid 10 (ObjNum 264 snap 77 seq_num 264) 2026-02-25T08:44:25.253 INFO:tasks.rados.rados.0.trial023.stdout:563: finishing write tid 4 to trial02342524-33 2026-02-25T08:44:25.254 INFO:tasks.rados.rados.0.trial023.stdout:563: oid 33 updating version 0 to 26 2026-02-25T08:44:25.254 INFO:tasks.rados.rados.0.trial023.stdout:563: oid 33 version 26 is already newer than 26 2026-02-25T08:44:25.254 INFO:tasks.rados.rados.0.trial023.stdout:563: oid 33 version 26 is already newer than 26 2026-02-25T08:44:25.254 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 33 v 26 (ObjNum 261 snap 77 seq_num 261) dirty exists 2026-02-25T08:44:25.254 INFO:tasks.rados.rados.0.trial023.stdout:563: left oid 33 (ObjNum 261 snap 77 seq_num 261) 2026-02-25T08:44:25.254 INFO:tasks.rados.rados.0.trial023.stdout:565: finishing write tid 3 to trial02342524-31 2026-02-25T08:44:25.254 INFO:tasks.rados.rados.0.trial023.stdout:565: finishing write tid 4 to trial02342524-31 2026-02-25T08:44:25.254 INFO:tasks.rados.rados.0.trial023.stdout:565: oid 31 updating version 0 to 29 2026-02-25T08:44:25.254 INFO:tasks.rados.rados.0.trial023.stdout:565: oid 31 updating version 29 to 31 2026-02-25T08:44:25.254 INFO:tasks.rados.rados.0.trial023.stdout:565: oid 31 version 31 is already newer than 31 2026-02-25T08:44:25.254 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 31 v 31 (ObjNum 262 snap 77 seq_num 262) dirty exists 2026-02-25T08:44:25.254 INFO:tasks.rados.rados.0.trial023.stdout:565: left oid 31 (ObjNum 262 snap 77 seq_num 262) 2026-02-25T08:44:25.327 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:25.404 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:25.400+0000 7f8d1a81a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:25.406 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:25.404+0000 7f8d1a81a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:25.422 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:44:25.448 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:25.448 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:25.535 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~165s 2026-02-25T08:44:25.863 INFO:tasks.rados.rados.0.trial023.stdout:562: done (5 left) 2026-02-25T08:44:25.863 INFO:tasks.rados.rados.0.trial023.stdout:563: done (4 left) 2026-02-25T08:44:25.863 INFO:tasks.rados.rados.0.trial023.stdout:565: done (3 left) 2026-02-25T08:44:25.863 INFO:tasks.rados.rados.0.trial023.stdout:567: done (2 left) 2026-02-25T08:44:25.863 INFO:tasks.rados.rados.0.trial023.stdout:568: done (1 left) 2026-02-25T08:44:25.864 INFO:tasks.rados.rados.0.trial023.stdout:571: done (0 left) 2026-02-25T08:44:25.864 INFO:tasks.rados.rados.0.trial023.stdout:572: snap_remove snap 70 2026-02-25T08:44:26.867 INFO:tasks.rados.rados.0.trial023.stdout:572: done (0 left) 2026-02-25T08:44:26.867 INFO:tasks.rados.rados.0.trial023.stdout:573: snap_remove snap 64 2026-02-25T08:44:27.449 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:27.526 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:27.523+0000 7f23ac1b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:27.529 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:27.527+0000 7f23ac1b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:27.544 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:44:27.572 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:27.573 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:27.869 INFO:tasks.rados.rados.0.trial023.stdout:573: done (0 left) 2026-02-25T08:44:27.869 INFO:tasks.rados.rados.0.trial023.stdout:574: write oid 26 current snap is 78 2026-02-25T08:44:27.869 INFO:tasks.rados.rados.0.trial023.stdout:574: seq_num 266 ranges {470662=772236,1832916=408094} 2026-02-25T08:44:27.873 INFO:tasks.rados.rados.0.trial023.stdout:574: writing trial02342524-26 from 470662 to 1242898 tid 1 2026-02-25T08:44:27.877 INFO:tasks.rados.rados.0.trial023.stdout:574: writing trial02342524-26 from 1832916 to 2241010 tid 2 2026-02-25T08:44:27.878 INFO:tasks.rados.rados.0.trial023.stdout:575: read oid 29 snap -1 2026-02-25T08:44:27.878 INFO:tasks.rados.rados.0.trial023.stdout:575: expect deleted 2026-02-25T08:44:27.878 INFO:tasks.rados.rados.0.trial023.stdout:576: snap_create 2026-02-25T08:44:27.908 INFO:tasks.rados.rados.0.trial023.stdout:574: finishing write tid 1 to trial02342524-26 2026-02-25T08:44:27.914 INFO:tasks.rados.rados.0.trial023.stdout:574: finishing write tid 2 to trial02342524-26 2026-02-25T08:44:27.919 INFO:tasks.rados.rados.0.trial023.stdout:574: finishing write tid 3 to trial02342524-26 2026-02-25T08:44:27.920 INFO:tasks.rados.rados.0.trial023.stdout:574: finishing write tid 4 to trial02342524-26 2026-02-25T08:44:27.920 INFO:tasks.rados.rados.0.trial023.stdout:574: oid 26 updating version 0 to 127 2026-02-25T08:44:27.921 INFO:tasks.rados.rados.0.trial023.stdout:574: oid 26 version 127 is already newer than 127 2026-02-25T08:44:27.921 INFO:tasks.rados.rados.0.trial023.stdout:574: oid 26 version 127 is already newer than 127 2026-02-25T08:44:27.921 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 26 v 127 (ObjNum 266 snap 78 seq_num 266) dirty exists 2026-02-25T08:44:27.921 INFO:tasks.rados.rados.0.trial023.stdout:574: left oid 26 (ObjNum 266 snap 78 seq_num 266) 2026-02-25T08:44:28.874 INFO:tasks.rados.rados.0.trial023.stdout:574: done (2 left) 2026-02-25T08:44:28.874 INFO:tasks.rados.rados.0.trial023.stdout:575: done (1 left) 2026-02-25T08:44:28.875 INFO:tasks.rados.rados.0.trial023.stdout:576: done (0 left) 2026-02-25T08:44:28.875 INFO:tasks.rados.rados.0.trial023.stdout:577: delete oid 5 current snap is 79 2026-02-25T08:44:28.890 INFO:tasks.rados.rados.0.trial023.stdout:577: done (0 left) 2026-02-25T08:44:28.890 INFO:tasks.rados.rados.0.trial023.stdout:578: write (excl) oid 16 current snap is 79 2026-02-25T08:44:28.890 INFO:tasks.rados.rados.0.trial023.stdout:578: seq_num 268 ranges {504276=639087,1666233=695410} 2026-02-25T08:44:28.894 INFO:tasks.rados.rados.0.trial023.stdout:578: writing trial02342524-16 from 504276 to 1143363 tid 1 2026-02-25T08:44:28.899 INFO:tasks.rados.rados.0.trial023.stdout:578: writing trial02342524-16 from 1666233 to 2361643 tid 2 2026-02-25T08:44:28.899 INFO:tasks.rados.rados.0.trial023.stdout:579: write oid 31 current snap is 79 2026-02-25T08:44:28.899 INFO:tasks.rados.rados.0.trial023.stdout:579: seq_num 269 ranges {521083=714336,1724715=555421} 2026-02-25T08:44:28.903 INFO:tasks.rados.rados.0.trial023.stdout:579: writing trial02342524-31 from 521083 to 1235419 tid 1 2026-02-25T08:44:28.908 INFO:tasks.rados.rados.0.trial023.stdout:579: writing trial02342524-31 from 1724715 to 2280136 tid 2 2026-02-25T08:44:28.908 INFO:tasks.rados.rados.0.trial023.stdout:580: write (excl) oid 30 current snap is 79 2026-02-25T08:44:28.908 INFO:tasks.rados.rados.0.trial023.stdout:580: seq_num 270 ranges {537890=789585,1783197=699079} 2026-02-25T08:44:28.913 INFO:tasks.rados.rados.0.trial023.stdout:580: writing trial02342524-30 from 537890 to 1327475 tid 1 2026-02-25T08:44:28.917 INFO:tasks.rados.rados.0.trial023.stdout:580: writing trial02342524-30 from 1783197 to 2482276 tid 2 2026-02-25T08:44:28.917 INFO:tasks.rados.rados.0.trial023.stdout:581: write (excl) oid 35 current snap is 79 2026-02-25T08:44:28.917 INFO:tasks.rados.rados.0.trial023.stdout:581: seq_num 271 ranges {554697=464834,1725326=559090} 2026-02-25T08:44:28.921 INFO:tasks.rados.rados.0.trial023.stdout:581: writing trial02342524-35 from 554697 to 1019531 tid 1 2026-02-25T08:44:28.926 INFO:tasks.rados.rados.0.trial023.stdout:581: writing trial02342524-35 from 1725326 to 2284416 tid 2 2026-02-25T08:44:28.926 INFO:tasks.rados.rados.0.trial023.stdout:582: write (excl) oid 10 current snap is 79 2026-02-25T08:44:28.926 INFO:tasks.rados.rados.0.trial023.stdout:578: finishing write tid 1 to trial02342524-16 2026-02-25T08:44:28.926 INFO:tasks.rados.rados.0.trial023.stdout:582: seq_num 272 ranges {571504=540083,1783808=702748} 2026-02-25T08:44:28.930 INFO:tasks.rados.rados.0.trial023.stdout:582: writing trial02342524-10 from 571504 to 1111587 tid 1 2026-02-25T08:44:28.936 INFO:tasks.rados.rados.0.trial023.stdout:582: writing trial02342524-10 from 1783808 to 2486556 tid 2 2026-02-25T08:44:28.936 INFO:tasks.rados.rados.0.trial023.stdout:583: write (excl) oid 15 current snap is 79 2026-02-25T08:44:28.936 INFO:tasks.rados.rados.0.trial023.stdout:578: finishing write tid 2 to trial02342524-16 2026-02-25T08:44:28.936 INFO:tasks.rados.rados.0.trial023.stdout:583: seq_num 273 ranges {588311=615332,1842290=562759} 2026-02-25T08:44:28.941 INFO:tasks.rados.rados.0.trial023.stdout:583: writing trial02342524-15 from 588311 to 1203643 tid 1 2026-02-25T08:44:28.946 INFO:tasks.rados.rados.0.trial023.stdout:583: writing trial02342524-15 from 1842290 to 2405049 tid 2 2026-02-25T08:44:28.946 INFO:tasks.rados.rados.0.trial023.stdout:584: write (excl) oid 7 current snap is 79 2026-02-25T08:44:28.946 INFO:tasks.rados.rados.0.trial023.stdout:578: finishing write tid 3 to trial02342524-16 2026-02-25T08:44:28.946 INFO:tasks.rados.rados.0.trial023.stdout:578: finishing write tid 4 to trial02342524-16 2026-02-25T08:44:28.946 INFO:tasks.rados.rados.0.trial023.stdout:578: oid 16 updating version 0 to 99 2026-02-25T08:44:28.946 INFO:tasks.rados.rados.0.trial023.stdout:578: oid 16 version 99 is already newer than 97 2026-02-25T08:44:28.946 INFO:tasks.rados.rados.0.trial023.stdout:578: oid 16 version 99 is already newer than 99 2026-02-25T08:44:28.946 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 16 v 99 (ObjNum 268 snap 79 seq_num 268) dirty exists 2026-02-25T08:44:28.947 INFO:tasks.rados.rados.0.trial023.stdout:578: left oid 16 (ObjNum 268 snap 79 seq_num 268) 2026-02-25T08:44:28.947 INFO:tasks.rados.rados.0.trial023.stdout:584: seq_num 274 ranges {605118=406934,1617125=706417} 2026-02-25T08:44:28.949 INFO:tasks.rados.rados.0.trial023.stdout:584: writing trial02342524-7 from 605118 to 1012052 tid 1 2026-02-25T08:44:28.955 INFO:tasks.rados.rados.0.trial023.stdout:584: writing trial02342524-7 from 1617125 to 2323542 tid 2 2026-02-25T08:44:28.955 INFO:tasks.rados.rados.0.trial023.stdout:578: done (6 left) 2026-02-25T08:44:28.955 INFO:tasks.rados.rados.0.trial023.stdout:585: read oid 11 snap 77 2026-02-25T08:44:28.955 INFO:tasks.rados.rados.0.trial023.stdout:585: expect (ObjNum 189 snap 48 seq_num 189) 2026-02-25T08:44:28.955 INFO:tasks.rados.rados.0.trial023.stdout:586: read oid 12 snap -1 2026-02-25T08:44:28.955 INFO:tasks.rados.rados.0.trial023.stdout:586: expect deleted 2026-02-25T08:44:28.955 INFO:tasks.rados.rados.0.trial023.stdout:587: write (excl) oid 33 current snap is 79 2026-02-25T08:44:28.955 INFO:tasks.rados.rados.0.trial023.stdout:587: seq_num 275 ranges {621925=482183,1559254=566428} 2026-02-25T08:44:28.958 INFO:tasks.rados.rados.0.trial023.stdout:587: writing trial02342524-33 from 621925 to 1104108 tid 1 2026-02-25T08:44:28.963 INFO:tasks.rados.rados.0.trial023.stdout:587: writing trial02342524-33 from 1559254 to 2125682 tid 2 2026-02-25T08:44:28.963 INFO:tasks.rados.rados.0.trial023.stdout:588: write (excl) oid 5 current snap is 79 2026-02-25T08:44:28.963 INFO:tasks.rados.rados.0.trial023.stdout:588: seq_num 276 ranges {638732=557432,1617736=710086} 2026-02-25T08:44:28.967 INFO:tasks.rados.rados.0.trial023.stdout:588: writing trial02342524-5 from 638732 to 1196164 tid 1 2026-02-25T08:44:28.971 INFO:tasks.rados.rados.0.trial023.stdout:588: writing trial02342524-5 from 1617736 to 2327822 tid 2 2026-02-25T08:44:28.971 INFO:tasks.rados.rados.0.trial023.stdout:589: write oid 14 current snap is 79 2026-02-25T08:44:28.972 INFO:tasks.rados.rados.0.trial023.stdout:589: seq_num 277 ranges {655539=632681,2076218=570097} 2026-02-25T08:44:28.976 INFO:tasks.rados.rados.0.trial023.stdout:589: writing trial02342524-14 from 655539 to 1288220 tid 1 2026-02-25T08:44:28.982 INFO:tasks.rados.rados.0.trial023.stdout:589: writing trial02342524-14 from 2076218 to 2646315 tid 2 2026-02-25T08:44:28.982 INFO:tasks.rados.rados.0.trial023.stdout:590: snap_create 2026-02-25T08:44:28.982 INFO:tasks.rados.rados.0.trial023.stdout:581: finishing write tid 1 to trial02342524-35 2026-02-25T08:44:28.982 INFO:tasks.rados.rados.0.trial023.stdout:582: finishing write tid 1 to trial02342524-10 2026-02-25T08:44:28.982 INFO:tasks.rados.rados.0.trial023.stdout:579: finishing write tid 1 to trial02342524-31 2026-02-25T08:44:28.982 INFO:tasks.rados.rados.0.trial023.stdout:585: expect (ObjNum 189 snap 48 seq_num 189) 2026-02-25T08:44:28.990 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2466982 2026-02-25T08:44:29.004 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2049370 2026-02-25T08:44:29.041 INFO:tasks.rados.rados.0.trial023.stdout:581: finishing write tid 2 to trial02342524-35 2026-02-25T08:44:29.041 INFO:tasks.rados.rados.0.trial023.stdout:582: finishing write tid 2 to trial02342524-10 2026-02-25T08:44:29.041 INFO:tasks.rados.rados.0.trial023.stdout:588: finishing write tid 1 to trial02342524-5 2026-02-25T08:44:29.042 INFO:tasks.rados.rados.0.trial023.stdout:582: finishing write tid 3 to trial02342524-10 2026-02-25T08:44:29.042 INFO:tasks.rados.rados.0.trial023.stdout:581: finishing write tid 3 to trial02342524-35 2026-02-25T08:44:29.042 INFO:tasks.rados.rados.0.trial023.stdout:582: finishing write tid 4 to trial02342524-10 2026-02-25T08:44:29.042 INFO:tasks.rados.rados.0.trial023.stdout:582: oid 10 updating version 0 to 129 2026-02-25T08:44:29.042 INFO:tasks.rados.rados.0.trial023.stdout:582: oid 10 updating version 129 to 131 2026-02-25T08:44:29.042 INFO:tasks.rados.rados.0.trial023.stdout:582: oid 10 version 131 is already newer than 131 2026-02-25T08:44:29.042 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 10 v 131 (ObjNum 272 snap 79 seq_num 272) dirty exists 2026-02-25T08:44:29.042 INFO:tasks.rados.rados.0.trial023.stdout:582: left oid 10 (ObjNum 272 snap 79 seq_num 272) 2026-02-25T08:44:29.043 INFO:tasks.rados.rados.0.trial023.stdout:581: finishing write tid 4 to trial02342524-35 2026-02-25T08:44:29.043 INFO:tasks.rados.rados.0.trial023.stdout:581: oid 35 updating version 0 to 172 2026-02-25T08:44:29.043 INFO:tasks.rados.rados.0.trial023.stdout:581: oid 35 version 172 is already newer than 172 2026-02-25T08:44:29.043 INFO:tasks.rados.rados.0.trial023.stdout:581: oid 35 version 172 is already newer than 170 2026-02-25T08:44:29.043 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 35 v 172 (ObjNum 271 snap 79 seq_num 271) dirty exists 2026-02-25T08:44:29.043 INFO:tasks.rados.rados.0.trial023.stdout:581: left oid 35 (ObjNum 271 snap 79 seq_num 271) 2026-02-25T08:44:29.043 INFO:tasks.rados.rados.0.trial023.stdout:579: finishing write tid 2 to trial02342524-31 2026-02-25T08:44:29.043 INFO:tasks.rados.rados.0.trial023.stdout:587: finishing write tid 1 to trial02342524-33 2026-02-25T08:44:29.043 INFO:tasks.rados.rados.0.trial023.stdout:588: finishing write tid 2 to trial02342524-5 2026-02-25T08:44:29.043 INFO:tasks.rados.rados.0.trial023.stdout:579: finishing write tid 3 to trial02342524-31 2026-02-25T08:44:29.043 INFO:tasks.rados.rados.0.trial023.stdout:588: finishing write tid 3 to trial02342524-5 2026-02-25T08:44:29.043 INFO:tasks.rados.rados.0.trial023.stdout:579: finishing write tid 4 to trial02342524-31 2026-02-25T08:44:29.044 INFO:tasks.rados.rados.0.trial023.stdout:579: oid 31 updating version 0 to 35 2026-02-25T08:44:29.044 INFO:tasks.rados.rados.0.trial023.stdout:579: oid 31 version 35 is already newer than 35 2026-02-25T08:44:29.044 INFO:tasks.rados.rados.0.trial023.stdout:579: oid 31 version 35 is already newer than 35 2026-02-25T08:44:29.044 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 31 v 35 (ObjNum 269 snap 79 seq_num 269) dirty exists 2026-02-25T08:44:29.044 INFO:tasks.rados.rados.0.trial023.stdout:579: left oid 31 (ObjNum 269 snap 79 seq_num 269) 2026-02-25T08:44:29.044 INFO:tasks.rados.rados.0.trial023.stdout:588: finishing write tid 4 to trial02342524-5 2026-02-25T08:44:29.044 INFO:tasks.rados.rados.0.trial023.stdout:588: oid 5 updating version 0 to 21 2026-02-25T08:44:29.044 INFO:tasks.rados.rados.0.trial023.stdout:588: oid 5 version 21 is already newer than 19 2026-02-25T08:44:29.044 INFO:tasks.rados.rados.0.trial023.stdout:588: oid 5 version 21 is already newer than 21 2026-02-25T08:44:29.044 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 5 v 21 (ObjNum 276 snap 79 seq_num 276) dirty exists 2026-02-25T08:44:29.044 INFO:tasks.rados.rados.0.trial023.stdout:588: left oid 5 (ObjNum 276 snap 79 seq_num 276) 2026-02-25T08:44:29.044 INFO:tasks.rados.rados.0.trial023.stdout:587: finishing write tid 2 to trial02342524-33 2026-02-25T08:44:29.045 INFO:tasks.rados.rados.0.trial023.stdout:587: finishing write tid 3 to trial02342524-33 2026-02-25T08:44:29.046 INFO:tasks.rados.rados.0.trial023.stdout:587: finishing write tid 4 to trial02342524-33 2026-02-25T08:44:29.046 INFO:tasks.rados.rados.0.trial023.stdout:587: oid 33 updating version 0 to 28 2026-02-25T08:44:29.046 INFO:tasks.rados.rados.0.trial023.stdout:587: oid 33 updating version 28 to 30 2026-02-25T08:44:29.046 INFO:tasks.rados.rados.0.trial023.stdout:587: oid 33 version 30 is already newer than 30 2026-02-25T08:44:29.046 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 33 v 30 (ObjNum 275 snap 79 seq_num 275) dirty exists 2026-02-25T08:44:29.046 INFO:tasks.rados.rados.0.trial023.stdout:587: left oid 33 (ObjNum 275 snap 79 seq_num 275) 2026-02-25T08:44:29.046 INFO:tasks.rados.rados.0.trial023.stdout:589: finishing write tid 1 to trial02342524-14 2026-02-25T08:44:29.053 INFO:tasks.rados.rados.0.trial023.stdout:583: finishing write tid 1 to trial02342524-15 2026-02-25T08:44:29.054 INFO:tasks.rados.rados.0.trial023.stdout:589: finishing write tid 2 to trial02342524-14 2026-02-25T08:44:29.065 INFO:tasks.rados.rados.0.trial023.stdout:589: finishing write tid 3 to trial02342524-14 2026-02-25T08:44:29.067 INFO:tasks.rados.rados.0.trial023.stdout:589: finishing write tid 4 to trial02342524-14 2026-02-25T08:44:29.067 INFO:tasks.rados.rados.0.trial023.stdout:589: oid 14 updating version 0 to 42 2026-02-25T08:44:29.067 INFO:tasks.rados.rados.0.trial023.stdout:589: oid 14 version 42 is already newer than 42 2026-02-25T08:44:29.068 INFO:tasks.rados.rados.0.trial023.stdout:589: oid 14 version 42 is already newer than 42 2026-02-25T08:44:29.068 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 14 v 42 (ObjNum 277 snap 79 seq_num 277) dirty exists 2026-02-25T08:44:29.068 INFO:tasks.rados.rados.0.trial023.stdout:589: left oid 14 (ObjNum 277 snap 79 seq_num 277) 2026-02-25T08:44:29.069 INFO:tasks.rados.rados.0.trial023.stdout:583: finishing write tid 2 to trial02342524-15 2026-02-25T08:44:29.074 INFO:tasks.rados.rados.0.trial023.stdout:583: finishing write tid 3 to trial02342524-15 2026-02-25T08:44:29.076 INFO:tasks.rados.rados.0.trial023.stdout:583: finishing write tid 4 to trial02342524-15 2026-02-25T08:44:29.076 INFO:tasks.rados.rados.0.trial023.stdout:583: oid 15 updating version 0 to 102 2026-02-25T08:44:29.076 INFO:tasks.rados.rados.0.trial023.stdout:583: oid 15 updating version 102 to 104 2026-02-25T08:44:29.076 INFO:tasks.rados.rados.0.trial023.stdout:583: oid 15 version 104 is already newer than 104 2026-02-25T08:44:29.076 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 15 v 104 (ObjNum 273 snap 79 seq_num 273) dirty exists 2026-02-25T08:44:29.076 INFO:tasks.rados.rados.0.trial023.stdout:583: left oid 15 (ObjNum 273 snap 79 seq_num 273) 2026-02-25T08:44:29.116 INFO:tasks.rados.rados.0.trial023.stdout:580: finishing write tid 1 to trial02342524-30 2026-02-25T08:44:29.131 INFO:tasks.rados.rados.0.trial023.stdout:580: finishing write tid 2 to trial02342524-30 2026-02-25T08:44:29.136 INFO:tasks.rados.rados.0.trial023.stdout:580: finishing write tid 3 to trial02342524-30 2026-02-25T08:44:29.139 INFO:tasks.rados.rados.0.trial023.stdout:580: finishing write tid 4 to trial02342524-30 2026-02-25T08:44:29.140 INFO:tasks.rados.rados.0.trial023.stdout:580: oid 30 updating version 0 to 81 2026-02-25T08:44:29.140 INFO:tasks.rados.rados.0.trial023.stdout:580: oid 30 updating version 81 to 83 2026-02-25T08:44:29.140 INFO:tasks.rados.rados.0.trial023.stdout:580: oid 30 version 83 is already newer than 83 2026-02-25T08:44:29.140 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 30 v 83 (ObjNum 270 snap 79 seq_num 270) dirty exists 2026-02-25T08:44:29.140 INFO:tasks.rados.rados.0.trial023.stdout:580: left oid 30 (ObjNum 270 snap 79 seq_num 270) 2026-02-25T08:44:29.146 INFO:tasks.rados.rados.0.trial023.stdout:584: finishing write tid 1 to trial02342524-7 2026-02-25T08:44:29.159 INFO:tasks.rados.rados.0.trial023.stdout:584: finishing write tid 2 to trial02342524-7 2026-02-25T08:44:29.160 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:44:29.163 INFO:tasks.rados.rados.0.trial023.stdout:584: finishing write tid 3 to trial02342524-7 2026-02-25T08:44:29.164 INFO:tasks.rados.rados.0.trial023.stdout:584: finishing write tid 4 to trial02342524-7 2026-02-25T08:44:29.164 INFO:tasks.rados.rados.0.trial023.stdout:584: oid 7 updating version 0 to 86 2026-02-25T08:44:29.164 INFO:tasks.rados.rados.0.trial023.stdout:584: oid 7 version 86 is already newer than 84 2026-02-25T08:44:29.164 INFO:tasks.rados.rados.0.trial023.stdout:584: oid 7 version 86 is already newer than 86 2026-02-25T08:44:29.164 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 7 v 86 (ObjNum 274 snap 79 seq_num 274) dirty exists 2026-02-25T08:44:29.164 INFO:tasks.rados.rados.0.trial023.stdout:584: left oid 7 (ObjNum 274 snap 79 seq_num 274) 2026-02-25T08:44:29.228 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:44:29.231 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:44:29.231 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:44:29.573 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:29.649 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:29.646+0000 7fa91becb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:29.652 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:29.650+0000 7fa91becb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:29.667 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:44:29.695 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:29.696 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:29.821 INFO:tasks.rados.rados.0.trial023.stdout:579: done (11 left) 2026-02-25T08:44:29.821 INFO:tasks.rados.rados.0.trial023.stdout:580: done (10 left) 2026-02-25T08:44:29.822 INFO:tasks.rados.rados.0.trial023.stdout:581: done (9 left) 2026-02-25T08:44:29.822 INFO:tasks.rados.rados.0.trial023.stdout:582: done (8 left) 2026-02-25T08:44:29.822 INFO:tasks.rados.rados.0.trial023.stdout:583: done (7 left) 2026-02-25T08:44:29.822 INFO:tasks.rados.rados.0.trial023.stdout:584: done (6 left) 2026-02-25T08:44:29.822 INFO:tasks.rados.rados.0.trial023.stdout:585: done (5 left) 2026-02-25T08:44:29.822 INFO:tasks.rados.rados.0.trial023.stdout:586: done (4 left) 2026-02-25T08:44:29.822 INFO:tasks.rados.rados.0.trial023.stdout:587: done (3 left) 2026-02-25T08:44:29.822 INFO:tasks.rados.rados.0.trial023.stdout:588: done (2 left) 2026-02-25T08:44:29.822 INFO:tasks.rados.rados.0.trial023.stdout:589: done (1 left) 2026-02-25T08:44:29.822 INFO:tasks.rados.rados.0.trial023.stdout:590: done (0 left) 2026-02-25T08:44:29.822 INFO:tasks.rados.rados.0.trial023.stdout:591: delete oid 31 current snap is 80 2026-02-25T08:44:29.864 INFO:tasks.rados.rados.0.trial023.stdout:591: done (0 left) 2026-02-25T08:44:29.865 INFO:tasks.rados.rados.0.trial023.stdout:592: snap_create 2026-02-25T08:44:30.822 INFO:tasks.rados.rados.0.trial023.stdout:592: done (0 left) 2026-02-25T08:44:30.822 INFO:tasks.rados.rados.0.trial023.stdout:593: write oid 17 current snap is 81 2026-02-25T08:44:30.822 INFO:tasks.rados.rados.0.trial023.stdout:593: seq_num 279 ranges {689153=783179,2076829=457413} 2026-02-25T08:44:30.827 INFO:tasks.rados.rados.0.trial023.stdout:593: writing trial02342524-17 from 689153 to 1472332 tid 1 2026-02-25T08:44:30.832 INFO:tasks.rados.rados.0.trial023.stdout:593: writing trial02342524-17 from 2076829 to 2534242 tid 2 2026-02-25T08:44:30.832 INFO:tasks.rados.rados.0.trial023.stdout:594: write oid 21 current snap is 81 2026-02-25T08:44:30.832 INFO:tasks.rados.rados.0.trial023.stdout:594: seq_num 280 ranges {705960=458428,1735311=717424} 2026-02-25T08:44:30.836 INFO:tasks.rados.rados.0.trial023.stdout:594: writing trial02342524-21 from 705960 to 1164388 tid 1 2026-02-25T08:44:30.841 INFO:tasks.rados.rados.0.trial023.stdout:594: writing trial02342524-21 from 1735311 to 2452735 tid 2 2026-02-25T08:44:30.841 INFO:tasks.rados.rados.0.trial023.stdout:595: write (excl) oid 16 current snap is 81 2026-02-25T08:44:30.841 INFO:tasks.rados.rados.0.trial023.stdout:595: seq_num 281 ranges {722767=533677,1793793=461082} 2026-02-25T08:44:30.846 INFO:tasks.rados.rados.0.trial023.stdout:595: writing trial02342524-16 from 722767 to 1256444 tid 1 2026-02-25T08:44:30.850 INFO:tasks.rados.rados.0.trial023.stdout:595: writing trial02342524-16 from 1793793 to 2254875 tid 2 2026-02-25T08:44:30.850 INFO:tasks.rados.rados.0.trial023.stdout:596: write (excl) oid 25 current snap is 81 2026-02-25T08:44:30.850 INFO:tasks.rados.rados.0.trial023.stdout:596: seq_num 282 ranges {739574=725279,1968628=721093} 2026-02-25T08:44:30.855 INFO:tasks.rados.rados.0.trial023.stdout:596: writing trial02342524-25 from 739574 to 1464853 tid 1 2026-02-25T08:44:30.860 INFO:tasks.rados.rados.0.trial023.stdout:596: writing trial02342524-25 from 1968628 to 2689721 tid 2 2026-02-25T08:44:30.860 INFO:tasks.rados.rados.0.trial023.stdout:597: write oid 47 current snap is 81 2026-02-25T08:44:30.860 INFO:tasks.rados.rados.0.trial023.stdout:597: seq_num 283 ranges {756381=400528,1910757=464751} 2026-02-25T08:44:30.864 INFO:tasks.rados.rados.0.trial023.stdout:597: writing trial02342524-47 from 756381 to 1156909 tid 1 2026-02-25T08:44:30.869 INFO:tasks.rados.rados.0.trial023.stdout:597: writing trial02342524-47 from 1910757 to 2375508 tid 2 2026-02-25T08:44:30.869 INFO:tasks.rados.rados.0.trial023.stdout:598: delete oid 22 current snap is 81 2026-02-25T08:44:30.869 INFO:tasks.rados.rados.0.trial023.stdout:594: finishing write tid 1 to trial02342524-21 2026-02-25T08:44:30.869 INFO:tasks.rados.rados.0.trial023.stdout:593: finishing write tid 1 to trial02342524-17 2026-02-25T08:44:30.878 INFO:tasks.rados.rados.0.trial023.stdout:594: finishing write tid 2 to trial02342524-21 2026-02-25T08:44:30.878 INFO:tasks.rados.rados.0.trial023.stdout:593: finishing write tid 2 to trial02342524-17 2026-02-25T08:44:30.882 INFO:tasks.rados.rados.0.trial023.stdout:594: finishing write tid 3 to trial02342524-21 2026-02-25T08:44:30.886 INFO:tasks.rados.rados.0.trial023.stdout:594: finishing write tid 4 to trial02342524-21 2026-02-25T08:44:30.886 INFO:tasks.rados.rados.0.trial023.stdout:594: oid 21 updating version 0 to 103 2026-02-25T08:44:30.886 INFO:tasks.rados.rados.0.trial023.stdout:594: oid 21 version 103 is already newer than 103 2026-02-25T08:44:30.886 INFO:tasks.rados.rados.0.trial023.stdout:594: oid 21 version 103 is already newer than 103 2026-02-25T08:44:30.886 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 21 v 103 (ObjNum 280 snap 81 seq_num 280) dirty exists 2026-02-25T08:44:30.886 INFO:tasks.rados.rados.0.trial023.stdout:594: left oid 21 (ObjNum 280 snap 81 seq_num 280) 2026-02-25T08:44:30.887 INFO:tasks.rados.rados.0.trial023.stdout:593: finishing write tid 3 to trial02342524-17 2026-02-25T08:44:30.888 INFO:tasks.rados.rados.0.trial023.stdout:596: finishing write tid 1 to trial02342524-25 2026-02-25T08:44:30.890 INFO:tasks.rados.rados.0.trial023.stdout:593: finishing write tid 4 to trial02342524-17 2026-02-25T08:44:30.890 INFO:tasks.rados.rados.0.trial023.stdout:593: oid 17 updating version 0 to 16 2026-02-25T08:44:30.890 INFO:tasks.rados.rados.0.trial023.stdout:593: oid 17 version 16 is already newer than 16 2026-02-25T08:44:30.890 INFO:tasks.rados.rados.0.trial023.stdout:593: oid 17 version 16 is already newer than 16 2026-02-25T08:44:30.890 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 17 v 16 (ObjNum 279 snap 81 seq_num 279) dirty exists 2026-02-25T08:44:30.890 INFO:tasks.rados.rados.0.trial023.stdout:593: left oid 17 (ObjNum 279 snap 81 seq_num 279) 2026-02-25T08:44:30.893 INFO:tasks.rados.rados.0.trial023.stdout:595: finishing write tid 1 to trial02342524-16 2026-02-25T08:44:30.902 INFO:tasks.rados.rados.0.trial023.stdout:597: finishing write tid 1 to trial02342524-47 2026-02-25T08:44:30.903 INFO:tasks.rados.rados.0.trial023.stdout:596: finishing write tid 2 to trial02342524-25 2026-02-25T08:44:30.909 INFO:tasks.rados.rados.0.trial023.stdout:593: done (5 left) 2026-02-25T08:44:30.909 INFO:tasks.rados.rados.0.trial023.stdout:594: done (4 left) 2026-02-25T08:44:30.909 INFO:tasks.rados.rados.0.trial023.stdout:598: done (3 left) 2026-02-25T08:44:30.909 INFO:tasks.rados.rados.0.trial023.stdout:599: delete oid 3 current snap is 81 2026-02-25T08:44:30.909 INFO:tasks.rados.rados.0.trial023.stdout:596: finishing write tid 3 to trial02342524-25 2026-02-25T08:44:30.910 INFO:tasks.rados.rados.0.trial023.stdout:596: finishing write tid 4 to trial02342524-25 2026-02-25T08:44:30.910 INFO:tasks.rados.rados.0.trial023.stdout:596: oid 25 updating version 0 to 145 2026-02-25T08:44:30.910 INFO:tasks.rados.rados.0.trial023.stdout:596: oid 25 version 145 is already newer than 143 2026-02-25T08:44:30.910 INFO:tasks.rados.rados.0.trial023.stdout:596: oid 25 version 145 is already newer than 145 2026-02-25T08:44:30.911 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 25 v 145 (ObjNum 282 snap 81 seq_num 282) dirty exists 2026-02-25T08:44:30.911 INFO:tasks.rados.rados.0.trial023.stdout:596: left oid 25 (ObjNum 282 snap 81 seq_num 282) 2026-02-25T08:44:30.911 INFO:tasks.rados.rados.0.trial023.stdout:597: finishing write tid 2 to trial02342524-47 2026-02-25T08:44:30.914 INFO:tasks.rados.rados.0.trial023.stdout:595: finishing write tid 2 to trial02342524-16 2026-02-25T08:44:30.918 INFO:tasks.rados.rados.0.trial023.stdout:597: finishing write tid 3 to trial02342524-47 2026-02-25T08:44:30.919 INFO:tasks.rados.rados.0.trial023.stdout:595: finishing write tid 3 to trial02342524-16 2026-02-25T08:44:30.921 INFO:tasks.rados.rados.0.trial023.stdout:597: finishing write tid 4 to trial02342524-47 2026-02-25T08:44:30.921 INFO:tasks.rados.rados.0.trial023.stdout:597: oid 47 updating version 0 to 41 2026-02-25T08:44:30.921 INFO:tasks.rados.rados.0.trial023.stdout:597: oid 47 version 41 is already newer than 41 2026-02-25T08:44:30.921 INFO:tasks.rados.rados.0.trial023.stdout:597: oid 47 version 41 is already newer than 41 2026-02-25T08:44:30.921 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 47 v 41 (ObjNum 283 snap 81 seq_num 283) dirty exists 2026-02-25T08:44:30.921 INFO:tasks.rados.rados.0.trial023.stdout:597: left oid 47 (ObjNum 283 snap 81 seq_num 283) 2026-02-25T08:44:30.921 INFO:tasks.rados.rados.0.trial023.stdout:595: finishing write tid 4 to trial02342524-16 2026-02-25T08:44:30.921 INFO:tasks.rados.rados.0.trial023.stdout:595: oid 16 updating version 0 to 107 2026-02-25T08:44:30.922 INFO:tasks.rados.rados.0.trial023.stdout:595: oid 16 version 107 is already newer than 105 2026-02-25T08:44:30.922 INFO:tasks.rados.rados.0.trial023.stdout:595: oid 16 version 107 is already newer than 107 2026-02-25T08:44:30.922 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 16 v 107 (ObjNum 281 snap 81 seq_num 281) dirty exists 2026-02-25T08:44:30.922 INFO:tasks.rados.rados.0.trial023.stdout:595: left oid 16 (ObjNum 281 snap 81 seq_num 281) 2026-02-25T08:44:30.940 INFO:tasks.rados.rados.0.trial023.stdout:595: done (3 left) 2026-02-25T08:44:30.940 INFO:tasks.rados.rados.0.trial023.stdout:596: done (2 left) 2026-02-25T08:44:30.941 INFO:tasks.rados.rados.0.trial023.stdout:597: done (1 left) 2026-02-25T08:44:30.941 INFO:tasks.rados.rados.0.trial023.stdout:599: done (0 left) 2026-02-25T08:44:30.941 INFO:tasks.rados.rados.0.trial023.stdout:600: delete oid 10 current snap is 81 2026-02-25T08:44:30.964 INFO:tasks.rados.rados.0.trial023.stdout:600: done (0 left) 2026-02-25T08:44:30.965 INFO:tasks.rados.rados.0.trial023.stdout:601: read oid 36 snap 77 2026-02-25T08:44:30.965 INFO:tasks.rados.rados.0.trial023.stdout:601: expect (ObjNum 196 snap 55 seq_num 196) 2026-02-25T08:44:30.965 INFO:tasks.rados.rados.0.trial023.stdout:602: read oid 5 snap -1 2026-02-25T08:44:30.965 INFO:tasks.rados.rados.0.trial023.stdout:602: expect (ObjNum 276 snap 79 seq_num 276) 2026-02-25T08:44:30.965 INFO:tasks.rados.rados.0.trial023.stdout:603: read oid 49 snap -1 2026-02-25T08:44:30.965 INFO:tasks.rados.rados.0.trial023.stdout:603: expect deleted 2026-02-25T08:44:30.965 INFO:tasks.rados.rados.0.trial023.stdout:604: delete oid 30 current snap is 81 2026-02-25T08:44:30.990 INFO:tasks.rados.rados.0.trial023.stdout:602: expect (ObjNum 276 snap 79 seq_num 276) 2026-02-25T08:44:30.999 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2224361 2026-02-25T08:44:31.038 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~171s 2026-02-25T08:44:31.043 INFO:tasks.rados.rados.0.trial023.stdout:601: expect (ObjNum 196 snap 55 seq_num 196) 2026-02-25T08:44:31.043 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:31.088 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:31.134 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:31.180 INFO:tasks.rados.rados.0.trial023.stdout:601: done (3 left) 2026-02-25T08:44:31.180 INFO:tasks.rados.rados.0.trial023.stdout:602: done (2 left) 2026-02-25T08:44:31.180 INFO:tasks.rados.rados.0.trial023.stdout:603: done (1 left) 2026-02-25T08:44:31.180 INFO:tasks.rados.rados.0.trial023.stdout:604: done (0 left) 2026-02-25T08:44:31.180 INFO:tasks.rados.rados.0.trial023.stdout:605: write oid 38 current snap is 81 2026-02-25T08:44:31.181 INFO:tasks.rados.rados.0.trial023.stdout:605: seq_num 288 ranges {440416=776773,1686814=732100} 2026-02-25T08:44:31.185 INFO:tasks.rados.rados.0.trial023.stdout:605: writing trial02342524-38 from 440416 to 1217189 tid 1 2026-02-25T08:44:31.189 INFO:tasks.rados.rados.0.trial023.stdout:605: writing trial02342524-38 from 1686814 to 2418914 tid 2 2026-02-25T08:44:31.189 INFO:tasks.rados.rados.0.trial023.stdout:606: snap_create 2026-02-25T08:44:31.218 INFO:tasks.rados.rados.0.trial023.stdout:605: finishing write tid 1 to trial02342524-38 2026-02-25T08:44:31.225 INFO:tasks.rados.rados.0.trial023.stdout:605: finishing write tid 2 to trial02342524-38 2026-02-25T08:44:31.235 INFO:tasks.rados.rados.0.trial023.stdout:605: finishing write tid 3 to trial02342524-38 2026-02-25T08:44:31.237 INFO:tasks.rados.rados.0.trial023.stdout:605: finishing write tid 4 to trial02342524-38 2026-02-25T08:44:31.237 INFO:tasks.rados.rados.0.trial023.stdout:605: oid 38 updating version 0 to 111 2026-02-25T08:44:31.237 INFO:tasks.rados.rados.0.trial023.stdout:605: oid 38 version 111 is already newer than 111 2026-02-25T08:44:31.237 INFO:tasks.rados.rados.0.trial023.stdout:605: oid 38 version 111 is already newer than 111 2026-02-25T08:44:31.237 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 38 v 111 (ObjNum 288 snap 81 seq_num 288) dirty exists 2026-02-25T08:44:31.237 INFO:tasks.rados.rados.0.trial023.stdout:605: left oid 38 (ObjNum 288 snap 81 seq_num 288) 2026-02-25T08:44:31.697 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:31.774 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:31.771+0000 7f57c79dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:31.777 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:31.775+0000 7f57c79dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:31.792 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:44:31.821 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:31.821 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:31.875 INFO:tasks.rados.rados.0.trial023.stdout:605: done (1 left) 2026-02-25T08:44:31.875 INFO:tasks.rados.rados.0.trial023.stdout:606: done (0 left) 2026-02-25T08:44:31.875 INFO:tasks.rados.rados.0.trial023.stdout:607: read oid 42 snap -1 2026-02-25T08:44:31.875 INFO:tasks.rados.rados.0.trial023.stdout:607: expect deleted 2026-02-25T08:44:31.875 INFO:tasks.rados.rados.0.trial023.stdout:608: write oid 30 current snap is 82 2026-02-25T08:44:31.875 INFO:tasks.rados.rados.0.trial023.stdout:608: seq_num 289 ranges {457223=568375,1461649=475758,2420186=437037} 2026-02-25T08:44:31.878 INFO:tasks.rados.rados.0.trial023.stdout:608: writing trial02342524-30 from 457223 to 1025598 tid 1 2026-02-25T08:44:31.882 INFO:tasks.rados.rados.0.trial023.stdout:608: writing trial02342524-30 from 1461649 to 1937407 tid 2 2026-02-25T08:44:31.885 INFO:tasks.rados.rados.0.trial023.stdout:608: writing trial02342524-30 from 2420186 to 2857223 tid 3 2026-02-25T08:44:31.885 INFO:tasks.rados.rados.0.trial023.stdout:609: write (excl) oid 7 current snap is 82 2026-02-25T08:44:31.886 INFO:tasks.rados.rados.0.trial023.stdout:609: seq_num 290 ranges {474030=643624,1520131=735769} 2026-02-25T08:44:31.889 INFO:tasks.rados.rados.0.trial023.stdout:609: writing trial02342524-7 from 474030 to 1117654 tid 1 2026-02-25T08:44:31.894 INFO:tasks.rados.rados.0.trial023.stdout:609: writing trial02342524-7 from 1520131 to 2255900 tid 2 2026-02-25T08:44:31.894 INFO:tasks.rados.rados.0.trial023.stdout:610: delete oid 43 current snap is 82 2026-02-25T08:44:31.917 INFO:tasks.rados.rados.0.trial023.stdout:608: finishing write tid 1 to trial02342524-30 2026-02-25T08:44:31.924 INFO:tasks.rados.rados.0.trial023.stdout:608: finishing write tid 2 to trial02342524-30 2026-02-25T08:44:31.930 INFO:tasks.rados.rados.0.trial023.stdout:608: finishing write tid 3 to trial02342524-30 2026-02-25T08:44:31.937 INFO:tasks.rados.rados.0.trial023.stdout:608: finishing write tid 4 to trial02342524-30 2026-02-25T08:44:31.941 INFO:tasks.rados.rados.0.trial023.stdout:608: finishing write tid 5 to trial02342524-30 2026-02-25T08:44:31.941 INFO:tasks.rados.rados.0.trial023.stdout:608: oid 30 updating version 0 to 92 2026-02-25T08:44:31.941 INFO:tasks.rados.rados.0.trial023.stdout:608: oid 30 version 92 is already newer than 92 2026-02-25T08:44:31.941 INFO:tasks.rados.rados.0.trial023.stdout:608: oid 30 version 92 is already newer than 92 2026-02-25T08:44:31.941 INFO:tasks.rados.rados.0.trial023.stdout:608: oid 30 version 92 is already newer than 92 2026-02-25T08:44:31.941 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 30 v 92 (ObjNum 289 snap 82 seq_num 289) dirty exists 2026-02-25T08:44:31.941 INFO:tasks.rados.rados.0.trial023.stdout:608: left oid 30 (ObjNum 289 snap 82 seq_num 289) 2026-02-25T08:44:31.942 INFO:tasks.rados.rados.0.trial023.stdout:607: done (3 left) 2026-02-25T08:44:31.943 INFO:tasks.rados.rados.0.trial023.stdout:608: done (2 left) 2026-02-25T08:44:31.943 INFO:tasks.rados.rados.0.trial023.stdout:610: done (1 left) 2026-02-25T08:44:31.943 INFO:tasks.rados.rados.0.trial023.stdout:611: read oid 25 snap -1 2026-02-25T08:44:31.943 INFO:tasks.rados.rados.0.trial023.stdout:611: expect (ObjNum 282 snap 81 seq_num 282) 2026-02-25T08:44:31.943 INFO:tasks.rados.rados.0.trial023.stdout:612: snap_create 2026-02-25T08:44:31.952 INFO:tasks.rados.rados.0.trial023.stdout:609: finishing write tid 1 to trial02342524-7 2026-02-25T08:44:31.959 INFO:tasks.rados.rados.0.trial023.stdout:611: expect (ObjNum 282 snap 81 seq_num 282) 2026-02-25T08:44:31.965 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1739944 2026-02-25T08:44:31.989 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1802417 2026-02-25T08:44:32.040 INFO:tasks.rados.rados.0.trial023.stdout:609: finishing write tid 2 to trial02342524-7 2026-02-25T08:44:32.041 INFO:tasks.rados.rados.0.trial023.stdout:609: finishing write tid 3 to trial02342524-7 2026-02-25T08:44:32.041 INFO:tasks.rados.rados.0.trial023.stdout:609: finishing write tid 4 to trial02342524-7 2026-02-25T08:44:32.041 INFO:tasks.rados.rados.0.trial023.stdout:609: oid 7 updating version 0 to 94 2026-02-25T08:44:32.041 INFO:tasks.rados.rados.0.trial023.stdout:609: oid 7 updating version 94 to 96 2026-02-25T08:44:32.041 INFO:tasks.rados.rados.0.trial023.stdout:609: oid 7 version 96 is already newer than 96 2026-02-25T08:44:32.041 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 7 v 96 (ObjNum 290 snap 82 seq_num 290) dirty exists 2026-02-25T08:44:32.041 INFO:tasks.rados.rados.0.trial023.stdout:609: left oid 7 (ObjNum 290 snap 82 seq_num 290) 2026-02-25T08:44:32.879 INFO:tasks.rados.rados.0.trial023.stdout:609: done (2 left) 2026-02-25T08:44:32.879 INFO:tasks.rados.rados.0.trial023.stdout:611: done (1 left) 2026-02-25T08:44:32.879 INFO:tasks.rados.rados.0.trial023.stdout:612: done (0 left) 2026-02-25T08:44:32.880 INFO:tasks.rados.rados.0.trial023.stdout:613: snap_create 2026-02-25T08:44:33.822 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:33.880 INFO:tasks.rados.rados.0.trial023.stdout:613: done (0 left) 2026-02-25T08:44:33.880 INFO:tasks.rados.rados.0.trial023.stdout:614: snap_create 2026-02-25T08:44:33.903 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:33.900+0000 7f9040051640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:33.906 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:33.904+0000 7f9040051640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:33.921 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:44:33.947 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:33.948 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:34.232 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:44:34.304 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:44:34.307 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:44:34.308 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:44:34.885 INFO:tasks.rados.rados.0.trial023.stdout:614: done (0 left) 2026-02-25T08:44:34.885 INFO:tasks.rados.rados.0.trial023.stdout:615: snap_remove snap 77 2026-02-25T08:44:35.889 INFO:tasks.rados.rados.0.trial023.stdout:615: done (0 left) 2026-02-25T08:44:35.889 INFO:tasks.rados.rados.0.trial023.stdout:616: read oid 29 snap -1 2026-02-25T08:44:35.889 INFO:tasks.rados.rados.0.trial023.stdout:616: expect deleted 2026-02-25T08:44:35.889 INFO:tasks.rados.rados.0.trial023.stdout:617: snap_create 2026-02-25T08:44:35.949 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:36.023 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:36.021+0000 7f9b308cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:36.025 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:36.024+0000 7f9b308cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:36.044 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:44:36.071 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:36.072 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:36.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~176s 2026-02-25T08:44:36.891 INFO:tasks.rados.rados.0.trial023.stdout:616: done (1 left) 2026-02-25T08:44:36.892 INFO:tasks.rados.rados.0.trial023.stdout:617: done (0 left) 2026-02-25T08:44:36.892 INFO:tasks.rados.rados.0.trial023.stdout:618: snap_remove snap 82 2026-02-25T08:44:37.896 INFO:tasks.rados.rados.0.trial023.stdout:618: done (0 left) 2026-02-25T08:44:37.896 INFO:tasks.rados.rados.0.trial023.stdout:619: write (excl) oid 28 current snap is 86 2026-02-25T08:44:37.896 INFO:tasks.rados.rados.0.trial023.stdout:619: seq_num 292 ranges {507644=794122,1920742=623085} 2026-02-25T08:44:37.901 INFO:tasks.rados.rados.0.trial023.stdout:619: writing trial02342524-28 from 507644 to 1301766 tid 1 2026-02-25T08:44:37.906 INFO:tasks.rados.rados.0.trial023.stdout:619: writing trial02342524-28 from 1920742 to 2543827 tid 2 2026-02-25T08:44:37.906 INFO:tasks.rados.rados.0.trial023.stdout:620: snap_create 2026-02-25T08:44:37.963 INFO:tasks.rados.rados.0.trial023.stdout:619: finishing write tid 1 to trial02342524-28 2026-02-25T08:44:37.977 INFO:tasks.rados.rados.0.trial023.stdout:619: finishing write tid 2 to trial02342524-28 2026-02-25T08:44:37.982 INFO:tasks.rados.rados.0.trial023.stdout:619: finishing write tid 3 to trial02342524-28 2026-02-25T08:44:37.984 INFO:tasks.rados.rados.0.trial023.stdout:619: finishing write tid 4 to trial02342524-28 2026-02-25T08:44:37.984 INFO:tasks.rados.rados.0.trial023.stdout:619: oid 28 updating version 0 to 86 2026-02-25T08:44:37.984 INFO:tasks.rados.rados.0.trial023.stdout:619: oid 28 version 86 is already newer than 84 2026-02-25T08:44:37.984 INFO:tasks.rados.rados.0.trial023.stdout:619: oid 28 version 86 is already newer than 86 2026-02-25T08:44:37.984 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 28 v 86 (ObjNum 292 snap 86 seq_num 292) dirty exists 2026-02-25T08:44:37.984 INFO:tasks.rados.rados.0.trial023.stdout:619: left oid 28 (ObjNum 292 snap 86 seq_num 292) 2026-02-25T08:44:38.072 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:38.148 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:38.146+0000 7faf597f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:38.150 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:38.150+0000 7faf597f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:38.167 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:44:38.193 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:38.193 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:38.901 INFO:tasks.rados.rados.0.trial023.stdout:619: done (1 left) 2026-02-25T08:44:38.901 INFO:tasks.rados.rados.0.trial023.stdout:620: done (0 left) 2026-02-25T08:44:38.901 INFO:tasks.rados.rados.0.trial023.stdout:621: delete oid 11 current snap is 87 2026-02-25T08:44:38.942 INFO:tasks.rados.rados.0.trial023.stdout:621: done (0 left) 2026-02-25T08:44:38.942 INFO:tasks.rados.rados.0.trial023.stdout:622: snap_remove snap 80 2026-02-25T08:44:39.309 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:44:39.381 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:44:39.384 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:44:39.385 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:44:39.824 INFO:tasks.rados.rados.0.trial023.stdout:622: done (0 left) 2026-02-25T08:44:39.824 INFO:tasks.rados.rados.0.trial023.stdout:623: read oid 44 snap -1 2026-02-25T08:44:39.824 INFO:tasks.rados.rados.0.trial023.stdout:623: expect (ObjNum 208 snap 60 seq_num 208) 2026-02-25T08:44:39.824 INFO:tasks.rados.rados.0.trial023.stdout:624: delete oid 45 current snap is 87 2026-02-25T08:44:39.861 INFO:tasks.rados.rados.0.trial023.stdout:623: expect (ObjNum 208 snap 60 seq_num 208) 2026-02-25T08:44:39.892 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:39.921 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:39.951 INFO:tasks.rados.rados.0.trial023.stdout:623: done (1 left) 2026-02-25T08:44:39.952 INFO:tasks.rados.rados.0.trial023.stdout:624: done (0 left) 2026-02-25T08:44:39.952 INFO:tasks.rados.rados.0.trial023.stdout:625: write (excl) oid 8 current snap is 87 2026-02-25T08:44:39.952 INFO:tasks.rados.rados.0.trial023.stdout:625: seq_num 295 ranges {558065=619869,1579835=486765,2468371=489694} 2026-02-25T08:44:39.956 INFO:tasks.rados.rados.0.trial023.stdout:625: writing trial02342524-8 from 558065 to 1177934 tid 1 2026-02-25T08:44:39.959 INFO:tasks.rados.rados.0.trial023.stdout:625: writing trial02342524-8 from 1579835 to 2066600 tid 2 2026-02-25T08:44:39.963 INFO:tasks.rados.rados.0.trial023.stdout:625: writing trial02342524-8 from 2468371 to 2958065 tid 3 2026-02-25T08:44:39.963 INFO:tasks.rados.rados.0.trial023.stdout:626: write oid 49 current snap is 87 2026-02-25T08:44:39.963 INFO:tasks.rados.rados.0.trial023.stdout:626: seq_num 296 ranges {574872=695118,2038317=630423} 2026-02-25T08:44:39.968 INFO:tasks.rados.rados.0.trial023.stdout:626: writing trial02342524-49 from 574872 to 1269990 tid 1 2026-02-25T08:44:39.974 INFO:tasks.rados.rados.0.trial023.stdout:626: writing trial02342524-49 from 2038317 to 2668740 tid 2 2026-02-25T08:44:39.974 INFO:tasks.rados.rados.0.trial023.stdout:627: read oid 38 snap -1 2026-02-25T08:44:39.974 INFO:tasks.rados.rados.0.trial023.stdout:627: expect (ObjNum 288 snap 81 seq_num 288) 2026-02-25T08:44:39.974 INFO:tasks.rados.rados.0.trial023.stdout:628: read oid 43 snap -1 2026-02-25T08:44:39.974 INFO:tasks.rados.rados.0.trial023.stdout:628: expect deleted 2026-02-25T08:44:39.974 INFO:tasks.rados.rados.0.trial023.stdout:629: snap_create 2026-02-25T08:44:39.986 INFO:tasks.rados.rados.0.trial023.stdout:625: finishing write tid 1 to trial02342524-8 2026-02-25T08:44:40.004 INFO:tasks.rados.rados.0.trial023.stdout:627: expect (ObjNum 288 snap 81 seq_num 288) 2026-02-25T08:44:40.050 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 451502 2026-02-25T08:44:40.112 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2427392 2026-02-25T08:44:40.120 INFO:tasks.rados.rados.0.trial023.stdout:625: finishing write tid 2 to trial02342524-8 2026-02-25T08:44:40.120 INFO:tasks.rados.rados.0.trial023.stdout:626: finishing write tid 1 to trial02342524-49 2026-02-25T08:44:40.120 INFO:tasks.rados.rados.0.trial023.stdout:625: finishing write tid 3 to trial02342524-8 2026-02-25T08:44:40.120 INFO:tasks.rados.rados.0.trial023.stdout:625: finishing write tid 4 to trial02342524-8 2026-02-25T08:44:40.120 INFO:tasks.rados.rados.0.trial023.stdout:625: finishing write tid 5 to trial02342524-8 2026-02-25T08:44:40.120 INFO:tasks.rados.rados.0.trial023.stdout:625: oid 8 updating version 0 to 29 2026-02-25T08:44:40.120 INFO:tasks.rados.rados.0.trial023.stdout:625: oid 8 version 29 is already newer than 26 2026-02-25T08:44:40.120 INFO:tasks.rados.rados.0.trial023.stdout:625: oid 8 version 29 is already newer than 29 2026-02-25T08:44:40.120 INFO:tasks.rados.rados.0.trial023.stdout:625: oid 8 version 29 is already newer than 29 2026-02-25T08:44:40.120 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 8 v 29 (ObjNum 295 snap 87 seq_num 295) dirty exists 2026-02-25T08:44:40.121 INFO:tasks.rados.rados.0.trial023.stdout:625: left oid 8 (ObjNum 295 snap 87 seq_num 295) 2026-02-25T08:44:40.121 INFO:tasks.rados.rados.0.trial023.stdout:626: finishing write tid 2 to trial02342524-49 2026-02-25T08:44:40.121 INFO:tasks.rados.rados.0.trial023.stdout:626: finishing write tid 3 to trial02342524-49 2026-02-25T08:44:40.121 INFO:tasks.rados.rados.0.trial023.stdout:626: finishing write tid 4 to trial02342524-49 2026-02-25T08:44:40.121 INFO:tasks.rados.rados.0.trial023.stdout:626: oid 49 updating version 0 to 133 2026-02-25T08:44:40.121 INFO:tasks.rados.rados.0.trial023.stdout:626: oid 49 version 133 is already newer than 133 2026-02-25T08:44:40.121 INFO:tasks.rados.rados.0.trial023.stdout:626: oid 49 version 133 is already newer than 133 2026-02-25T08:44:40.121 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 49 v 133 (ObjNum 296 snap 87 seq_num 296) dirty exists 2026-02-25T08:44:40.121 INFO:tasks.rados.rados.0.trial023.stdout:626: left oid 49 (ObjNum 296 snap 87 seq_num 296) 2026-02-25T08:44:40.194 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:40.273 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:40.270+0000 7f9659e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:40.275 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:40.274+0000 7f9659e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:40.291 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:44:40.319 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:40.319 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:40.905 INFO:tasks.rados.rados.0.trial023.stdout:625: done (4 left) 2026-02-25T08:44:40.906 INFO:tasks.rados.rados.0.trial023.stdout:626: done (3 left) 2026-02-25T08:44:40.906 INFO:tasks.rados.rados.0.trial023.stdout:627: done (2 left) 2026-02-25T08:44:40.906 INFO:tasks.rados.rados.0.trial023.stdout:628: done (1 left) 2026-02-25T08:44:40.906 INFO:tasks.rados.rados.0.trial023.stdout:629: done (0 left) 2026-02-25T08:44:40.906 INFO:tasks.rados.rados.0.trial023.stdout:630: delete oid 3 current snap is 88 2026-02-25T08:44:40.920 INFO:tasks.rados.rados.0.trial023.stdout:630: done (0 left) 2026-02-25T08:44:40.920 INFO:tasks.rados.rados.0.trial023.stdout:631: read oid 26 snap -1 2026-02-25T08:44:40.920 INFO:tasks.rados.rados.0.trial023.stdout:631: expect (ObjNum 266 snap 78 seq_num 266) 2026-02-25T08:44:40.920 INFO:tasks.rados.rados.0.trial023.stdout:632: delete oid 19 current snap is 88 2026-02-25T08:44:40.936 INFO:tasks.rados.rados.0.trial023.stdout:631: expect (ObjNum 266 snap 78 seq_num 266) 2026-02-25T08:44:40.956 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:40.977 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:40.998 INFO:tasks.rados.rados.0.trial023.stdout:631: done (1 left) 2026-02-25T08:44:40.998 INFO:tasks.rados.rados.0.trial023.stdout:632: done (0 left) 2026-02-25T08:44:40.999 INFO:tasks.rados.rados.0.trial023.stdout:633: delete oid 27 current snap is 88 2026-02-25T08:44:41.000 INFO:tasks.rados.rados.0.trial023.stdout:633: done (0 left) 2026-02-25T08:44:41.000 INFO:tasks.rados.rados.0.trial023.stdout:634: delete oid 18 current snap is 88 2026-02-25T08:44:41.003 INFO:tasks.rados.rados.0.trial023.stdout:634: done (0 left) 2026-02-25T08:44:41.003 INFO:tasks.rados.rados.0.trial023.stdout:635: delete oid 39 current snap is 88 2026-02-25T08:44:41.033 INFO:tasks.rados.rados.0.trial023.stdout:635: done (0 left) 2026-02-25T08:44:41.033 INFO:tasks.rados.rados.0.trial023.stdout:636: snap_remove snap 81 2026-02-25T08:44:41.909 INFO:tasks.rados.rados.0.trial023.stdout:636: done (0 left) 2026-02-25T08:44:41.909 INFO:tasks.rados.rados.0.trial023.stdout:637: read oid 6 snap -1 2026-02-25T08:44:41.909 INFO:tasks.rados.rados.0.trial023.stdout:637: expect deleted 2026-02-25T08:44:41.909 INFO:tasks.rados.rados.0.trial023.stdout:638: write oid 25 current snap is 88 2026-02-25T08:44:41.909 INFO:tasks.rados.rados.0.trial023.stdout:638: seq_num 302 ranges {675714=462965,1589209=641430,2943979=131735} 2026-02-25T08:44:41.913 INFO:tasks.rados.rados.0.trial023.stdout:638: writing trial02342524-25 from 675714 to 1138679 tid 1 2026-02-25T08:44:41.917 INFO:tasks.rados.rados.0.trial023.stdout:638: writing trial02342524-25 from 1589209 to 2230639 tid 2 2026-02-25T08:44:41.920 INFO:tasks.rados.rados.0.trial023.stdout:638: writing trial02342524-25 from 2943979 to 3075714 tid 3 2026-02-25T08:44:41.920 INFO:tasks.rados.rados.0.trial023.stdout:639: snap_create 2026-02-25T08:44:41.971 INFO:tasks.rados.rados.0.trial023.stdout:638: finishing write tid 1 to trial02342524-25 2026-02-25T08:44:41.978 INFO:tasks.rados.rados.0.trial023.stdout:638: finishing write tid 2 to trial02342524-25 2026-02-25T08:44:41.984 INFO:tasks.rados.rados.0.trial023.stdout:638: finishing write tid 3 to trial02342524-25 2026-02-25T08:44:41.986 INFO:tasks.rados.rados.0.trial023.stdout:638: finishing write tid 4 to trial02342524-25 2026-02-25T08:44:41.988 INFO:tasks.rados.rados.0.trial023.stdout:638: finishing write tid 5 to trial02342524-25 2026-02-25T08:44:41.988 INFO:tasks.rados.rados.0.trial023.stdout:638: oid 25 updating version 0 to 156 2026-02-25T08:44:41.988 INFO:tasks.rados.rados.0.trial023.stdout:638: oid 25 version 156 is already newer than 156 2026-02-25T08:44:41.988 INFO:tasks.rados.rados.0.trial023.stdout:638: oid 25 version 156 is already newer than 156 2026-02-25T08:44:41.988 INFO:tasks.rados.rados.0.trial023.stdout:638: oid 25 version 156 is already newer than 156 2026-02-25T08:44:41.988 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 25 v 156 (ObjNum 302 snap 88 seq_num 302) dirty exists 2026-02-25T08:44:41.988 INFO:tasks.rados.rados.0.trial023.stdout:638: left oid 25 (ObjNum 302 snap 88 seq_num 302) 2026-02-25T08:44:42.045 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~182s 2026-02-25T08:44:42.320 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:42.397 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:42.395+0000 7ff6f8c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:42.399 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:42.398+0000 7ff6f8c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:42.414 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:44:42.442 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:42.442 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:42.912 INFO:tasks.rados.rados.0.trial023.stdout:637: done (2 left) 2026-02-25T08:44:42.913 INFO:tasks.rados.rados.0.trial023.stdout:638: done (1 left) 2026-02-25T08:44:42.913 INFO:tasks.rados.rados.0.trial023.stdout:639: done (0 left) 2026-02-25T08:44:42.913 INFO:tasks.rados.rados.0.trial023.stdout:640: read oid 3 snap 62 2026-02-25T08:44:42.913 INFO:tasks.rados.rados.0.trial023.stdout:640: expect (ObjNum 112 snap 12 seq_num 112) 2026-02-25T08:44:42.913 INFO:tasks.rados.rados.0.trial023.stdout:641: write oid 27 current snap is 89 2026-02-25T08:44:42.913 INFO:tasks.rados.rados.0.trial023.stdout:641: seq_num 303 ranges {692521=538214,1931338=785088} 2026-02-25T08:44:42.917 INFO:tasks.rados.rados.0.trial023.stdout:641: writing trial02342524-27 from 692521 to 1230735 tid 1 2026-02-25T08:44:42.923 INFO:tasks.rados.rados.0.trial023.stdout:641: writing trial02342524-27 from 1931338 to 2716426 tid 2 2026-02-25T08:44:42.923 INFO:tasks.rados.rados.0.trial023.stdout:642: snap_create 2026-02-25T08:44:42.944 INFO:tasks.rados.rados.0.trial023.stdout:640: expect (ObjNum 112 snap 12 seq_num 112) 2026-02-25T08:44:42.964 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 3188707 2026-02-25T08:44:42.988 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1876361 2026-02-25T08:44:43.043 INFO:tasks.rados.rados.0.trial023.stdout:641: finishing write tid 1 to trial02342524-27 2026-02-25T08:44:43.044 INFO:tasks.rados.rados.0.trial023.stdout:641: finishing write tid 2 to trial02342524-27 2026-02-25T08:44:43.044 INFO:tasks.rados.rados.0.trial023.stdout:641: finishing write tid 3 to trial02342524-27 2026-02-25T08:44:43.044 INFO:tasks.rados.rados.0.trial023.stdout:641: finishing write tid 4 to trial02342524-27 2026-02-25T08:44:43.044 INFO:tasks.rados.rados.0.trial023.stdout:641: oid 27 updating version 0 to 32 2026-02-25T08:44:43.044 INFO:tasks.rados.rados.0.trial023.stdout:641: oid 27 version 32 is already newer than 32 2026-02-25T08:44:43.044 INFO:tasks.rados.rados.0.trial023.stdout:641: oid 27 version 32 is already newer than 32 2026-02-25T08:44:43.044 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 27 v 32 (ObjNum 303 snap 89 seq_num 303) dirty exists 2026-02-25T08:44:43.044 INFO:tasks.rados.rados.0.trial023.stdout:641: left oid 27 (ObjNum 303 snap 89 seq_num 303) 2026-02-25T08:44:43.915 INFO:tasks.rados.rados.0.trial023.stdout:640: done (2 left) 2026-02-25T08:44:43.916 INFO:tasks.rados.rados.0.trial023.stdout:641: done (1 left) 2026-02-25T08:44:43.916 INFO:tasks.rados.rados.0.trial023.stdout:642: done (0 left) 2026-02-25T08:44:43.916 INFO:tasks.rados.rados.0.trial023.stdout:643: read oid 1 snap -1 2026-02-25T08:44:43.916 INFO:tasks.rados.rados.0.trial023.stdout:643: expect deleted 2026-02-25T08:44:43.916 INFO:tasks.rados.rados.0.trial023.stdout:644: read oid 41 snap 88 2026-02-25T08:44:43.916 INFO:tasks.rados.rados.0.trial023.stdout:644: expect (ObjNum 194 snap 51 seq_num 194) 2026-02-25T08:44:43.916 INFO:tasks.rados.rados.0.trial023.stdout:645: delete oid 43 current snap is 90 2026-02-25T08:44:43.938 INFO:tasks.rados.rados.0.trial023.stdout:643: done (2 left) 2026-02-25T08:44:43.938 INFO:tasks.rados.rados.0.trial023.stdout:645: done (1 left) 2026-02-25T08:44:43.938 INFO:tasks.rados.rados.0.trial023.stdout:646: read oid 42 snap 78 2026-02-25T08:44:43.938 INFO:tasks.rados.rados.0.trial023.stdout:646: expect deleted 2026-02-25T08:44:43.938 INFO:tasks.rados.rados.0.trial023.stdout:647: write oid 27 current snap is 90 2026-02-25T08:44:43.938 INFO:tasks.rados.rados.0.trial023.stdout:647: seq_num 305 ranges {726135=405065,1764655=788757,3026248=99887} 2026-02-25T08:44:43.942 INFO:tasks.rados.rados.0.trial023.stdout:647: writing trial02342524-27 from 726135 to 1131200 tid 1 2026-02-25T08:44:43.948 INFO:tasks.rados.rados.0.trial023.stdout:647: writing trial02342524-27 from 1764655 to 2553412 tid 2 2026-02-25T08:44:43.950 INFO:tasks.rados.rados.0.trial023.stdout:647: writing trial02342524-27 from 3026248 to 3126135 tid 3 2026-02-25T08:44:43.951 INFO:tasks.rados.rados.0.trial023.stdout:648: read oid 24 snap -1 2026-02-25T08:44:43.951 INFO:tasks.rados.rados.0.trial023.stdout:648: expect (ObjNum 246 snap 72 seq_num 246) 2026-02-25T08:44:43.951 INFO:tasks.rados.rados.0.trial023.stdout:649: write (excl) oid 35 current snap is 90 2026-02-25T08:44:43.951 INFO:tasks.rados.rados.0.trial023.stdout:649: seq_num 306 ranges {742942=480314,1823137=648768} 2026-02-25T08:44:43.955 INFO:tasks.rados.rados.0.trial023.stdout:649: writing trial02342524-35 from 742942 to 1223256 tid 1 2026-02-25T08:44:43.960 INFO:tasks.rados.rados.0.trial023.stdout:649: writing trial02342524-35 from 1823137 to 2471905 tid 2 2026-02-25T08:44:43.960 INFO:tasks.rados.rados.0.trial023.stdout:650: write (excl) oid 45 current snap is 90 2026-02-25T08:44:43.960 INFO:tasks.rados.rados.0.trial023.stdout:650: seq_num 307 ranges {759749=555563,1765266=792426,2964741=195008} 2026-02-25T08:44:43.965 INFO:tasks.rados.rados.0.trial023.stdout:650: writing trial02342524-45 from 759749 to 1315312 tid 1 2026-02-25T08:44:43.970 INFO:tasks.rados.rados.0.trial023.stdout:650: writing trial02342524-45 from 1765266 to 2557692 tid 2 2026-02-25T08:44:43.972 INFO:tasks.rados.rados.0.trial023.stdout:650: writing trial02342524-45 from 2964741 to 3159749 tid 3 2026-02-25T08:44:43.972 INFO:tasks.rados.rados.0.trial023.stdout:651: snap_create 2026-02-25T08:44:43.973 INFO:tasks.rados.rados.0.trial023.stdout:644: expect (ObjNum 194 snap 51 seq_num 194) 2026-02-25T08:44:43.973 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:44.064 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2637131 2026-02-25T08:44:44.072 INFO:tasks.rados.rados.0.trial023.stdout:647: finishing write tid 1 to trial02342524-27 2026-02-25T08:44:44.073 INFO:tasks.rados.rados.0.trial023.stdout:647: finishing write tid 2 to trial02342524-27 2026-02-25T08:44:44.073 INFO:tasks.rados.rados.0.trial023.stdout:650: finishing write tid 1 to trial02342524-45 2026-02-25T08:44:44.073 INFO:tasks.rados.rados.0.trial023.stdout:648: expect (ObjNum 246 snap 72 seq_num 246) 2026-02-25T08:44:44.076 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1073259 2026-02-25T08:44:44.141 INFO:tasks.rados.rados.0.trial023.stdout:647: finishing write tid 3 to trial02342524-27 2026-02-25T08:44:44.141 INFO:tasks.rados.rados.0.trial023.stdout:650: finishing write tid 2 to trial02342524-45 2026-02-25T08:44:44.141 INFO:tasks.rados.rados.0.trial023.stdout:647: finishing write tid 4 to trial02342524-27 2026-02-25T08:44:44.141 INFO:tasks.rados.rados.0.trial023.stdout:647: finishing write tid 5 to trial02342524-27 2026-02-25T08:44:44.141 INFO:tasks.rados.rados.0.trial023.stdout:647: oid 27 updating version 0 to 37 2026-02-25T08:44:44.141 INFO:tasks.rados.rados.0.trial023.stdout:647: oid 27 version 37 is already newer than 37 2026-02-25T08:44:44.141 INFO:tasks.rados.rados.0.trial023.stdout:647: oid 27 version 37 is already newer than 37 2026-02-25T08:44:44.141 INFO:tasks.rados.rados.0.trial023.stdout:647: oid 27 version 37 is already newer than 37 2026-02-25T08:44:44.141 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 27 v 37 (ObjNum 305 snap 90 seq_num 305) dirty exists 2026-02-25T08:44:44.141 INFO:tasks.rados.rados.0.trial023.stdout:647: left oid 27 (ObjNum 305 snap 90 seq_num 305) 2026-02-25T08:44:44.142 INFO:tasks.rados.rados.0.trial023.stdout:650: finishing write tid 3 to trial02342524-45 2026-02-25T08:44:44.142 INFO:tasks.rados.rados.0.trial023.stdout:650: finishing write tid 4 to trial02342524-45 2026-02-25T08:44:44.142 INFO:tasks.rados.rados.0.trial023.stdout:650: finishing write tid 5 to trial02342524-45 2026-02-25T08:44:44.142 INFO:tasks.rados.rados.0.trial023.stdout:650: oid 45 updating version 0 to 160 2026-02-25T08:44:44.142 INFO:tasks.rados.rados.0.trial023.stdout:650: oid 45 version 160 is already newer than 160 2026-02-25T08:44:44.142 INFO:tasks.rados.rados.0.trial023.stdout:650: oid 45 version 160 is already newer than 157 2026-02-25T08:44:44.142 INFO:tasks.rados.rados.0.trial023.stdout:650: oid 45 version 160 is already newer than 160 2026-02-25T08:44:44.142 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 45 v 160 (ObjNum 307 snap 90 seq_num 307) dirty exists 2026-02-25T08:44:44.142 INFO:tasks.rados.rados.0.trial023.stdout:650: left oid 45 (ObjNum 307 snap 90 seq_num 307) 2026-02-25T08:44:44.142 INFO:tasks.rados.rados.0.trial023.stdout:649: finishing write tid 1 to trial02342524-35 2026-02-25T08:44:44.142 INFO:tasks.rados.rados.0.trial023.stdout:649: finishing write tid 2 to trial02342524-35 2026-02-25T08:44:44.142 INFO:tasks.rados.rados.0.trial023.stdout:649: finishing write tid 3 to trial02342524-35 2026-02-25T08:44:44.143 INFO:tasks.rados.rados.0.trial023.stdout:649: finishing write tid 4 to trial02342524-35 2026-02-25T08:44:44.143 INFO:tasks.rados.rados.0.trial023.stdout:649: oid 35 updating version 0 to 178 2026-02-25T08:44:44.143 INFO:tasks.rados.rados.0.trial023.stdout:649: oid 35 version 178 is already newer than 178 2026-02-25T08:44:44.143 INFO:tasks.rados.rados.0.trial023.stdout:649: oid 35 version 178 is already newer than 176 2026-02-25T08:44:44.143 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 35 v 178 (ObjNum 306 snap 90 seq_num 306) dirty exists 2026-02-25T08:44:44.143 INFO:tasks.rados.rados.0.trial023.stdout:649: left oid 35 (ObjNum 306 snap 90 seq_num 306) 2026-02-25T08:44:44.385 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:44:44.443 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:44.457 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:44:44.460 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:44:44.460 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:44:44.518 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:44.515+0000 7fbf2e606640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:44.520 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:44.519+0000 7fbf2e606640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:44.535 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:44:44.563 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:44.563 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:44.921 INFO:tasks.rados.rados.0.trial023.stdout:644: done (6 left) 2026-02-25T08:44:44.921 INFO:tasks.rados.rados.0.trial023.stdout:646: done (5 left) 2026-02-25T08:44:44.921 INFO:tasks.rados.rados.0.trial023.stdout:647: done (4 left) 2026-02-25T08:44:44.921 INFO:tasks.rados.rados.0.trial023.stdout:648: done (3 left) 2026-02-25T08:44:44.921 INFO:tasks.rados.rados.0.trial023.stdout:649: done (2 left) 2026-02-25T08:44:44.921 INFO:tasks.rados.rados.0.trial023.stdout:650: done (1 left) 2026-02-25T08:44:44.922 INFO:tasks.rados.rados.0.trial023.stdout:651: done (0 left) 2026-02-25T08:44:44.922 INFO:tasks.rados.rados.0.trial023.stdout:652: write oid 3 current snap is 91 2026-02-25T08:44:44.922 INFO:tasks.rados.rados.0.trial023.stdout:652: seq_num 308 ranges {776556=630812,1823748=652437,3108517=68039} 2026-02-25T08:44:44.926 INFO:tasks.rados.rados.0.trial023.stdout:652: writing trial02342524-3 from 776556 to 1407368 tid 1 2026-02-25T08:44:44.930 INFO:tasks.rados.rados.0.trial023.stdout:652: writing trial02342524-3 from 1823748 to 2476185 tid 2 2026-02-25T08:44:44.933 INFO:tasks.rados.rados.0.trial023.stdout:652: writing trial02342524-3 from 3108517 to 3176556 tid 3 2026-02-25T08:44:44.933 INFO:tasks.rados.rados.0.trial023.stdout:653: read oid 50 snap 85 2026-02-25T08:44:44.933 INFO:tasks.rados.rados.0.trial023.stdout:653: expect (ObjNum 259 snap 76 seq_num 259) 2026-02-25T08:44:44.934 INFO:tasks.rados.rados.0.trial023.stdout:654: delete oid 17 current snap is 91 2026-02-25T08:44:44.955 INFO:tasks.rados.rados.0.trial023.stdout:652: finishing write tid 1 to trial02342524-3 2026-02-25T08:44:44.965 INFO:tasks.rados.rados.0.trial023.stdout:654: done (2 left) 2026-02-25T08:44:44.965 INFO:tasks.rados.rados.0.trial023.stdout:655: read oid 49 snap -1 2026-02-25T08:44:44.965 INFO:tasks.rados.rados.0.trial023.stdout:655: expect (ObjNum 296 snap 87 seq_num 296) 2026-02-25T08:44:44.965 INFO:tasks.rados.rados.0.trial023.stdout:656: snap_remove snap 87 2026-02-25T08:44:45.924 INFO:tasks.rados.rados.0.trial023.stdout:656: done (3 left) 2026-02-25T08:44:45.924 INFO:tasks.rados.rados.0.trial023.stdout:657: read oid 43 snap -1 2026-02-25T08:44:45.925 INFO:tasks.rados.rados.0.trial023.stdout:657: expect deleted 2026-02-25T08:44:45.925 INFO:tasks.rados.rados.0.trial023.stdout:658: read oid 41 snap -1 2026-02-25T08:44:45.925 INFO:tasks.rados.rados.0.trial023.stdout:658: expect (ObjNum 194 snap 51 seq_num 194) 2026-02-25T08:44:45.925 INFO:tasks.rados.rados.0.trial023.stdout:659: snap_remove snap 88 2026-02-25T08:44:46.564 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:46.642 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:46.640+0000 7f56a659c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:46.645 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:46.643+0000 7f56a659c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:46.660 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:44:46.687 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:46.688 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:46.929 INFO:tasks.rados.rados.0.trial023.stdout:659: done (5 left) 2026-02-25T08:44:46.929 INFO:tasks.rados.rados.0.trial023.stdout:660: write (excl) oid 16 current snap is 91 2026-02-25T08:44:46.929 INFO:tasks.rados.rados.0.trial023.stdout:652: finishing write tid 2 to trial02342524-3 2026-02-25T08:44:46.929 INFO:tasks.rados.rados.0.trial023.stdout:660: seq_num 310 ranges {410170=781310,1940712=656106,3163363=46807} 2026-02-25T08:44:46.933 INFO:tasks.rados.rados.0.trial023.stdout:660: writing trial02342524-16 from 410170 to 1191480 tid 1 2026-02-25T08:44:46.938 INFO:tasks.rados.rados.0.trial023.stdout:660: writing trial02342524-16 from 1940712 to 2596818 tid 2 2026-02-25T08:44:46.941 INFO:tasks.rados.rados.0.trial023.stdout:660: writing trial02342524-16 from 3163363 to 3210170 tid 3 2026-02-25T08:44:46.941 INFO:tasks.rados.rados.0.trial023.stdout:661: read oid 32 snap 85 2026-02-25T08:44:46.941 INFO:tasks.rados.rados.0.trial023.stdout:661: expect (ObjNum 224 snap 67 seq_num 224) 2026-02-25T08:44:46.941 INFO:tasks.rados.rados.0.trial023.stdout:662: write oid 14 current snap is 91 2026-02-25T08:44:46.942 INFO:tasks.rados.rados.0.trial023.stdout:662: seq_num 311 ranges {426977=456559,1482841=799764,2958080=268897} 2026-02-25T08:44:46.944 INFO:tasks.rados.rados.0.trial023.stdout:662: writing trial02342524-14 from 426977 to 883536 tid 1 2026-02-25T08:44:46.949 INFO:tasks.rados.rados.0.trial023.stdout:662: writing trial02342524-14 from 1482841 to 2282605 tid 2 2026-02-25T08:44:46.953 INFO:tasks.rados.rados.0.trial023.stdout:662: writing trial02342524-14 from 2958080 to 3226977 tid 3 2026-02-25T08:44:46.953 INFO:tasks.rados.rados.0.trial023.stdout:663: write oid 10 current snap is 91 2026-02-25T08:44:46.954 INFO:tasks.rados.rados.0.trial023.stdout:663: seq_num 312 ranges {443784=648161,1657676=659775,2818209=425575} 2026-02-25T08:44:46.957 INFO:tasks.rados.rados.0.trial023.stdout:663: writing trial02342524-10 from 443784 to 1091945 tid 1 2026-02-25T08:44:46.962 INFO:tasks.rados.rados.0.trial023.stdout:663: writing trial02342524-10 from 1657676 to 2317451 tid 2 2026-02-25T08:44:46.966 INFO:tasks.rados.rados.0.trial023.stdout:663: writing trial02342524-10 from 2818209 to 3243784 tid 3 2026-02-25T08:44:46.966 INFO:tasks.rados.rados.0.trial023.stdout:664: snap_remove snap 89 2026-02-25T08:44:47.549 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~187s 2026-02-25T08:44:47.932 INFO:tasks.rados.rados.0.trial023.stdout:664: done (9 left) 2026-02-25T08:44:47.932 INFO:tasks.rados.rados.0.trial023.stdout:665: snap_create 2026-02-25T08:44:47.932 INFO:tasks.rados.rados.0.trial023.stdout:653: expect (ObjNum 259 snap 76 seq_num 259) 2026-02-25T08:44:47.953 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:47.994 INFO:tasks.rados.rados.0.trial023.stdout:655: expect (ObjNum 296 snap 87 seq_num 296) 2026-02-25T08:44:47.997 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 879318 2026-02-25T08:44:48.068 INFO:tasks.rados.rados.0.trial023.stdout:652: finishing write tid 3 to trial02342524-3 2026-02-25T08:44:48.068 INFO:tasks.rados.rados.0.trial023.stdout:652: finishing write tid 4 to trial02342524-3 2026-02-25T08:44:48.068 INFO:tasks.rados.rados.0.trial023.stdout:652: finishing write tid 5 to trial02342524-3 2026-02-25T08:44:48.068 INFO:tasks.rados.rados.0.trial023.stdout:652: oid 3 updating version 0 to 164 2026-02-25T08:44:48.068 INFO:tasks.rados.rados.0.trial023.stdout:652: oid 3 version 164 is already newer than 164 2026-02-25T08:44:48.068 INFO:tasks.rados.rados.0.trial023.stdout:652: oid 3 version 164 is already newer than 164 2026-02-25T08:44:48.069 INFO:tasks.rados.rados.0.trial023.stdout:652: oid 3 version 164 is already newer than 164 2026-02-25T08:44:48.069 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 3 v 164 (ObjNum 308 snap 91 seq_num 308) dirty exists 2026-02-25T08:44:48.069 INFO:tasks.rados.rados.0.trial023.stdout:652: left oid 3 (ObjNum 308 snap 91 seq_num 308) 2026-02-25T08:44:48.069 INFO:tasks.rados.rados.0.trial023.stdout:658: expect (ObjNum 194 snap 51 seq_num 194) 2026-02-25T08:44:48.069 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:48.141 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:48.179 INFO:tasks.rados.rados.0.trial023.stdout:660: finishing write tid 1 to trial02342524-16 2026-02-25T08:44:48.179 INFO:tasks.rados.rados.0.trial023.stdout:663: finishing write tid 1 to trial02342524-10 2026-02-25T08:44:48.179 INFO:tasks.rados.rados.0.trial023.stdout:661: expect (ObjNum 224 snap 67 seq_num 224) 2026-02-25T08:44:48.192 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1792230 2026-02-25T08:44:48.230 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 710805 2026-02-25T08:44:48.302 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2863891 2026-02-25T08:44:48.314 INFO:tasks.rados.rados.0.trial023.stdout:663: finishing write tid 2 to trial02342524-10 2026-02-25T08:44:48.314 INFO:tasks.rados.rados.0.trial023.stdout:660: finishing write tid 2 to trial02342524-16 2026-02-25T08:44:48.314 INFO:tasks.rados.rados.0.trial023.stdout:662: finishing write tid 1 to trial02342524-14 2026-02-25T08:44:48.314 INFO:tasks.rados.rados.0.trial023.stdout:663: finishing write tid 3 to trial02342524-10 2026-02-25T08:44:48.314 INFO:tasks.rados.rados.0.trial023.stdout:660: finishing write tid 3 to trial02342524-16 2026-02-25T08:44:48.314 INFO:tasks.rados.rados.0.trial023.stdout:663: finishing write tid 4 to trial02342524-10 2026-02-25T08:44:48.314 INFO:tasks.rados.rados.0.trial023.stdout:663: finishing write tid 5 to trial02342524-10 2026-02-25T08:44:48.314 INFO:tasks.rados.rados.0.trial023.stdout:663: oid 10 updating version 0 to 137 2026-02-25T08:44:48.314 INFO:tasks.rados.rados.0.trial023.stdout:663: oid 10 version 137 is already newer than 137 2026-02-25T08:44:48.314 INFO:tasks.rados.rados.0.trial023.stdout:663: oid 10 version 137 is already newer than 137 2026-02-25T08:44:48.314 INFO:tasks.rados.rados.0.trial023.stdout:663: oid 10 version 137 is already newer than 137 2026-02-25T08:44:48.314 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 10 v 137 (ObjNum 312 snap 91 seq_num 312) dirty exists 2026-02-25T08:44:48.314 INFO:tasks.rados.rados.0.trial023.stdout:663: left oid 10 (ObjNum 312 snap 91 seq_num 312) 2026-02-25T08:44:48.314 INFO:tasks.rados.rados.0.trial023.stdout:660: finishing write tid 4 to trial02342524-16 2026-02-25T08:44:48.314 INFO:tasks.rados.rados.0.trial023.stdout:662: finishing write tid 2 to trial02342524-14 2026-02-25T08:44:48.314 INFO:tasks.rados.rados.0.trial023.stdout:660: finishing write tid 5 to trial02342524-16 2026-02-25T08:44:48.314 INFO:tasks.rados.rados.0.trial023.stdout:660: oid 16 updating version 0 to 118 2026-02-25T08:44:48.314 INFO:tasks.rados.rados.0.trial023.stdout:660: oid 16 version 118 is already newer than 118 2026-02-25T08:44:48.314 INFO:tasks.rados.rados.0.trial023.stdout:660: oid 16 version 118 is already newer than 118 2026-02-25T08:44:48.314 INFO:tasks.rados.rados.0.trial023.stdout:660: oid 16 version 118 is already newer than 115 2026-02-25T08:44:48.315 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 16 v 118 (ObjNum 310 snap 91 seq_num 310) dirty exists 2026-02-25T08:44:48.315 INFO:tasks.rados.rados.0.trial023.stdout:660: left oid 16 (ObjNum 310 snap 91 seq_num 310) 2026-02-25T08:44:48.315 INFO:tasks.rados.rados.0.trial023.stdout:662: finishing write tid 3 to trial02342524-14 2026-02-25T08:44:48.315 INFO:tasks.rados.rados.0.trial023.stdout:662: finishing write tid 4 to trial02342524-14 2026-02-25T08:44:48.315 INFO:tasks.rados.rados.0.trial023.stdout:662: finishing write tid 5 to trial02342524-14 2026-02-25T08:44:48.315 INFO:tasks.rados.rados.0.trial023.stdout:662: oid 14 updating version 0 to 47 2026-02-25T08:44:48.315 INFO:tasks.rados.rados.0.trial023.stdout:662: oid 14 version 47 is already newer than 47 2026-02-25T08:44:48.315 INFO:tasks.rados.rados.0.trial023.stdout:662: oid 14 version 47 is already newer than 47 2026-02-25T08:44:48.315 INFO:tasks.rados.rados.0.trial023.stdout:662: oid 14 version 47 is already newer than 47 2026-02-25T08:44:48.315 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 14 v 47 (ObjNum 311 snap 91 seq_num 311) dirty exists 2026-02-25T08:44:48.315 INFO:tasks.rados.rados.0.trial023.stdout:662: left oid 14 (ObjNum 311 snap 91 seq_num 311) 2026-02-25T08:44:48.689 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:48.764 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:48.762+0000 7fa5c2451640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:48.767 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:48.766+0000 7fa5c2451640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:48.782 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:44:48.809 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:48.809 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:48.937 INFO:tasks.rados.rados.0.trial023.stdout:652: done (9 left) 2026-02-25T08:44:48.937 INFO:tasks.rados.rados.0.trial023.stdout:653: done (8 left) 2026-02-25T08:44:48.937 INFO:tasks.rados.rados.0.trial023.stdout:655: done (7 left) 2026-02-25T08:44:48.937 INFO:tasks.rados.rados.0.trial023.stdout:657: done (6 left) 2026-02-25T08:44:48.937 INFO:tasks.rados.rados.0.trial023.stdout:658: done (5 left) 2026-02-25T08:44:48.937 INFO:tasks.rados.rados.0.trial023.stdout:660: done (4 left) 2026-02-25T08:44:48.937 INFO:tasks.rados.rados.0.trial023.stdout:661: done (3 left) 2026-02-25T08:44:48.937 INFO:tasks.rados.rados.0.trial023.stdout:662: done (2 left) 2026-02-25T08:44:48.937 INFO:tasks.rados.rados.0.trial023.stdout:663: done (1 left) 2026-02-25T08:44:48.938 INFO:tasks.rados.rados.0.trial023.stdout:665: done (0 left) 2026-02-25T08:44:48.938 INFO:tasks.rados.rados.0.trial023.stdout:666: read oid 7 snap -1 2026-02-25T08:44:48.938 INFO:tasks.rados.rados.0.trial023.stdout:666: expect (ObjNum 290 snap 82 seq_num 290) 2026-02-25T08:44:48.938 INFO:tasks.rados.rados.0.trial023.stdout:667: read oid 42 snap -1 2026-02-25T08:44:48.938 INFO:tasks.rados.rados.0.trial023.stdout:667: expect deleted 2026-02-25T08:44:48.938 INFO:tasks.rados.rados.0.trial023.stdout:668: read oid 3 snap -1 2026-02-25T08:44:48.938 INFO:tasks.rados.rados.0.trial023.stdout:668: expect (ObjNum 308 snap 91 seq_num 308) 2026-02-25T08:44:48.938 INFO:tasks.rados.rados.0.trial023.stdout:669: write (excl) oid 19 current snap is 92 2026-02-25T08:44:48.938 INFO:tasks.rados.rados.0.trial023.stdout:669: seq_num 313 ranges {460591=723410,1716158=403433,2729279=440140} 2026-02-25T08:44:48.940 INFO:tasks.rados.rados.0.trial023.stdout:669: writing trial02342524-19 from 460591 to 1184001 tid 1 2026-02-25T08:44:48.943 INFO:tasks.rados.rados.0.trial023.stdout:669: writing trial02342524-19 from 1716158 to 2119591 tid 2 2026-02-25T08:44:48.947 INFO:tasks.rados.rados.0.trial023.stdout:669: writing trial02342524-19 from 2729279 to 3169419 tid 3 2026-02-25T08:44:48.947 INFO:tasks.rados.rados.0.trial023.stdout:670: snap_remove snap 84 2026-02-25T08:44:49.461 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:44:49.533 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:44:49.536 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:44:49.536 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:44:49.825 INFO:tasks.rados.rados.0.trial023.stdout:670: done (4 left) 2026-02-25T08:44:49.825 INFO:tasks.rados.rados.0.trial023.stdout:671: write oid 27 current snap is 92 2026-02-25T08:44:49.825 INFO:tasks.rados.rados.0.trial023.stdout:671: seq_num 314 ranges {477398=798659,1774640=663444,2873055=404343} 2026-02-25T08:44:49.830 INFO:tasks.rados.rados.0.trial023.stdout:671: writing trial02342524-27 from 477398 to 1276057 tid 1 2026-02-25T08:44:49.835 INFO:tasks.rados.rados.0.trial023.stdout:671: writing trial02342524-27 from 1774640 to 2438084 tid 2 2026-02-25T08:44:49.839 INFO:tasks.rados.rados.0.trial023.stdout:671: writing trial02342524-27 from 2873055 to 3277398 tid 3 2026-02-25T08:44:49.839 INFO:tasks.rados.rados.0.trial023.stdout:672: snap_create 2026-02-25T08:44:49.839 INFO:tasks.rados.rados.0.trial023.stdout:668: expect (ObjNum 308 snap 91 seq_num 308) 2026-02-25T08:44:49.839 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 564204 2026-02-25T08:44:49.914 INFO:tasks.rados.rados.0.trial023.stdout:666: expect (ObjNum 290 snap 82 seq_num 290) 2026-02-25T08:44:49.966 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:50.001 INFO:tasks.rados.rados.0.trial023.stdout:669: finishing write tid 1 to trial02342524-19 2026-02-25T08:44:50.001 INFO:tasks.rados.rados.0.trial023.stdout:669: finishing write tid 2 to trial02342524-19 2026-02-25T08:44:50.001 INFO:tasks.rados.rados.0.trial023.stdout:669: finishing write tid 3 to trial02342524-19 2026-02-25T08:44:50.001 INFO:tasks.rados.rados.0.trial023.stdout:669: finishing write tid 4 to trial02342524-19 2026-02-25T08:44:50.001 INFO:tasks.rados.rados.0.trial023.stdout:669: finishing write tid 5 to trial02342524-19 2026-02-25T08:44:50.001 INFO:tasks.rados.rados.0.trial023.stdout:669: oid 19 updating version 0 to 119 2026-02-25T08:44:50.001 INFO:tasks.rados.rados.0.trial023.stdout:669: oid 19 updating version 119 to 122 2026-02-25T08:44:50.002 INFO:tasks.rados.rados.0.trial023.stdout:669: oid 19 version 122 is already newer than 122 2026-02-25T08:44:50.002 INFO:tasks.rados.rados.0.trial023.stdout:669: oid 19 version 122 is already newer than 122 2026-02-25T08:44:50.002 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 19 v 122 (ObjNum 313 snap 92 seq_num 313) dirty exists 2026-02-25T08:44:50.002 INFO:tasks.rados.rados.0.trial023.stdout:669: left oid 19 (ObjNum 313 snap 92 seq_num 313) 2026-02-25T08:44:50.002 INFO:tasks.rados.rados.0.trial023.stdout:671: finishing write tid 1 to trial02342524-27 2026-02-25T08:44:50.002 INFO:tasks.rados.rados.0.trial023.stdout:671: finishing write tid 2 to trial02342524-27 2026-02-25T08:44:50.002 INFO:tasks.rados.rados.0.trial023.stdout:671: finishing write tid 3 to trial02342524-27 2026-02-25T08:44:50.002 INFO:tasks.rados.rados.0.trial023.stdout:671: finishing write tid 4 to trial02342524-27 2026-02-25T08:44:50.002 INFO:tasks.rados.rados.0.trial023.stdout:671: finishing write tid 5 to trial02342524-27 2026-02-25T08:44:50.002 INFO:tasks.rados.rados.0.trial023.stdout:671: oid 27 updating version 0 to 42 2026-02-25T08:44:50.002 INFO:tasks.rados.rados.0.trial023.stdout:671: oid 27 version 42 is already newer than 42 2026-02-25T08:44:50.002 INFO:tasks.rados.rados.0.trial023.stdout:671: oid 27 version 42 is already newer than 42 2026-02-25T08:44:50.002 INFO:tasks.rados.rados.0.trial023.stdout:671: oid 27 version 42 is already newer than 42 2026-02-25T08:44:50.003 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 27 v 42 (ObjNum 314 snap 92 seq_num 314) dirty exists 2026-02-25T08:44:50.003 INFO:tasks.rados.rados.0.trial023.stdout:671: left oid 27 (ObjNum 314 snap 92 seq_num 314) 2026-02-25T08:44:50.810 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:50.825 INFO:tasks.rados.rados.0.trial023.stdout:666: done (5 left) 2026-02-25T08:44:50.825 INFO:tasks.rados.rados.0.trial023.stdout:667: done (4 left) 2026-02-25T08:44:50.825 INFO:tasks.rados.rados.0.trial023.stdout:668: done (3 left) 2026-02-25T08:44:50.825 INFO:tasks.rados.rados.0.trial023.stdout:669: done (2 left) 2026-02-25T08:44:50.825 INFO:tasks.rados.rados.0.trial023.stdout:671: done (1 left) 2026-02-25T08:44:50.825 INFO:tasks.rados.rados.0.trial023.stdout:672: done (0 left) 2026-02-25T08:44:50.825 INFO:tasks.rados.rados.0.trial023.stdout:673: read oid 11 snap -1 2026-02-25T08:44:50.825 INFO:tasks.rados.rados.0.trial023.stdout:673: expect deleted 2026-02-25T08:44:50.826 INFO:tasks.rados.rados.0.trial023.stdout:674: write (excl) oid 4 current snap is 93 2026-02-25T08:44:50.826 INFO:tasks.rados.rados.0.trial023.stdout:674: seq_num 315 ranges {494205=473908,1433122=407102,2384125=462684} 2026-02-25T08:44:50.828 INFO:tasks.rados.rados.0.trial023.stdout:674: writing trial02342524-4 from 494205 to 968113 tid 1 2026-02-25T08:44:50.832 INFO:tasks.rados.rados.0.trial023.stdout:674: writing trial02342524-4 from 1433122 to 1840224 tid 2 2026-02-25T08:44:50.835 INFO:tasks.rados.rados.0.trial023.stdout:674: writing trial02342524-4 from 2384125 to 2846809 tid 3 2026-02-25T08:44:50.836 INFO:tasks.rados.rados.0.trial023.stdout:675: delete oid 48 current snap is 93 2026-02-25T08:44:50.842 INFO:tasks.rados.rados.0.trial023.stdout:675: done (2 left) 2026-02-25T08:44:50.842 INFO:tasks.rados.rados.0.trial023.stdout:676: snap_remove snap 79 2026-02-25T08:44:50.891 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:50.888+0000 7fd08a65a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:50.893 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:50.892+0000 7fd08a65a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:50.909 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:44:50.934 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:50.935 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:51.827 INFO:tasks.rados.rados.0.trial023.stdout:676: done (2 left) 2026-02-25T08:44:51.827 INFO:tasks.rados.rados.0.trial023.stdout:677: write (excl) oid 22 current snap is 93 2026-02-25T08:44:51.827 INFO:tasks.rados.rados.0.trial023.stdout:677: seq_num 317 ranges {527819=624406,1833733=410771,2722618=601581} 2026-02-25T08:44:51.831 INFO:tasks.rados.rados.0.trial023.stdout:677: writing trial02342524-22 from 527819 to 1152225 tid 1 2026-02-25T08:44:51.835 INFO:tasks.rados.rados.0.trial023.stdout:677: writing trial02342524-22 from 1833733 to 2244504 tid 2 2026-02-25T08:44:51.840 INFO:tasks.rados.rados.0.trial023.stdout:677: writing trial02342524-22 from 2722618 to 3324199 tid 3 2026-02-25T08:44:51.840 INFO:tasks.rados.rados.0.trial023.stdout:678: snap_remove snap 75 2026-02-25T08:44:52.831 INFO:tasks.rados.rados.0.trial023.stdout:678: done (3 left) 2026-02-25T08:44:52.831 INFO:tasks.rados.rados.0.trial023.stdout:679: snap_remove snap 62 2026-02-25T08:44:52.936 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:53.013 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:53.010+0000 7f6d6e30b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:53.015 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:53.014+0000 7f6d6d082640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:53.031 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:44:53.052 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~193s 2026-02-25T08:44:53.057 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:53.058 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:53.834 INFO:tasks.rados.rados.0.trial023.stdout:679: done (3 left) 2026-02-25T08:44:53.834 INFO:tasks.rados.rados.0.trial023.stdout:680: read oid 50 snap 78 2026-02-25T08:44:53.835 INFO:tasks.rados.rados.0.trial023.stdout:680: expect (ObjNum 259 snap 76 seq_num 259) 2026-02-25T08:44:53.835 INFO:tasks.rados.rados.0.trial023.stdout:681: delete oid 10 current snap is 93 2026-02-25T08:44:53.835 INFO:tasks.rados.rados.0.trial023.stdout:674: finishing write tid 1 to trial02342524-4 2026-02-25T08:44:53.835 INFO:tasks.rados.rados.0.trial023.stdout:674: finishing write tid 2 to trial02342524-4 2026-02-25T08:44:53.835 INFO:tasks.rados.rados.0.trial023.stdout:674: finishing write tid 3 to trial02342524-4 2026-02-25T08:44:53.835 INFO:tasks.rados.rados.0.trial023.stdout:674: finishing write tid 4 to trial02342524-4 2026-02-25T08:44:53.835 INFO:tasks.rados.rados.0.trial023.stdout:674: finishing write tid 5 to trial02342524-4 2026-02-25T08:44:53.835 INFO:tasks.rados.rados.0.trial023.stdout:674: oid 4 updating version 0 to 169 2026-02-25T08:44:53.835 INFO:tasks.rados.rados.0.trial023.stdout:674: oid 4 version 169 is already newer than 166 2026-02-25T08:44:53.835 INFO:tasks.rados.rados.0.trial023.stdout:674: oid 4 version 169 is already newer than 169 2026-02-25T08:44:53.835 INFO:tasks.rados.rados.0.trial023.stdout:674: oid 4 version 169 is already newer than 169 2026-02-25T08:44:53.835 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 4 v 169 (ObjNum 315 snap 93 seq_num 315) dirty exists 2026-02-25T08:44:53.836 INFO:tasks.rados.rados.0.trial023.stdout:674: left oid 4 (ObjNum 315 snap 93 seq_num 315) 2026-02-25T08:44:53.836 INFO:tasks.rados.rados.0.trial023.stdout:677: finishing write tid 1 to trial02342524-22 2026-02-25T08:44:53.836 INFO:tasks.rados.rados.0.trial023.stdout:677: finishing write tid 2 to trial02342524-22 2026-02-25T08:44:53.836 INFO:tasks.rados.rados.0.trial023.stdout:677: finishing write tid 3 to trial02342524-22 2026-02-25T08:44:53.836 INFO:tasks.rados.rados.0.trial023.stdout:677: finishing write tid 4 to trial02342524-22 2026-02-25T08:44:53.836 INFO:tasks.rados.rados.0.trial023.stdout:677: finishing write tid 5 to trial02342524-22 2026-02-25T08:44:53.836 INFO:tasks.rados.rados.0.trial023.stdout:677: oid 22 updating version 0 to 50 2026-02-25T08:44:53.836 INFO:tasks.rados.rados.0.trial023.stdout:677: oid 22 version 50 is already newer than 50 2026-02-25T08:44:53.836 INFO:tasks.rados.rados.0.trial023.stdout:677: oid 22 version 50 is already newer than 50 2026-02-25T08:44:53.836 INFO:tasks.rados.rados.0.trial023.stdout:677: oid 22 version 50 is already newer than 47 2026-02-25T08:44:53.836 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 22 v 50 (ObjNum 317 snap 93 seq_num 317) dirty exists 2026-02-25T08:44:53.836 INFO:tasks.rados.rados.0.trial023.stdout:677: left oid 22 (ObjNum 317 snap 93 seq_num 317) 2026-02-25T08:44:53.861 INFO:tasks.rados.rados.0.trial023.stdout:680: expect (ObjNum 259 snap 76 seq_num 259) 2026-02-25T08:44:53.909 INFO:tasks.rados.rados.0.trial023.stdout:673: done (4 left) 2026-02-25T08:44:53.909 INFO:tasks.rados.rados.0.trial023.stdout:674: done (3 left) 2026-02-25T08:44:53.909 INFO:tasks.rados.rados.0.trial023.stdout:677: done (2 left) 2026-02-25T08:44:53.909 INFO:tasks.rados.rados.0.trial023.stdout:680: done (1 left) 2026-02-25T08:44:53.909 INFO:tasks.rados.rados.0.trial023.stdout:681: done (0 left) 2026-02-25T08:44:53.909 INFO:tasks.rados.rados.0.trial023.stdout:682: read oid 19 snap -1 2026-02-25T08:44:53.909 INFO:tasks.rados.rados.0.trial023.stdout:682: expect (ObjNum 313 snap 92 seq_num 313) 2026-02-25T08:44:53.910 INFO:tasks.rados.rados.0.trial023.stdout:683: snap_remove snap 90 2026-02-25T08:44:54.538 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:44:54.609 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:44:54.611 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:44:54.612 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:44:54.838 INFO:tasks.rados.rados.0.trial023.stdout:683: done (1 left) 2026-02-25T08:44:54.839 INFO:tasks.rados.rados.0.trial023.stdout:684: write oid 36 current snap is 93 2026-02-25T08:44:54.839 INFO:tasks.rados.rados.0.trial023.stdout:684: seq_num 319 ranges {561433=774904,1950697=414440,2777464=583969} 2026-02-25T08:44:54.844 INFO:tasks.rados.rados.0.trial023.stdout:684: writing trial02342524-36 from 561433 to 1336337 tid 1 2026-02-25T08:44:54.848 INFO:tasks.rados.rados.0.trial023.stdout:684: writing trial02342524-36 from 1950697 to 2365137 tid 2 2026-02-25T08:44:54.852 INFO:tasks.rados.rados.0.trial023.stdout:684: writing trial02342524-36 from 2777464 to 3361433 tid 3 2026-02-25T08:44:54.852 INFO:tasks.rados.rados.0.trial023.stdout:685: read oid 49 snap -1 2026-02-25T08:44:54.852 INFO:tasks.rados.rados.0.trial023.stdout:685: expect (ObjNum 296 snap 87 seq_num 296) 2026-02-25T08:44:54.852 INFO:tasks.rados.rados.0.trial023.stdout:686: write oid 10 current snap is 93 2026-02-25T08:44:54.852 INFO:tasks.rados.rados.0.trial023.stdout:686: seq_num 320 ranges {578240=566506,1609179=558098,2804887=551750} 2026-02-25T08:44:54.856 INFO:tasks.rados.rados.0.trial023.stdout:686: writing trial02342524-10 from 578240 to 1144746 tid 1 2026-02-25T08:44:54.860 INFO:tasks.rados.rados.0.trial023.stdout:686: writing trial02342524-10 from 1609179 to 2167277 tid 2 2026-02-25T08:44:54.864 INFO:tasks.rados.rados.0.trial023.stdout:686: writing trial02342524-10 from 2804887 to 3356637 tid 3 2026-02-25T08:44:54.864 INFO:tasks.rados.rados.0.trial023.stdout:687: write (excl) oid 41 current snap is 93 2026-02-25T08:44:54.864 INFO:tasks.rados.rados.0.trial023.stdout:682: expect (ObjNum 313 snap 92 seq_num 313) 2026-02-25T08:44:54.872 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1977414 2026-02-25T08:44:54.938 INFO:tasks.rados.rados.0.trial023.stdout:687: seq_num 321 ranges {595047=641755,1667661=418109,2548663=763022} 2026-02-25T08:44:54.943 INFO:tasks.rados.rados.0.trial023.stdout:687: writing trial02342524-41 from 595047 to 1236802 tid 1 2026-02-25T08:44:54.947 INFO:tasks.rados.rados.0.trial023.stdout:687: writing trial02342524-41 from 1667661 to 2085770 tid 2 2026-02-25T08:44:54.952 INFO:tasks.rados.rados.0.trial023.stdout:687: writing trial02342524-41 from 2548663 to 3311685 tid 3 2026-02-25T08:44:54.952 INFO:tasks.rados.rados.0.trial023.stdout:682: done (4 left) 2026-02-25T08:44:54.952 INFO:tasks.rados.rados.0.trial023.stdout:688: delete oid 45 current snap is 93 2026-02-25T08:44:54.952 INFO:tasks.rados.rados.0.trial023.stdout:685: expect (ObjNum 296 snap 87 seq_num 296) 2026-02-25T08:44:54.953 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:55.029 INFO:tasks.rados.rados.0.trial023.stdout:686: finishing write tid 1 to trial02342524-10 2026-02-25T08:44:55.029 INFO:tasks.rados.rados.0.trial023.stdout:686: finishing write tid 2 to trial02342524-10 2026-02-25T08:44:55.029 INFO:tasks.rados.rados.0.trial023.stdout:686: finishing write tid 3 to trial02342524-10 2026-02-25T08:44:55.029 INFO:tasks.rados.rados.0.trial023.stdout:684: finishing write tid 1 to trial02342524-36 2026-02-25T08:44:55.029 INFO:tasks.rados.rados.0.trial023.stdout:686: finishing write tid 4 to trial02342524-10 2026-02-25T08:44:55.029 INFO:tasks.rados.rados.0.trial023.stdout:686: finishing write tid 5 to trial02342524-10 2026-02-25T08:44:55.029 INFO:tasks.rados.rados.0.trial023.stdout:686: oid 10 updating version 0 to 143 2026-02-25T08:44:55.030 INFO:tasks.rados.rados.0.trial023.stdout:686: oid 10 version 143 is already newer than 143 2026-02-25T08:44:55.030 INFO:tasks.rados.rados.0.trial023.stdout:686: oid 10 version 143 is already newer than 143 2026-02-25T08:44:55.030 INFO:tasks.rados.rados.0.trial023.stdout:686: oid 10 version 143 is already newer than 143 2026-02-25T08:44:55.030 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 10 v 143 (ObjNum 320 snap 93 seq_num 320) dirty exists 2026-02-25T08:44:55.030 INFO:tasks.rados.rados.0.trial023.stdout:686: left oid 10 (ObjNum 320 snap 93 seq_num 320) 2026-02-25T08:44:55.030 INFO:tasks.rados.rados.0.trial023.stdout:684: finishing write tid 2 to trial02342524-36 2026-02-25T08:44:55.030 INFO:tasks.rados.rados.0.trial023.stdout:684: finishing write tid 3 to trial02342524-36 2026-02-25T08:44:55.030 INFO:tasks.rados.rados.0.trial023.stdout:684: finishing write tid 4 to trial02342524-36 2026-02-25T08:44:55.030 INFO:tasks.rados.rados.0.trial023.stdout:685: done (4 left) 2026-02-25T08:44:55.030 INFO:tasks.rados.rados.0.trial023.stdout:686: done (3 left) 2026-02-25T08:44:55.030 INFO:tasks.rados.rados.0.trial023.stdout:688: done (2 left) 2026-02-25T08:44:55.030 INFO:tasks.rados.rados.0.trial023.stdout:689: read oid 15 snap -1 2026-02-25T08:44:55.031 INFO:tasks.rados.rados.0.trial023.stdout:689: expect (ObjNum 273 snap 79 seq_num 273) 2026-02-25T08:44:55.031 INFO:tasks.rados.rados.0.trial023.stdout:690: read oid 10 snap -1 2026-02-25T08:44:55.031 INFO:tasks.rados.rados.0.trial023.stdout:690: expect (ObjNum 320 snap 93 seq_num 320) 2026-02-25T08:44:55.031 INFO:tasks.rados.rados.0.trial023.stdout:691: read oid 1 snap 92 2026-02-25T08:44:55.031 INFO:tasks.rados.rados.0.trial023.stdout:691: expect deleted 2026-02-25T08:44:55.031 INFO:tasks.rados.rados.0.trial023.stdout:692: read oid 29 snap -1 2026-02-25T08:44:55.031 INFO:tasks.rados.rados.0.trial023.stdout:692: expect deleted 2026-02-25T08:44:55.031 INFO:tasks.rados.rados.0.trial023.stdout:693: write oid 33 current snap is 93 2026-02-25T08:44:55.031 INFO:tasks.rados.rados.0.trial023.stdout:684: finishing write tid 5 to trial02342524-36 2026-02-25T08:44:55.031 INFO:tasks.rados.rados.0.trial023.stdout:684: oid 36 updating version 0 to 113 2026-02-25T08:44:55.031 INFO:tasks.rados.rados.0.trial023.stdout:684: oid 36 version 113 is already newer than 113 2026-02-25T08:44:55.031 INFO:tasks.rados.rados.0.trial023.stdout:684: oid 36 version 113 is already newer than 113 2026-02-25T08:44:55.032 INFO:tasks.rados.rados.0.trial023.stdout:684: oid 36 version 113 is already newer than 113 2026-02-25T08:44:55.032 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 36 v 113 (ObjNum 319 snap 93 seq_num 319) dirty exists 2026-02-25T08:44:55.032 INFO:tasks.rados.rados.0.trial023.stdout:684: left oid 36 (ObjNum 319 snap 93 seq_num 319) 2026-02-25T08:44:55.032 INFO:tasks.rados.rados.0.trial023.stdout:693: seq_num 323 ranges {628661=792253,2184625=421778,3403509=25152} 2026-02-25T08:44:55.033 INFO:tasks.rados.rados.0.trial023.stdout:693: writing trial02342524-33 from 628661 to 1420914 tid 1 2026-02-25T08:44:55.037 INFO:tasks.rados.rados.0.trial023.stdout:693: writing trial02342524-33 from 2184625 to 2606403 tid 2 2026-02-25T08:44:55.041 INFO:tasks.rados.rados.0.trial023.stdout:693: writing trial02342524-33 from 3403509 to 3428661 tid 3 2026-02-25T08:44:55.041 INFO:tasks.rados.rados.0.trial023.stdout:684: done (6 left) 2026-02-25T08:44:55.041 INFO:tasks.rados.rados.0.trial023.stdout:694: write (excl) oid 32 current snap is 93 2026-02-25T08:44:55.041 INFO:tasks.rados.rados.0.trial023.stdout:694: seq_num 324 ranges {645468=467502,1726754=565436,2798226=596838} 2026-02-25T08:44:55.045 INFO:tasks.rados.rados.0.trial023.stdout:694: writing trial02342524-32 from 645468 to 1112970 tid 1 2026-02-25T08:44:55.049 INFO:tasks.rados.rados.0.trial023.stdout:694: writing trial02342524-32 from 1726754 to 2292190 tid 2 2026-02-25T08:44:55.053 INFO:tasks.rados.rados.0.trial023.stdout:694: writing trial02342524-32 from 2798226 to 3395064 tid 3 2026-02-25T08:44:55.054 INFO:tasks.rados.rados.0.trial023.stdout:695: snap_remove snap 83 2026-02-25T08:44:55.059 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:55.136 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:55.133+0000 7f9bd0fdc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:55.138 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:55.136+0000 7f9bd0fdc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:55.154 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:44:55.179 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:55.180 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:55.842 INFO:tasks.rados.rados.0.trial023.stdout:695: done (7 left) 2026-02-25T08:44:55.843 INFO:tasks.rados.rados.0.trial023.stdout:696: write (excl) oid 14 current snap is 93 2026-02-25T08:44:55.843 INFO:tasks.rados.rados.0.trial023.stdout:687: finishing write tid 1 to trial02342524-41 2026-02-25T08:44:55.843 INFO:tasks.rados.rados.0.trial023.stdout:696: seq_num 325 ranges {662275=542751,1785236=425447,2942002=520273} 2026-02-25T08:44:55.847 INFO:tasks.rados.rados.0.trial023.stdout:696: writing trial02342524-14 from 662275 to 1205026 tid 1 2026-02-25T08:44:55.851 INFO:tasks.rados.rados.0.trial023.stdout:696: writing trial02342524-14 from 1785236 to 2210683 tid 2 2026-02-25T08:44:55.856 INFO:tasks.rados.rados.0.trial023.stdout:696: writing trial02342524-14 from 2942002 to 3462275 tid 3 2026-02-25T08:44:55.856 INFO:tasks.rados.rados.0.trial023.stdout:697: write (excl) oid 34 current snap is 93 2026-02-25T08:44:55.856 INFO:tasks.rados.rados.0.trial023.stdout:687: finishing write tid 2 to trial02342524-41 2026-02-25T08:44:55.856 INFO:tasks.rados.rados.0.trial023.stdout:697: seq_num 326 ranges {679082=618000,1843718=569105,2853072=626010} 2026-02-25T08:44:55.860 INFO:tasks.rados.rados.0.trial023.stdout:697: writing trial02342524-34 from 679082 to 1297082 tid 1 2026-02-25T08:44:55.865 INFO:tasks.rados.rados.0.trial023.stdout:697: writing trial02342524-34 from 1843718 to 2412823 tid 2 2026-02-25T08:44:55.870 INFO:tasks.rados.rados.0.trial023.stdout:697: writing trial02342524-34 from 2853072 to 3479082 tid 3 2026-02-25T08:44:55.870 INFO:tasks.rados.rados.0.trial023.stdout:698: read oid 28 snap -1 2026-02-25T08:44:55.870 INFO:tasks.rados.rados.0.trial023.stdout:698: expect (ObjNum 292 snap 86 seq_num 292) 2026-02-25T08:44:55.870 INFO:tasks.rados.rados.0.trial023.stdout:699: write oid 2 current snap is 93 2026-02-25T08:44:55.870 INFO:tasks.rados.rados.0.trial023.stdout:687: finishing write tid 3 to trial02342524-41 2026-02-25T08:44:55.870 INFO:tasks.rados.rados.0.trial023.stdout:699: seq_num 327 ranges {695889=409602,1618553=712763,2996848=499041} 2026-02-25T08:44:55.873 INFO:tasks.rados.rados.0.trial023.stdout:699: writing trial02342524-2 from 695889 to 1105491 tid 1 2026-02-25T08:44:55.878 INFO:tasks.rados.rados.0.trial023.stdout:699: writing trial02342524-2 from 1618553 to 2331316 tid 2 2026-02-25T08:44:55.883 INFO:tasks.rados.rados.0.trial023.stdout:699: writing trial02342524-2 from 2996848 to 3495889 tid 3 2026-02-25T08:44:55.884 INFO:tasks.rados.rados.0.trial023.stdout:700: delete oid 19 current snap is 93 2026-02-25T08:44:55.884 INFO:tasks.rados.rados.0.trial023.stdout:687: finishing write tid 4 to trial02342524-41 2026-02-25T08:44:55.884 INFO:tasks.rados.rados.0.trial023.stdout:687: finishing write tid 5 to trial02342524-41 2026-02-25T08:44:55.884 INFO:tasks.rados.rados.0.trial023.stdout:687: oid 41 updating version 0 to 98 2026-02-25T08:44:55.884 INFO:tasks.rados.rados.0.trial023.stdout:687: oid 41 updating version 98 to 101 2026-02-25T08:44:55.884 INFO:tasks.rados.rados.0.trial023.stdout:687: oid 41 version 101 is already newer than 101 2026-02-25T08:44:55.884 INFO:tasks.rados.rados.0.trial023.stdout:687: oid 41 version 101 is already newer than 101 2026-02-25T08:44:55.884 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 41 v 101 (ObjNum 321 snap 93 seq_num 321) dirty exists 2026-02-25T08:44:55.884 INFO:tasks.rados.rados.0.trial023.stdout:687: left oid 41 (ObjNum 321 snap 93 seq_num 321) 2026-02-25T08:44:55.884 INFO:tasks.rados.rados.0.trial023.stdout:690: expect (ObjNum 320 snap 93 seq_num 320) 2026-02-25T08:44:55.905 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:55.936 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1497052 2026-02-25T08:44:55.954 INFO:tasks.rados.rados.0.trial023.stdout:689: expect (ObjNum 273 snap 79 seq_num 273) 2026-02-25T08:44:56.057 INFO:tasks.rados.rados.0.trial023.stdout:693: finishing write tid 1 to trial02342524-33 2026-02-25T08:44:56.057 INFO:tasks.rados.rados.0.trial023.stdout:694: finishing write tid 1 to trial02342524-32 2026-02-25T08:44:56.057 INFO:tasks.rados.rados.0.trial023.stdout:693: finishing write tid 2 to trial02342524-33 2026-02-25T08:44:56.057 INFO:tasks.rados.rados.0.trial023.stdout:687: done (11 left) 2026-02-25T08:44:56.057 INFO:tasks.rados.rados.0.trial023.stdout:689: done (10 left) 2026-02-25T08:44:56.057 INFO:tasks.rados.rados.0.trial023.stdout:690: done (9 left) 2026-02-25T08:44:56.057 INFO:tasks.rados.rados.0.trial023.stdout:691: done (8 left) 2026-02-25T08:44:56.057 INFO:tasks.rados.rados.0.trial023.stdout:692: done (7 left) 2026-02-25T08:44:56.058 INFO:tasks.rados.rados.0.trial023.stdout:700: done (6 left) 2026-02-25T08:44:56.058 INFO:tasks.rados.rados.0.trial023.stdout:701: delete oid 44 current snap is 93 2026-02-25T08:44:56.058 INFO:tasks.rados.rados.0.trial023.stdout:693: finishing write tid 3 to trial02342524-33 2026-02-25T08:44:56.058 INFO:tasks.rados.rados.0.trial023.stdout:694: finishing write tid 2 to trial02342524-32 2026-02-25T08:44:56.058 INFO:tasks.rados.rados.0.trial023.stdout:693: finishing write tid 4 to trial02342524-33 2026-02-25T08:44:56.058 INFO:tasks.rados.rados.0.trial023.stdout:693: finishing write tid 5 to trial02342524-33 2026-02-25T08:44:56.058 INFO:tasks.rados.rados.0.trial023.stdout:693: oid 33 updating version 0 to 35 2026-02-25T08:44:56.058 INFO:tasks.rados.rados.0.trial023.stdout:693: oid 33 version 35 is already newer than 35 2026-02-25T08:44:56.058 INFO:tasks.rados.rados.0.trial023.stdout:693: oid 33 version 35 is already newer than 35 2026-02-25T08:44:56.058 INFO:tasks.rados.rados.0.trial023.stdout:693: oid 33 version 35 is already newer than 35 2026-02-25T08:44:56.058 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 33 v 35 (ObjNum 323 snap 93 seq_num 323) dirty exists 2026-02-25T08:44:56.058 INFO:tasks.rados.rados.0.trial023.stdout:693: left oid 33 (ObjNum 323 snap 93 seq_num 323) 2026-02-25T08:44:56.058 INFO:tasks.rados.rados.0.trial023.stdout:694: finishing write tid 3 to trial02342524-32 2026-02-25T08:44:56.059 INFO:tasks.rados.rados.0.trial023.stdout:694: finishing write tid 4 to trial02342524-32 2026-02-25T08:44:56.059 INFO:tasks.rados.rados.0.trial023.stdout:694: finishing write tid 5 to trial02342524-32 2026-02-25T08:44:56.059 INFO:tasks.rados.rados.0.trial023.stdout:694: oid 32 updating version 0 to 106 2026-02-25T08:44:56.059 INFO:tasks.rados.rados.0.trial023.stdout:694: oid 32 version 106 is already newer than 106 2026-02-25T08:44:56.059 INFO:tasks.rados.rados.0.trial023.stdout:694: oid 32 version 106 is already newer than 106 2026-02-25T08:44:56.059 INFO:tasks.rados.rados.0.trial023.stdout:694: oid 32 version 106 is already newer than 103 2026-02-25T08:44:56.059 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 32 v 106 (ObjNum 324 snap 93 seq_num 324) dirty exists 2026-02-25T08:44:56.059 INFO:tasks.rados.rados.0.trial023.stdout:694: left oid 32 (ObjNum 324 snap 93 seq_num 324) 2026-02-25T08:44:56.059 INFO:tasks.rados.rados.0.trial023.stdout:698: expect (ObjNum 292 snap 86 seq_num 292) 2026-02-25T08:44:56.083 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:44:56.165 INFO:tasks.rados.rados.0.trial023.stdout:696: finishing write tid 1 to trial02342524-14 2026-02-25T08:44:56.165 INFO:tasks.rados.rados.0.trial023.stdout:696: finishing write tid 2 to trial02342524-14 2026-02-25T08:44:56.165 INFO:tasks.rados.rados.0.trial023.stdout:696: finishing write tid 3 to trial02342524-14 2026-02-25T08:44:56.165 INFO:tasks.rados.rados.0.trial023.stdout:696: finishing write tid 4 to trial02342524-14 2026-02-25T08:44:56.165 INFO:tasks.rados.rados.0.trial023.stdout:696: finishing write tid 5 to trial02342524-14 2026-02-25T08:44:56.166 INFO:tasks.rados.rados.0.trial023.stdout:696: oid 14 updating version 0 to 52 2026-02-25T08:44:56.166 INFO:tasks.rados.rados.0.trial023.stdout:696: oid 14 version 52 is already newer than 49 2026-02-25T08:44:56.166 INFO:tasks.rados.rados.0.trial023.stdout:696: oid 14 version 52 is already newer than 52 2026-02-25T08:44:56.166 INFO:tasks.rados.rados.0.trial023.stdout:696: oid 14 version 52 is already newer than 52 2026-02-25T08:44:56.166 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 14 v 52 (ObjNum 325 snap 93 seq_num 325) dirty exists 2026-02-25T08:44:56.166 INFO:tasks.rados.rados.0.trial023.stdout:696: left oid 14 (ObjNum 325 snap 93 seq_num 325) 2026-02-25T08:44:56.166 INFO:tasks.rados.rados.0.trial023.stdout:699: finishing write tid 1 to trial02342524-2 2026-02-25T08:44:56.166 INFO:tasks.rados.rados.0.trial023.stdout:699: finishing write tid 2 to trial02342524-2 2026-02-25T08:44:56.166 INFO:tasks.rados.rados.0.trial023.stdout:699: finishing write tid 3 to trial02342524-2 2026-02-25T08:44:56.166 INFO:tasks.rados.rados.0.trial023.stdout:699: finishing write tid 4 to trial02342524-2 2026-02-25T08:44:56.166 INFO:tasks.rados.rados.0.trial023.stdout:693: done (6 left) 2026-02-25T08:44:56.166 INFO:tasks.rados.rados.0.trial023.stdout:694: done (5 left) 2026-02-25T08:44:56.167 INFO:tasks.rados.rados.0.trial023.stdout:696: done (4 left) 2026-02-25T08:44:56.167 INFO:tasks.rados.rados.0.trial023.stdout:698: done (3 left) 2026-02-25T08:44:56.167 INFO:tasks.rados.rados.0.trial023.stdout:701: done (2 left) 2026-02-25T08:44:56.167 INFO:tasks.rados.rados.0.trial023.stdout:702: read oid 29 snap -1 2026-02-25T08:44:56.167 INFO:tasks.rados.rados.0.trial023.stdout:702: expect deleted 2026-02-25T08:44:56.167 INFO:tasks.rados.rados.0.trial023.stdout:703: write (excl) oid 26 current snap is 93 2026-02-25T08:44:56.167 INFO:tasks.rados.rados.0.trial023.stdout:699: finishing write tid 5 to trial02342524-2 2026-02-25T08:44:56.167 INFO:tasks.rados.rados.0.trial023.stdout:699: oid 2 updating version 0 to 57 2026-02-25T08:44:56.167 INFO:tasks.rados.rados.0.trial023.stdout:699: oid 2 version 57 is already newer than 57 2026-02-25T08:44:56.167 INFO:tasks.rados.rados.0.trial023.stdout:699: oid 2 version 57 is already newer than 57 2026-02-25T08:44:56.167 INFO:tasks.rados.rados.0.trial023.stdout:699: oid 2 version 57 is already newer than 57 2026-02-25T08:44:56.167 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 2 v 57 (ObjNum 327 snap 93 seq_num 327) dirty exists 2026-02-25T08:44:56.168 INFO:tasks.rados.rados.0.trial023.stdout:699: left oid 2 (ObjNum 327 snap 93 seq_num 327) 2026-02-25T08:44:56.168 INFO:tasks.rados.rados.0.trial023.stdout:697: finishing write tid 1 to trial02342524-34 2026-02-25T08:44:56.168 INFO:tasks.rados.rados.0.trial023.stdout:697: finishing write tid 2 to trial02342524-34 2026-02-25T08:44:56.168 INFO:tasks.rados.rados.0.trial023.stdout:703: seq_num 330 ranges {746310=635349,2077646=576443,3362764=183546} 2026-02-25T08:44:56.169 INFO:tasks.rados.rados.0.trial023.stdout:703: writing trial02342524-26 from 746310 to 1381659 tid 1 2026-02-25T08:44:56.174 INFO:tasks.rados.rados.0.trial023.stdout:703: writing trial02342524-26 from 2077646 to 2654089 tid 2 2026-02-25T08:44:56.177 INFO:tasks.rados.rados.0.trial023.stdout:703: writing trial02342524-26 from 3362764 to 3546310 tid 3 2026-02-25T08:44:56.177 INFO:tasks.rados.rados.0.trial023.stdout:699: done (3 left) 2026-02-25T08:44:56.177 INFO:tasks.rados.rados.0.trial023.stdout:704: write (excl) oid 40 current snap is 93 2026-02-25T08:44:56.177 INFO:tasks.rados.rados.0.trial023.stdout:704: seq_num 331 ranges {763117=710598,2136128=720101,3390187=172930} 2026-02-25T08:44:56.182 INFO:tasks.rados.rados.0.trial023.stdout:704: writing trial02342524-40 from 763117 to 1473715 tid 1 2026-02-25T08:44:56.188 INFO:tasks.rados.rados.0.trial023.stdout:704: writing trial02342524-40 from 2136128 to 2856229 tid 2 2026-02-25T08:44:56.191 INFO:tasks.rados.rados.0.trial023.stdout:704: writing trial02342524-40 from 3390187 to 3563117 tid 3 2026-02-25T08:44:56.191 INFO:tasks.rados.rados.0.trial023.stdout:705: snap_create 2026-02-25T08:44:56.191 INFO:tasks.rados.rados.0.trial023.stdout:697: finishing write tid 3 to trial02342524-34 2026-02-25T08:44:56.191 INFO:tasks.rados.rados.0.trial023.stdout:697: finishing write tid 4 to trial02342524-34 2026-02-25T08:44:56.191 INFO:tasks.rados.rados.0.trial023.stdout:697: finishing write tid 5 to trial02342524-34 2026-02-25T08:44:56.191 INFO:tasks.rados.rados.0.trial023.stdout:697: oid 34 updating version 0 to 55 2026-02-25T08:44:56.191 INFO:tasks.rados.rados.0.trial023.stdout:697: oid 34 version 55 is already newer than 52 2026-02-25T08:44:56.191 INFO:tasks.rados.rados.0.trial023.stdout:697: oid 34 version 55 is already newer than 55 2026-02-25T08:44:56.191 INFO:tasks.rados.rados.0.trial023.stdout:697: oid 34 version 55 is already newer than 55 2026-02-25T08:44:56.192 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 34 v 55 (ObjNum 326 snap 93 seq_num 326) dirty exists 2026-02-25T08:44:56.192 INFO:tasks.rados.rados.0.trial023.stdout:697: left oid 34 (ObjNum 326 snap 93 seq_num 326) 2026-02-25T08:44:56.196 INFO:tasks.rados.rados.0.trial023.stdout:703: finishing write tid 1 to trial02342524-26 2026-02-25T08:44:56.209 INFO:tasks.rados.rados.0.trial023.stdout:704: finishing write tid 1 to trial02342524-40 2026-02-25T08:44:56.215 INFO:tasks.rados.rados.0.trial023.stdout:703: finishing write tid 2 to trial02342524-26 2026-02-25T08:44:56.220 INFO:tasks.rados.rados.0.trial023.stdout:703: finishing write tid 3 to trial02342524-26 2026-02-25T08:44:56.223 INFO:tasks.rados.rados.0.trial023.stdout:703: finishing write tid 4 to trial02342524-26 2026-02-25T08:44:56.226 INFO:tasks.rados.rados.0.trial023.stdout:703: finishing write tid 5 to trial02342524-26 2026-02-25T08:44:56.226 INFO:tasks.rados.rados.0.trial023.stdout:703: oid 26 updating version 0 to 145 2026-02-25T08:44:56.226 INFO:tasks.rados.rados.0.trial023.stdout:703: oid 26 updating version 145 to 148 2026-02-25T08:44:56.226 INFO:tasks.rados.rados.0.trial023.stdout:703: oid 26 version 148 is already newer than 148 2026-02-25T08:44:56.226 INFO:tasks.rados.rados.0.trial023.stdout:703: oid 26 version 148 is already newer than 148 2026-02-25T08:44:56.226 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 26 v 148 (ObjNum 330 snap 93 seq_num 330) dirty exists 2026-02-25T08:44:56.226 INFO:tasks.rados.rados.0.trial023.stdout:703: left oid 26 (ObjNum 330 snap 93 seq_num 330) 2026-02-25T08:44:56.226 INFO:tasks.rados.rados.0.trial023.stdout:704: finishing write tid 2 to trial02342524-40 2026-02-25T08:44:56.234 INFO:tasks.rados.rados.0.trial023.stdout:704: finishing write tid 3 to trial02342524-40 2026-02-25T08:44:56.240 INFO:tasks.rados.rados.0.trial023.stdout:704: finishing write tid 4 to trial02342524-40 2026-02-25T08:44:56.242 INFO:tasks.rados.rados.0.trial023.stdout:704: finishing write tid 5 to trial02342524-40 2026-02-25T08:44:56.242 INFO:tasks.rados.rados.0.trial023.stdout:704: oid 40 updating version 0 to 120 2026-02-25T08:44:56.242 INFO:tasks.rados.rados.0.trial023.stdout:704: oid 40 version 120 is already newer than 117 2026-02-25T08:44:56.242 INFO:tasks.rados.rados.0.trial023.stdout:704: oid 40 version 120 is already newer than 120 2026-02-25T08:44:56.242 INFO:tasks.rados.rados.0.trial023.stdout:704: oid 40 version 120 is already newer than 120 2026-02-25T08:44:56.242 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 40 v 120 (ObjNum 331 snap 93 seq_num 331) dirty exists 2026-02-25T08:44:56.242 INFO:tasks.rados.rados.0.trial023.stdout:704: left oid 40 (ObjNum 331 snap 93 seq_num 331) 2026-02-25T08:44:56.847 INFO:tasks.rados.rados.0.trial023.stdout:697: done (4 left) 2026-02-25T08:44:56.847 INFO:tasks.rados.rados.0.trial023.stdout:702: done (3 left) 2026-02-25T08:44:56.847 INFO:tasks.rados.rados.0.trial023.stdout:703: done (2 left) 2026-02-25T08:44:56.847 INFO:tasks.rados.rados.0.trial023.stdout:704: done (1 left) 2026-02-25T08:44:56.848 INFO:tasks.rados.rados.0.trial023.stdout:705: done (0 left) 2026-02-25T08:44:56.848 INFO:tasks.rados.rados.0.trial023.stdout:706: read oid 48 snap -1 2026-02-25T08:44:56.848 INFO:tasks.rados.rados.0.trial023.stdout:706: expect deleted 2026-02-25T08:44:56.848 INFO:tasks.rados.rados.0.trial023.stdout:707: write oid 30 current snap is 94 2026-02-25T08:44:56.848 INFO:tasks.rados.rados.0.trial023.stdout:707: seq_num 332 ranges {779924=785847,2078257=580112,3301257=278667} 2026-02-25T08:44:56.853 INFO:tasks.rados.rados.0.trial023.stdout:707: writing trial02342524-30 from 779924 to 1565771 tid 1 2026-02-25T08:44:56.857 INFO:tasks.rados.rados.0.trial023.stdout:707: writing trial02342524-30 from 2078257 to 2658369 tid 2 2026-02-25T08:44:56.861 INFO:tasks.rados.rados.0.trial023.stdout:707: writing trial02342524-30 from 3301257 to 3579924 tid 3 2026-02-25T08:44:56.861 INFO:tasks.rados.rados.0.trial023.stdout:708: write oid 46 current snap is 94 2026-02-25T08:44:56.861 INFO:tasks.rados.rados.0.trial023.stdout:708: seq_num 333 ranges {796731=461096,1736739=723770,2928680=668051} 2026-02-25T08:44:56.866 INFO:tasks.rados.rados.0.trial023.stdout:708: writing trial02342524-46 from 796731 to 1257827 tid 1 2026-02-25T08:44:56.870 INFO:tasks.rados.rados.0.trial023.stdout:708: writing trial02342524-46 from 1736739 to 2460509 tid 2 2026-02-25T08:44:56.875 INFO:tasks.rados.rados.0.trial023.stdout:708: writing trial02342524-46 from 2928680 to 3596731 tid 3 2026-02-25T08:44:56.875 INFO:tasks.rados.rados.0.trial023.stdout:709: write (excl) oid 37 current snap is 94 2026-02-25T08:44:56.875 INFO:tasks.rados.rados.0.trial023.stdout:709: seq_num 334 ranges {413538=536345,1395221=583781,2556103=658617} 2026-02-25T08:44:56.879 INFO:tasks.rados.rados.0.trial023.stdout:709: writing trial02342524-37 from 413538 to 949883 tid 1 2026-02-25T08:44:56.882 INFO:tasks.rados.rados.0.trial023.stdout:709: writing trial02342524-37 from 1395221 to 1979002 tid 2 2026-02-25T08:44:56.887 INFO:tasks.rados.rados.0.trial023.stdout:709: writing trial02342524-37 from 2556103 to 3214720 tid 3 2026-02-25T08:44:56.887 INFO:tasks.rados.rados.0.trial023.stdout:710: write (excl) oid 25 current snap is 94 2026-02-25T08:44:56.887 INFO:tasks.rados.rados.0.trial023.stdout:710: seq_num 335 ranges {430345=727947,1570056=727439,2699879=469889} 2026-02-25T08:44:56.891 INFO:tasks.rados.rados.0.trial023.stdout:710: writing trial02342524-25 from 430345 to 1158292 tid 1 2026-02-25T08:44:56.896 INFO:tasks.rados.rados.0.trial023.stdout:710: writing trial02342524-25 from 1570056 to 2297495 tid 2 2026-02-25T08:44:56.900 INFO:tasks.rados.rados.0.trial023.stdout:710: writing trial02342524-25 from 2699879 to 3169768 tid 3 2026-02-25T08:44:56.900 INFO:tasks.rados.rados.0.trial023.stdout:711: delete oid 29 current snap is 94 2026-02-25T08:44:56.907 INFO:tasks.rados.rados.0.trial023.stdout:707: finishing write tid 1 to trial02342524-30 2026-02-25T08:44:56.910 INFO:tasks.rados.rados.0.trial023.stdout:709: finishing write tid 1 to trial02342524-37 2026-02-25T08:44:56.914 INFO:tasks.rados.rados.0.trial023.stdout:708: finishing write tid 1 to trial02342524-46 2026-02-25T08:44:56.915 INFO:tasks.rados.rados.0.trial023.stdout:707: finishing write tid 2 to trial02342524-30 2026-02-25T08:44:56.922 INFO:tasks.rados.rados.0.trial023.stdout:706: done (5 left) 2026-02-25T08:44:56.922 INFO:tasks.rados.rados.0.trial023.stdout:711: done (4 left) 2026-02-25T08:44:56.922 INFO:tasks.rados.rados.0.trial023.stdout:712: delete oid 12 current snap is 94 2026-02-25T08:44:56.923 INFO:tasks.rados.rados.0.trial023.stdout:707: finishing write tid 3 to trial02342524-30 2026-02-25T08:44:56.929 INFO:tasks.rados.rados.0.trial023.stdout:708: finishing write tid 2 to trial02342524-46 2026-02-25T08:44:56.930 INFO:tasks.rados.rados.0.trial023.stdout:707: finishing write tid 4 to trial02342524-30 2026-02-25T08:44:56.933 INFO:tasks.rados.rados.0.trial023.stdout:707: finishing write tid 5 to trial02342524-30 2026-02-25T08:44:56.933 INFO:tasks.rados.rados.0.trial023.stdout:707: oid 30 updating version 0 to 111 2026-02-25T08:44:56.933 INFO:tasks.rados.rados.0.trial023.stdout:707: oid 30 version 111 is already newer than 111 2026-02-25T08:44:56.934 INFO:tasks.rados.rados.0.trial023.stdout:707: oid 30 version 111 is already newer than 111 2026-02-25T08:44:56.934 INFO:tasks.rados.rados.0.trial023.stdout:707: oid 30 version 111 is already newer than 111 2026-02-25T08:44:56.934 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 30 v 111 (ObjNum 332 snap 94 seq_num 332) dirty exists 2026-02-25T08:44:56.934 INFO:tasks.rados.rados.0.trial023.stdout:707: left oid 30 (ObjNum 332 snap 94 seq_num 332) 2026-02-25T08:44:56.937 INFO:tasks.rados.rados.0.trial023.stdout:709: finishing write tid 2 to trial02342524-37 2026-02-25T08:44:56.937 INFO:tasks.rados.rados.0.trial023.stdout:708: finishing write tid 3 to trial02342524-46 2026-02-25T08:44:56.944 INFO:tasks.rados.rados.0.trial023.stdout:707: done (4 left) 2026-02-25T08:44:56.944 INFO:tasks.rados.rados.0.trial023.stdout:712: done (3 left) 2026-02-25T08:44:56.944 INFO:tasks.rados.rados.0.trial023.stdout:713: snap_create 2026-02-25T08:44:56.944 INFO:tasks.rados.rados.0.trial023.stdout:709: finishing write tid 3 to trial02342524-37 2026-02-25T08:44:56.944 INFO:tasks.rados.rados.0.trial023.stdout:708: finishing write tid 4 to trial02342524-46 2026-02-25T08:44:56.946 INFO:tasks.rados.rados.0.trial023.stdout:708: finishing write tid 5 to trial02342524-46 2026-02-25T08:44:56.947 INFO:tasks.rados.rados.0.trial023.stdout:708: oid 46 updating version 0 to 195 2026-02-25T08:44:56.947 INFO:tasks.rados.rados.0.trial023.stdout:708: oid 46 version 195 is already newer than 195 2026-02-25T08:44:56.947 INFO:tasks.rados.rados.0.trial023.stdout:708: oid 46 version 195 is already newer than 195 2026-02-25T08:44:56.947 INFO:tasks.rados.rados.0.trial023.stdout:708: oid 46 version 195 is already newer than 195 2026-02-25T08:44:56.947 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 46 v 195 (ObjNum 333 snap 94 seq_num 333) dirty exists 2026-02-25T08:44:56.947 INFO:tasks.rados.rados.0.trial023.stdout:708: left oid 46 (ObjNum 333 snap 94 seq_num 333) 2026-02-25T08:44:56.949 INFO:tasks.rados.rados.0.trial023.stdout:709: finishing write tid 4 to trial02342524-37 2026-02-25T08:44:56.951 INFO:tasks.rados.rados.0.trial023.stdout:709: finishing write tid 5 to trial02342524-37 2026-02-25T08:44:56.952 INFO:tasks.rados.rados.0.trial023.stdout:709: oid 37 updating version 0 to 153 2026-02-25T08:44:56.952 INFO:tasks.rados.rados.0.trial023.stdout:709: oid 37 version 153 is already newer than 153 2026-02-25T08:44:56.952 INFO:tasks.rados.rados.0.trial023.stdout:709: oid 37 version 153 is already newer than 150 2026-02-25T08:44:56.952 INFO:tasks.rados.rados.0.trial023.stdout:709: oid 37 version 153 is already newer than 153 2026-02-25T08:44:56.952 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 37 v 153 (ObjNum 334 snap 94 seq_num 334) dirty exists 2026-02-25T08:44:56.952 INFO:tasks.rados.rados.0.trial023.stdout:709: left oid 37 (ObjNum 334 snap 94 seq_num 334) 2026-02-25T08:44:56.959 INFO:tasks.rados.rados.0.trial023.stdout:710: finishing write tid 1 to trial02342524-25 2026-02-25T08:44:56.975 INFO:tasks.rados.rados.0.trial023.stdout:710: finishing write tid 2 to trial02342524-25 2026-02-25T08:44:56.982 INFO:tasks.rados.rados.0.trial023.stdout:710: finishing write tid 3 to trial02342524-25 2026-02-25T08:44:56.987 INFO:tasks.rados.rados.0.trial023.stdout:710: finishing write tid 4 to trial02342524-25 2026-02-25T08:44:56.989 INFO:tasks.rados.rados.0.trial023.stdout:710: finishing write tid 5 to trial02342524-25 2026-02-25T08:44:56.989 INFO:tasks.rados.rados.0.trial023.stdout:710: oid 25 updating version 0 to 176 2026-02-25T08:44:56.989 INFO:tasks.rados.rados.0.trial023.stdout:710: oid 25 version 176 is already newer than 176 2026-02-25T08:44:56.989 INFO:tasks.rados.rados.0.trial023.stdout:710: oid 25 version 176 is already newer than 176 2026-02-25T08:44:56.989 INFO:tasks.rados.rados.0.trial023.stdout:710: oid 25 version 176 is already newer than 173 2026-02-25T08:44:56.989 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 25 v 176 (ObjNum 335 snap 94 seq_num 335) dirty exists 2026-02-25T08:44:56.989 INFO:tasks.rados.rados.0.trial023.stdout:710: left oid 25 (ObjNum 335 snap 94 seq_num 335) 2026-02-25T08:44:57.180 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:57.258 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:57.254+0000 7f00c73ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:57.260 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:57.258+0000 7f00c73ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:57.275 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:44:57.301 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:57.302 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:57.851 INFO:tasks.rados.rados.0.trial023.stdout:708: done (3 left) 2026-02-25T08:44:57.851 INFO:tasks.rados.rados.0.trial023.stdout:709: done (2 left) 2026-02-25T08:44:57.851 INFO:tasks.rados.rados.0.trial023.stdout:710: done (1 left) 2026-02-25T08:44:57.851 INFO:tasks.rados.rados.0.trial023.stdout:713: done (0 left) 2026-02-25T08:44:57.851 INFO:tasks.rados.rados.0.trial023.stdout:714: read oid 42 snap -1 2026-02-25T08:44:57.851 INFO:tasks.rados.rados.0.trial023.stdout:714: expect deleted 2026-02-25T08:44:57.852 INFO:tasks.rados.rados.0.trial023.stdout:715: snap_remove snap 72 2026-02-25T08:44:58.556 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~198s 2026-02-25T08:44:58.855 INFO:tasks.rados.rados.0.trial023.stdout:715: done (1 left) 2026-02-25T08:44:58.855 INFO:tasks.rados.rados.0.trial023.stdout:716: delete oid 31 current snap is 95 2026-02-25T08:44:58.877 INFO:tasks.rados.rados.0.trial023.stdout:714: done (1 left) 2026-02-25T08:44:58.877 INFO:tasks.rados.rados.0.trial023.stdout:716: done (0 left) 2026-02-25T08:44:58.877 INFO:tasks.rados.rados.0.trial023.stdout:717: snap_remove snap 91 2026-02-25T08:44:59.302 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:59.380 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:59.377+0000 7fb16785a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:59.382 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:44:59.380+0000 7fb16785a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:59.397 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:44:59.423 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:59.424 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:59.613 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:44:59.685 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:44:59.687 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:44:59.688 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:44:59.828 INFO:tasks.rados.rados.0.trial023.stdout:717: done (0 left) 2026-02-25T08:44:59.828 INFO:tasks.rados.rados.0.trial023.stdout:718: write (excl) oid 40 current snap is 95 2026-02-25T08:44:59.829 INFO:tasks.rados.rados.0.trial023.stdout:718: seq_num 339 ranges {497573=628943,1687631=734777,3093218=604355} 2026-02-25T08:44:59.832 INFO:tasks.rados.rados.0.trial023.stdout:718: writing trial02342524-40 from 497573 to 1126516 tid 1 2026-02-25T08:44:59.838 INFO:tasks.rados.rados.0.trial023.stdout:718: writing trial02342524-40 from 1687631 to 2422408 tid 2 2026-02-25T08:44:59.843 INFO:tasks.rados.rados.0.trial023.stdout:718: writing trial02342524-40 from 3093218 to 3697573 tid 3 2026-02-25T08:44:59.843 INFO:tasks.rados.rados.0.trial023.stdout:719: snap_create 2026-02-25T08:44:59.868 INFO:tasks.rados.rados.0.trial023.stdout:718: finishing write tid 1 to trial02342524-40 2026-02-25T08:44:59.884 INFO:tasks.rados.rados.0.trial023.stdout:718: finishing write tid 2 to trial02342524-40 2026-02-25T08:44:59.892 INFO:tasks.rados.rados.0.trial023.stdout:718: finishing write tid 3 to trial02342524-40 2026-02-25T08:44:59.896 INFO:tasks.rados.rados.0.trial023.stdout:718: finishing write tid 4 to trial02342524-40 2026-02-25T08:44:59.898 INFO:tasks.rados.rados.0.trial023.stdout:718: finishing write tid 5 to trial02342524-40 2026-02-25T08:44:59.898 INFO:tasks.rados.rados.0.trial023.stdout:718: oid 40 updating version 0 to 131 2026-02-25T08:44:59.898 INFO:tasks.rados.rados.0.trial023.stdout:718: oid 40 version 131 is already newer than 128 2026-02-25T08:44:59.899 INFO:tasks.rados.rados.0.trial023.stdout:718: oid 40 version 131 is already newer than 131 2026-02-25T08:44:59.899 INFO:tasks.rados.rados.0.trial023.stdout:718: oid 40 version 131 is already newer than 131 2026-02-25T08:44:59.899 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 40 v 131 (ObjNum 339 snap 95 seq_num 339) dirty exists 2026-02-25T08:44:59.899 INFO:tasks.rados.rados.0.trial023.stdout:718: left oid 40 (ObjNum 339 snap 95 seq_num 339) 2026-02-25T08:45:00.832 INFO:tasks.rados.rados.0.trial023.stdout:718: done (1 left) 2026-02-25T08:45:00.832 INFO:tasks.rados.rados.0.trial023.stdout:719: done (0 left) 2026-02-25T08:45:00.833 INFO:tasks.rados.rados.0.trial023.stdout:720: snap_remove snap 78 2026-02-25T08:45:01.425 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:01.501 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:01.498+0000 7f0acdccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:01.503 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:01.501+0000 7f0acdccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:01.518 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:45:01.545 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:01.546 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:01.832 INFO:tasks.rados.rados.0.trial023.stdout:720: done (0 left) 2026-02-25T08:45:01.832 INFO:tasks.rados.rados.0.trial023.stdout:721: read oid 12 snap -1 2026-02-25T08:45:01.832 INFO:tasks.rados.rados.0.trial023.stdout:721: expect deleted 2026-02-25T08:45:01.833 INFO:tasks.rados.rados.0.trial023.stdout:722: snap_create 2026-02-25T08:45:02.837 INFO:tasks.rados.rados.0.trial023.stdout:721: done (1 left) 2026-02-25T08:45:02.838 INFO:tasks.rados.rados.0.trial023.stdout:722: done (0 left) 2026-02-25T08:45:02.838 INFO:tasks.rados.rados.0.trial023.stdout:723: write (excl) oid 7 current snap is 97 2026-02-25T08:45:02.838 INFO:tasks.rados.rados.0.trial023.stdout:723: seq_num 340 ranges {514380=704192,1629760=478435,2604288=442602} 2026-02-25T08:45:02.842 INFO:tasks.rados.rados.0.trial023.stdout:723: writing trial02342524-7 from 514380 to 1218572 tid 1 2026-02-25T08:45:02.846 INFO:tasks.rados.rados.0.trial023.stdout:723: writing trial02342524-7 from 1629760 to 2108195 tid 2 2026-02-25T08:45:02.850 INFO:tasks.rados.rados.0.trial023.stdout:723: writing trial02342524-7 from 2604288 to 3046890 tid 3 2026-02-25T08:45:02.850 INFO:tasks.rados.rados.0.trial023.stdout:724: read oid 24 snap -1 2026-02-25T08:45:02.850 INFO:tasks.rados.rados.0.trial023.stdout:724: expect (ObjNum 246 snap 72 seq_num 246) 2026-02-25T08:45:02.850 INFO:tasks.rados.rados.0.trial023.stdout:725: write (excl) oid 45 current snap is 97 2026-02-25T08:45:02.850 INFO:tasks.rados.rados.0.trial023.stdout:725: seq_num 341 ranges {531187=779441,2088242=738446,3431711=299476} 2026-02-25T08:45:02.855 INFO:tasks.rados.rados.0.trial023.stdout:725: writing trial02342524-45 from 531187 to 1310628 tid 1 2026-02-25T08:45:02.861 INFO:tasks.rados.rados.0.trial023.stdout:725: writing trial02342524-45 from 2088242 to 2826688 tid 2 2026-02-25T08:45:02.864 INFO:tasks.rados.rados.0.trial023.stdout:725: writing trial02342524-45 from 3431711 to 3731187 tid 3 2026-02-25T08:45:02.864 INFO:tasks.rados.rados.0.trial023.stdout:726: write oid 18 current snap is 97 2026-02-25T08:45:02.865 INFO:tasks.rados.rados.0.trial023.stdout:726: seq_num 342 ranges {547994=454690,1746724=482104,2659134=465146,3605976=142018} 2026-02-25T08:45:02.868 INFO:tasks.rados.rados.0.trial023.stdout:726: writing trial02342524-18 from 547994 to 1002684 tid 1 2026-02-25T08:45:02.874 INFO:tasks.rados.rados.0.trial023.stdout:726: writing trial02342524-18 from 1746724 to 2228828 tid 2 2026-02-25T08:45:02.877 INFO:tasks.rados.rados.0.trial023.stdout:726: writing trial02342524-18 from 2659134 to 3124280 tid 3 2026-02-25T08:45:02.880 INFO:tasks.rados.rados.0.trial023.stdout:726: writing trial02342524-18 from 3605976 to 3747994 tid 4 2026-02-25T08:45:02.880 INFO:tasks.rados.rados.0.trial023.stdout:727: snap_remove snap 71 2026-02-25T08:45:03.547 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:03.623 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:03.619+0000 7faa9461a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:03.626 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:03.624+0000 7faa9461a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:03.641 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:45:03.668 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:03.668 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:03.842 INFO:tasks.rados.rados.0.trial023.stdout:727: done (4 left) 2026-02-25T08:45:03.842 INFO:tasks.rados.rados.0.trial023.stdout:728: snap_create 2026-02-25T08:45:03.842 INFO:tasks.rados.rados.0.trial023.stdout:725: finishing write tid 1 to trial02342524-45 2026-02-25T08:45:03.842 INFO:tasks.rados.rados.0.trial023.stdout:723: finishing write tid 1 to trial02342524-7 2026-02-25T08:45:03.842 INFO:tasks.rados.rados.0.trial023.stdout:726: finishing write tid 1 to trial02342524-18 2026-02-25T08:45:03.842 INFO:tasks.rados.rados.0.trial023.stdout:726: finishing write tid 2 to trial02342524-18 2026-02-25T08:45:03.842 INFO:tasks.rados.rados.0.trial023.stdout:725: finishing write tid 2 to trial02342524-45 2026-02-25T08:45:03.843 INFO:tasks.rados.rados.0.trial023.stdout:726: finishing write tid 3 to trial02342524-18 2026-02-25T08:45:03.843 INFO:tasks.rados.rados.0.trial023.stdout:725: finishing write tid 3 to trial02342524-45 2026-02-25T08:45:03.843 INFO:tasks.rados.rados.0.trial023.stdout:723: finishing write tid 2 to trial02342524-7 2026-02-25T08:45:03.843 INFO:tasks.rados.rados.0.trial023.stdout:725: finishing write tid 4 to trial02342524-45 2026-02-25T08:45:03.843 INFO:tasks.rados.rados.0.trial023.stdout:725: finishing write tid 5 to trial02342524-45 2026-02-25T08:45:03.843 INFO:tasks.rados.rados.0.trial023.stdout:725: oid 45 updating version 0 to 180 2026-02-25T08:45:03.843 INFO:tasks.rados.rados.0.trial023.stdout:725: oid 45 version 180 is already newer than 180 2026-02-25T08:45:03.843 INFO:tasks.rados.rados.0.trial023.stdout:725: oid 45 version 180 is already newer than 177 2026-02-25T08:45:03.843 INFO:tasks.rados.rados.0.trial023.stdout:725: oid 45 version 180 is already newer than 180 2026-02-25T08:45:03.843 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 45 v 180 (ObjNum 341 snap 97 seq_num 341) dirty exists 2026-02-25T08:45:03.843 INFO:tasks.rados.rados.0.trial023.stdout:725: left oid 45 (ObjNum 341 snap 97 seq_num 341) 2026-02-25T08:45:03.843 INFO:tasks.rados.rados.0.trial023.stdout:726: finishing write tid 4 to trial02342524-18 2026-02-25T08:45:03.843 INFO:tasks.rados.rados.0.trial023.stdout:726: finishing write tid 5 to trial02342524-18 2026-02-25T08:45:03.844 INFO:tasks.rados.rados.0.trial023.stdout:726: finishing write tid 6 to trial02342524-18 2026-02-25T08:45:03.844 INFO:tasks.rados.rados.0.trial023.stdout:726: oid 18 updating version 0 to 129 2026-02-25T08:45:03.844 INFO:tasks.rados.rados.0.trial023.stdout:726: oid 18 version 129 is already newer than 129 2026-02-25T08:45:03.844 INFO:tasks.rados.rados.0.trial023.stdout:726: oid 18 version 129 is already newer than 129 2026-02-25T08:45:03.844 INFO:tasks.rados.rados.0.trial023.stdout:726: oid 18 version 129 is already newer than 129 2026-02-25T08:45:03.844 INFO:tasks.rados.rados.0.trial023.stdout:726: oid 18 version 129 is already newer than 129 2026-02-25T08:45:03.844 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 18 v 129 (ObjNum 342 snap 97 seq_num 342) dirty exists 2026-02-25T08:45:03.844 INFO:tasks.rados.rados.0.trial023.stdout:726: left oid 18 (ObjNum 342 snap 97 seq_num 342) 2026-02-25T08:45:03.844 INFO:tasks.rados.rados.0.trial023.stdout:723: finishing write tid 3 to trial02342524-7 2026-02-25T08:45:03.844 INFO:tasks.rados.rados.0.trial023.stdout:724: expect (ObjNum 246 snap 72 seq_num 246) 2026-02-25T08:45:03.845 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 652679 2026-02-25T08:45:03.879 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:03.904 INFO:tasks.rados.rados.0.trial023.stdout:723: finishing write tid 4 to trial02342524-7 2026-02-25T08:45:03.904 INFO:tasks.rados.rados.0.trial023.stdout:723: finishing write tid 5 to trial02342524-7 2026-02-25T08:45:03.904 INFO:tasks.rados.rados.0.trial023.stdout:723: oid 7 updating version 0 to 116 2026-02-25T08:45:03.904 INFO:tasks.rados.rados.0.trial023.stdout:723: oid 7 version 116 is already newer than 116 2026-02-25T08:45:03.904 INFO:tasks.rados.rados.0.trial023.stdout:723: oid 7 version 116 is already newer than 113 2026-02-25T08:45:03.904 INFO:tasks.rados.rados.0.trial023.stdout:723: oid 7 version 116 is already newer than 116 2026-02-25T08:45:03.904 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 7 v 116 (ObjNum 340 snap 97 seq_num 340) dirty exists 2026-02-25T08:45:03.905 INFO:tasks.rados.rados.0.trial023.stdout:723: left oid 7 (ObjNum 340 snap 97 seq_num 340) 2026-02-25T08:45:04.060 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~204s 2026-02-25T08:45:04.689 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:45:04.761 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:45:04.764 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:45:04.765 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:45:04.844 INFO:tasks.rados.rados.0.trial023.stdout:723: done (4 left) 2026-02-25T08:45:04.844 INFO:tasks.rados.rados.0.trial023.stdout:724: done (3 left) 2026-02-25T08:45:04.844 INFO:tasks.rados.rados.0.trial023.stdout:725: done (2 left) 2026-02-25T08:45:04.844 INFO:tasks.rados.rados.0.trial023.stdout:726: done (1 left) 2026-02-25T08:45:04.844 INFO:tasks.rados.rados.0.trial023.stdout:728: done (0 left) 2026-02-25T08:45:04.844 INFO:tasks.rados.rados.0.trial023.stdout:729: snap_remove snap 94 2026-02-25T08:45:05.670 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:05.746 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:05.743+0000 7f8a3eb0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:05.749 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:05.747+0000 7f8a3eb0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:05.763 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:45:05.791 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:05.791 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:05.846 INFO:tasks.rados.rados.0.trial023.stdout:729: done (0 left) 2026-02-25T08:45:05.846 INFO:tasks.rados.rados.0.trial023.stdout:730: delete oid 19 current snap is 98 2026-02-25T08:45:05.868 INFO:tasks.rados.rados.0.trial023.stdout:730: done (0 left) 2026-02-25T08:45:05.868 INFO:tasks.rados.rados.0.trial023.stdout:731: write oid 48 current snap is 98 2026-02-25T08:45:05.868 INFO:tasks.rados.rados.0.trial023.stdout:731: seq_num 344 ranges {581608=721541,1863688=485773,3113980=604043} 2026-02-25T08:45:05.873 INFO:tasks.rados.rados.0.trial023.stdout:731: writing trial02342524-48 from 581608 to 1303149 tid 1 2026-02-25T08:45:05.877 INFO:tasks.rados.rados.0.trial023.stdout:731: writing trial02342524-48 from 1863688 to 2349461 tid 2 2026-02-25T08:45:05.883 INFO:tasks.rados.rados.0.trial023.stdout:731: writing trial02342524-48 from 3113980 to 3718023 tid 3 2026-02-25T08:45:05.883 INFO:tasks.rados.rados.0.trial023.stdout:732: delete oid 50 current snap is 98 2026-02-25T08:45:05.924 INFO:tasks.rados.rados.0.trial023.stdout:732: done (1 left) 2026-02-25T08:45:05.924 INFO:tasks.rados.rados.0.trial023.stdout:733: snap_remove snap 93 2026-02-25T08:45:06.852 INFO:tasks.rados.rados.0.trial023.stdout:733: done (1 left) 2026-02-25T08:45:06.852 INFO:tasks.rados.rados.0.trial023.stdout:734: write oid 47 current snap is 98 2026-02-25T08:45:06.852 INFO:tasks.rados.rados.0.trial023.stdout:731: finishing write tid 1 to trial02342524-48 2026-02-25T08:45:06.852 INFO:tasks.rados.rados.0.trial023.stdout:734: seq_num 346 ranges {615222=472039,1580652=489442,2768826=626587} 2026-02-25T08:45:06.856 INFO:tasks.rados.rados.0.trial023.stdout:734: writing trial02342524-47 from 615222 to 1087261 tid 1 2026-02-25T08:45:06.860 INFO:tasks.rados.rados.0.trial023.stdout:734: writing trial02342524-47 from 1580652 to 2070094 tid 2 2026-02-25T08:45:06.866 INFO:tasks.rados.rados.0.trial023.stdout:734: writing trial02342524-47 from 2768826 to 3395413 tid 3 2026-02-25T08:45:06.866 INFO:tasks.rados.rados.0.trial023.stdout:735: snap_remove snap 92 2026-02-25T08:45:07.793 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:07.854 INFO:tasks.rados.rados.0.trial023.stdout:735: done (2 left) 2026-02-25T08:45:07.855 INFO:tasks.rados.rados.0.trial023.stdout:736: snap_create 2026-02-25T08:45:07.855 INFO:tasks.rados.rados.0.trial023.stdout:731: finishing write tid 2 to trial02342524-48 2026-02-25T08:45:07.855 INFO:tasks.rados.rados.0.trial023.stdout:731: finishing write tid 3 to trial02342524-48 2026-02-25T08:45:07.855 INFO:tasks.rados.rados.0.trial023.stdout:731: finishing write tid 4 to trial02342524-48 2026-02-25T08:45:07.855 INFO:tasks.rados.rados.0.trial023.stdout:731: finishing write tid 5 to trial02342524-48 2026-02-25T08:45:07.855 INFO:tasks.rados.rados.0.trial023.stdout:731: oid 48 updating version 0 to 128 2026-02-25T08:45:07.855 INFO:tasks.rados.rados.0.trial023.stdout:731: oid 48 version 128 is already newer than 128 2026-02-25T08:45:07.855 INFO:tasks.rados.rados.0.trial023.stdout:731: oid 48 version 128 is already newer than 128 2026-02-25T08:45:07.855 INFO:tasks.rados.rados.0.trial023.stdout:731: oid 48 version 128 is already newer than 128 2026-02-25T08:45:07.855 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 48 v 128 (ObjNum 344 snap 98 seq_num 344) dirty exists 2026-02-25T08:45:07.855 INFO:tasks.rados.rados.0.trial023.stdout:731: left oid 48 (ObjNum 344 snap 98 seq_num 344) 2026-02-25T08:45:07.856 INFO:tasks.rados.rados.0.trial023.stdout:734: finishing write tid 1 to trial02342524-47 2026-02-25T08:45:07.856 INFO:tasks.rados.rados.0.trial023.stdout:734: finishing write tid 2 to trial02342524-47 2026-02-25T08:45:07.856 INFO:tasks.rados.rados.0.trial023.stdout:734: finishing write tid 3 to trial02342524-47 2026-02-25T08:45:07.856 INFO:tasks.rados.rados.0.trial023.stdout:734: finishing write tid 4 to trial02342524-47 2026-02-25T08:45:07.856 INFO:tasks.rados.rados.0.trial023.stdout:734: finishing write tid 5 to trial02342524-47 2026-02-25T08:45:07.856 INFO:tasks.rados.rados.0.trial023.stdout:734: oid 47 updating version 0 to 46 2026-02-25T08:45:07.856 INFO:tasks.rados.rados.0.trial023.stdout:734: oid 47 version 46 is already newer than 46 2026-02-25T08:45:07.856 INFO:tasks.rados.rados.0.trial023.stdout:734: oid 47 version 46 is already newer than 46 2026-02-25T08:45:07.856 INFO:tasks.rados.rados.0.trial023.stdout:734: oid 47 version 46 is already newer than 46 2026-02-25T08:45:07.856 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 47 v 46 (ObjNum 346 snap 98 seq_num 346) dirty exists 2026-02-25T08:45:07.856 INFO:tasks.rados.rados.0.trial023.stdout:734: left oid 47 (ObjNum 346 snap 98 seq_num 346) 2026-02-25T08:45:07.873 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:07.870+0000 7fb7f75ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:07.876 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:07.874+0000 7fb7f75ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:07.890 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:45:07.916 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:07.917 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:08.857 INFO:tasks.rados.rados.0.trial023.stdout:731: done (2 left) 2026-02-25T08:45:08.858 INFO:tasks.rados.rados.0.trial023.stdout:734: done (1 left) 2026-02-25T08:45:08.858 INFO:tasks.rados.rados.0.trial023.stdout:736: done (0 left) 2026-02-25T08:45:08.858 INFO:tasks.rados.rados.0.trial023.stdout:737: read oid 44 snap 95 2026-02-25T08:45:08.858 INFO:tasks.rados.rados.0.trial023.stdout:737: expect deleted 2026-02-25T08:45:08.858 INFO:tasks.rados.rados.0.trial023.stdout:738: read oid 47 snap -1 2026-02-25T08:45:08.858 INFO:tasks.rados.rados.0.trial023.stdout:738: expect (ObjNum 346 snap 98 seq_num 346) 2026-02-25T08:45:08.858 INFO:tasks.rados.rados.0.trial023.stdout:739: delete oid 12 current snap is 99 2026-02-25T08:45:08.880 INFO:tasks.rados.rados.0.trial023.stdout:737: done (2 left) 2026-02-25T08:45:08.880 INFO:tasks.rados.rados.0.trial023.stdout:739: done (1 left) 2026-02-25T08:45:08.880 INFO:tasks.rados.rados.0.trial023.stdout:740: write (excl) oid 19 current snap is 99 2026-02-25T08:45:08.881 INFO:tasks.rados.rados.0.trial023.stdout:740: seq_num 348 ranges {648836=622537,1981263=493111,3107319=741517} 2026-02-25T08:45:08.885 INFO:tasks.rados.rados.0.trial023.stdout:740: writing trial02342524-19 from 648836 to 1271373 tid 1 2026-02-25T08:45:08.889 INFO:tasks.rados.rados.0.trial023.stdout:740: writing trial02342524-19 from 1981263 to 2474374 tid 2 2026-02-25T08:45:08.895 INFO:tasks.rados.rados.0.trial023.stdout:740: writing trial02342524-19 from 3107319 to 3848836 tid 3 2026-02-25T08:45:08.895 INFO:tasks.rados.rados.0.trial023.stdout:741: delete oid 9 current snap is 99 2026-02-25T08:45:08.895 INFO:tasks.rados.rados.0.trial023.stdout:738: expect (ObjNum 346 snap 98 seq_num 346) 2026-02-25T08:45:08.949 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:09.018 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1693343 2026-02-25T08:45:09.049 INFO:tasks.rados.rados.0.trial023.stdout:740: finishing write tid 1 to trial02342524-19 2026-02-25T08:45:09.049 INFO:tasks.rados.rados.0.trial023.stdout:740: finishing write tid 2 to trial02342524-19 2026-02-25T08:45:09.049 INFO:tasks.rados.rados.0.trial023.stdout:740: finishing write tid 3 to trial02342524-19 2026-02-25T08:45:09.050 INFO:tasks.rados.rados.0.trial023.stdout:740: finishing write tid 4 to trial02342524-19 2026-02-25T08:45:09.050 INFO:tasks.rados.rados.0.trial023.stdout:738: done (2 left) 2026-02-25T08:45:09.050 INFO:tasks.rados.rados.0.trial023.stdout:741: done (1 left) 2026-02-25T08:45:09.050 INFO:tasks.rados.rados.0.trial023.stdout:742: write (excl) oid 13 current snap is 99 2026-02-25T08:45:09.050 INFO:tasks.rados.rados.0.trial023.stdout:740: finishing write tid 5 to trial02342524-19 2026-02-25T08:45:09.050 INFO:tasks.rados.rados.0.trial023.stdout:740: oid 19 updating version 0 to 133 2026-02-25T08:45:09.050 INFO:tasks.rados.rados.0.trial023.stdout:740: oid 19 version 133 is already newer than 130 2026-02-25T08:45:09.050 INFO:tasks.rados.rados.0.trial023.stdout:740: oid 19 version 133 is already newer than 133 2026-02-25T08:45:09.050 INFO:tasks.rados.rados.0.trial023.stdout:740: oid 19 version 133 is already newer than 133 2026-02-25T08:45:09.050 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 19 v 133 (ObjNum 348 snap 99 seq_num 348) dirty exists 2026-02-25T08:45:09.050 INFO:tasks.rados.rados.0.trial023.stdout:740: left oid 19 (ObjNum 348 snap 99 seq_num 348) 2026-02-25T08:45:09.050 INFO:tasks.rados.rados.0.trial023.stdout:742: seq_num 350 ranges {682450=489388,1814580=496780,2878518=788028} 2026-02-25T08:45:09.053 INFO:tasks.rados.rados.0.trial023.stdout:742: writing trial02342524-13 from 682450 to 1171838 tid 1 2026-02-25T08:45:09.058 INFO:tasks.rados.rados.0.trial023.stdout:742: writing trial02342524-13 from 1814580 to 2311360 tid 2 2026-02-25T08:45:09.063 INFO:tasks.rados.rados.0.trial023.stdout:742: writing trial02342524-13 from 2878518 to 3666546 tid 3 2026-02-25T08:45:09.063 INFO:tasks.rados.rados.0.trial023.stdout:740: done (1 left) 2026-02-25T08:45:09.063 INFO:tasks.rados.rados.0.trial023.stdout:743: delete oid 19 current snap is 99 2026-02-25T08:45:09.078 INFO:tasks.rados.rados.0.trial023.stdout:743: done (1 left) 2026-02-25T08:45:09.078 INFO:tasks.rados.rados.0.trial023.stdout:744: write oid 3 current snap is 99 2026-02-25T08:45:09.078 INFO:tasks.rados.rados.0.trial023.stdout:744: seq_num 352 ranges {716064=639886,1815191=500449,2817011=526925} 2026-02-25T08:45:09.083 INFO:tasks.rados.rados.0.trial023.stdout:744: writing trial02342524-3 from 716064 to 1355950 tid 1 2026-02-25T08:45:09.087 INFO:tasks.rados.rados.0.trial023.stdout:744: writing trial02342524-3 from 1815191 to 2315640 tid 2 2026-02-25T08:45:09.091 INFO:tasks.rados.rados.0.trial023.stdout:744: writing trial02342524-3 from 2817011 to 3343936 tid 3 2026-02-25T08:45:09.091 INFO:tasks.rados.rados.0.trial023.stdout:745: read oid 9 snap -1 2026-02-25T08:45:09.091 INFO:tasks.rados.rados.0.trial023.stdout:745: expect deleted 2026-02-25T08:45:09.091 INFO:tasks.rados.rados.0.trial023.stdout:746: write oid 19 current snap is 99 2026-02-25T08:45:09.092 INFO:tasks.rados.rados.0.trial023.stdout:742: finishing write tid 1 to trial02342524-13 2026-02-25T08:45:09.092 INFO:tasks.rados.rados.0.trial023.stdout:746: seq_num 353 ranges {732871=715135,1873673=644107,3244434=688437} 2026-02-25T08:45:09.096 INFO:tasks.rados.rados.0.trial023.stdout:746: writing trial02342524-19 from 732871 to 1448006 tid 1 2026-02-25T08:45:09.101 INFO:tasks.rados.rados.0.trial023.stdout:746: writing trial02342524-19 from 1873673 to 2517780 tid 2 2026-02-25T08:45:09.107 INFO:tasks.rados.rados.0.trial023.stdout:746: writing trial02342524-19 from 3244434 to 3932871 tid 3 2026-02-25T08:45:09.107 INFO:tasks.rados.rados.0.trial023.stdout:747: write oid 28 current snap is 99 2026-02-25T08:45:09.108 INFO:tasks.rados.rados.0.trial023.stdout:742: finishing write tid 2 to trial02342524-13 2026-02-25T08:45:09.108 INFO:tasks.rados.rados.0.trial023.stdout:747: seq_num 354 ranges {749678=790384,2332155=504118,3271857=549469} 2026-02-25T08:45:09.113 INFO:tasks.rados.rados.0.trial023.stdout:747: writing trial02342524-28 from 749678 to 1540062 tid 1 2026-02-25T08:45:09.118 INFO:tasks.rados.rados.0.trial023.stdout:747: writing trial02342524-28 from 2332155 to 2836273 tid 2 2026-02-25T08:45:09.122 INFO:tasks.rados.rados.0.trial023.stdout:747: writing trial02342524-28 from 3271857 to 3821326 tid 3 2026-02-25T08:45:09.122 INFO:tasks.rados.rados.0.trial023.stdout:748: read oid 17 snap 97 2026-02-25T08:45:09.123 INFO:tasks.rados.rados.0.trial023.stdout:748: expect deleted 2026-02-25T08:45:09.123 INFO:tasks.rados.rados.0.trial023.stdout:749: delete oid 29 current snap is 99 2026-02-25T08:45:09.123 INFO:tasks.rados.rados.0.trial023.stdout:742: finishing write tid 3 to trial02342524-13 2026-02-25T08:45:09.123 INFO:tasks.rados.rados.0.trial023.stdout:742: finishing write tid 4 to trial02342524-13 2026-02-25T08:45:09.123 INFO:tasks.rados.rados.0.trial023.stdout:742: finishing write tid 5 to trial02342524-13 2026-02-25T08:45:09.123 INFO:tasks.rados.rados.0.trial023.stdout:742: oid 13 updating version 0 to 157 2026-02-25T08:45:09.123 INFO:tasks.rados.rados.0.trial023.stdout:742: oid 13 version 157 is already newer than 157 2026-02-25T08:45:09.123 INFO:tasks.rados.rados.0.trial023.stdout:742: oid 13 version 157 is already newer than 154 2026-02-25T08:45:09.123 INFO:tasks.rados.rados.0.trial023.stdout:742: oid 13 version 157 is already newer than 157 2026-02-25T08:45:09.123 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 13 v 157 (ObjNum 350 snap 99 seq_num 350) dirty exists 2026-02-25T08:45:09.123 INFO:tasks.rados.rados.0.trial023.stdout:742: left oid 13 (ObjNum 350 snap 99 seq_num 350) 2026-02-25T08:45:09.124 INFO:tasks.rados.rados.0.trial023.stdout:744: finishing write tid 1 to trial02342524-3 2026-02-25T08:45:09.124 INFO:tasks.rados.rados.0.trial023.stdout:746: finishing write tid 1 to trial02342524-19 2026-02-25T08:45:09.125 INFO:tasks.rados.rados.0.trial023.stdout:744: finishing write tid 2 to trial02342524-3 2026-02-25T08:45:09.127 INFO:tasks.rados.rados.0.trial023.stdout:742: done (6 left) 2026-02-25T08:45:09.127 INFO:tasks.rados.rados.0.trial023.stdout:745: done (5 left) 2026-02-25T08:45:09.127 INFO:tasks.rados.rados.0.trial023.stdout:749: done (4 left) 2026-02-25T08:45:09.127 INFO:tasks.rados.rados.0.trial023.stdout:750: write oid 21 current snap is 99 2026-02-25T08:45:09.127 INFO:tasks.rados.rados.0.trial023.stdout:750: seq_num 356 ranges {783292=540882,1932766=507787,3210350=572013} 2026-02-25T08:45:09.133 INFO:tasks.rados.rados.0.trial023.stdout:750: writing trial02342524-21 from 783292 to 1324174 tid 1 2026-02-25T08:45:09.137 INFO:tasks.rados.rados.0.trial023.stdout:750: writing trial02342524-21 from 1932766 to 2440553 tid 2 2026-02-25T08:45:09.142 INFO:tasks.rados.rados.0.trial023.stdout:750: writing trial02342524-21 from 3210350 to 3782363 tid 3 2026-02-25T08:45:09.142 INFO:tasks.rados.rados.0.trial023.stdout:751: snap_remove snap 98 2026-02-25T08:45:09.564 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~209s 2026-02-25T08:45:09.765 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:45:09.829 INFO:tasks.rados.rados.0.trial023.stdout:751: done (5 left) 2026-02-25T08:45:09.829 INFO:tasks.rados.rados.0.trial023.stdout:752: delete oid 5 current snap is 99 2026-02-25T08:45:09.829 INFO:tasks.rados.rados.0.trial023.stdout:744: finishing write tid 3 to trial02342524-3 2026-02-25T08:45:09.829 INFO:tasks.rados.rados.0.trial023.stdout:746: finishing write tid 2 to trial02342524-19 2026-02-25T08:45:09.829 INFO:tasks.rados.rados.0.trial023.stdout:744: finishing write tid 4 to trial02342524-3 2026-02-25T08:45:09.829 INFO:tasks.rados.rados.0.trial023.stdout:744: finishing write tid 5 to trial02342524-3 2026-02-25T08:45:09.829 INFO:tasks.rados.rados.0.trial023.stdout:744: oid 3 updating version 0 to 187 2026-02-25T08:45:09.829 INFO:tasks.rados.rados.0.trial023.stdout:744: oid 3 version 187 is already newer than 187 2026-02-25T08:45:09.830 INFO:tasks.rados.rados.0.trial023.stdout:744: oid 3 version 187 is already newer than 187 2026-02-25T08:45:09.830 INFO:tasks.rados.rados.0.trial023.stdout:744: oid 3 version 187 is already newer than 187 2026-02-25T08:45:09.830 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 3 v 187 (ObjNum 352 snap 99 seq_num 352) dirty exists 2026-02-25T08:45:09.830 INFO:tasks.rados.rados.0.trial023.stdout:744: left oid 3 (ObjNum 352 snap 99 seq_num 352) 2026-02-25T08:45:09.830 INFO:tasks.rados.rados.0.trial023.stdout:746: finishing write tid 3 to trial02342524-19 2026-02-25T08:45:09.830 INFO:tasks.rados.rados.0.trial023.stdout:747: finishing write tid 1 to trial02342524-28 2026-02-25T08:45:09.830 INFO:tasks.rados.rados.0.trial023.stdout:746: finishing write tid 4 to trial02342524-19 2026-02-25T08:45:09.830 INFO:tasks.rados.rados.0.trial023.stdout:746: finishing write tid 5 to trial02342524-19 2026-02-25T08:45:09.830 INFO:tasks.rados.rados.0.trial023.stdout:746: oid 19 updating version 0 to 138 2026-02-25T08:45:09.830 INFO:tasks.rados.rados.0.trial023.stdout:746: oid 19 version 138 is already newer than 138 2026-02-25T08:45:09.830 INFO:tasks.rados.rados.0.trial023.stdout:746: oid 19 version 138 is already newer than 138 2026-02-25T08:45:09.830 INFO:tasks.rados.rados.0.trial023.stdout:746: oid 19 version 138 is already newer than 138 2026-02-25T08:45:09.831 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 19 v 138 (ObjNum 353 snap 99 seq_num 353) dirty exists 2026-02-25T08:45:09.831 INFO:tasks.rados.rados.0.trial023.stdout:746: left oid 19 (ObjNum 353 snap 99 seq_num 353) 2026-02-25T08:45:09.831 INFO:tasks.rados.rados.0.trial023.stdout:747: finishing write tid 2 to trial02342524-28 2026-02-25T08:45:09.831 INFO:tasks.rados.rados.0.trial023.stdout:747: finishing write tid 3 to trial02342524-28 2026-02-25T08:45:09.831 INFO:tasks.rados.rados.0.trial023.stdout:747: finishing write tid 4 to trial02342524-28 2026-02-25T08:45:09.831 INFO:tasks.rados.rados.0.trial023.stdout:747: finishing write tid 5 to trial02342524-28 2026-02-25T08:45:09.831 INFO:tasks.rados.rados.0.trial023.stdout:747: oid 28 updating version 0 to 133 2026-02-25T08:45:09.831 INFO:tasks.rados.rados.0.trial023.stdout:747: oid 28 version 133 is already newer than 133 2026-02-25T08:45:09.831 INFO:tasks.rados.rados.0.trial023.stdout:747: oid 28 version 133 is already newer than 133 2026-02-25T08:45:09.831 INFO:tasks.rados.rados.0.trial023.stdout:747: oid 28 version 133 is already newer than 133 2026-02-25T08:45:09.831 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 28 v 133 (ObjNum 354 snap 99 seq_num 354) dirty exists 2026-02-25T08:45:09.831 INFO:tasks.rados.rados.0.trial023.stdout:747: left oid 28 (ObjNum 354 snap 99 seq_num 354) 2026-02-25T08:45:09.831 INFO:tasks.rados.rados.0.trial023.stdout:750: finishing write tid 1 to trial02342524-21 2026-02-25T08:45:09.832 INFO:tasks.rados.rados.0.trial023.stdout:750: finishing write tid 2 to trial02342524-21 2026-02-25T08:45:09.832 INFO:tasks.rados.rados.0.trial023.stdout:750: finishing write tid 3 to trial02342524-21 2026-02-25T08:45:09.832 INFO:tasks.rados.rados.0.trial023.stdout:750: finishing write tid 4 to trial02342524-21 2026-02-25T08:45:09.832 INFO:tasks.rados.rados.0.trial023.stdout:750: finishing write tid 5 to trial02342524-21 2026-02-25T08:45:09.832 INFO:tasks.rados.rados.0.trial023.stdout:750: oid 21 updating version 0 to 143 2026-02-25T08:45:09.832 INFO:tasks.rados.rados.0.trial023.stdout:750: oid 21 version 143 is already newer than 143 2026-02-25T08:45:09.832 INFO:tasks.rados.rados.0.trial023.stdout:750: oid 21 version 143 is already newer than 143 2026-02-25T08:45:09.832 INFO:tasks.rados.rados.0.trial023.stdout:750: oid 21 version 143 is already newer than 143 2026-02-25T08:45:09.832 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 21 v 143 (ObjNum 356 snap 99 seq_num 356) dirty exists 2026-02-25T08:45:09.832 INFO:tasks.rados.rados.0.trial023.stdout:750: left oid 21 (ObjNum 356 snap 99 seq_num 356) 2026-02-25T08:45:09.839 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:45:09.842 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:45:09.843 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:45:09.871 INFO:tasks.rados.rados.0.trial023.stdout:744: done (5 left) 2026-02-25T08:45:09.871 INFO:tasks.rados.rados.0.trial023.stdout:746: done (4 left) 2026-02-25T08:45:09.871 INFO:tasks.rados.rados.0.trial023.stdout:747: done (3 left) 2026-02-25T08:45:09.872 INFO:tasks.rados.rados.0.trial023.stdout:748: done (2 left) 2026-02-25T08:45:09.872 INFO:tasks.rados.rados.0.trial023.stdout:750: done (1 left) 2026-02-25T08:45:09.872 INFO:tasks.rados.rados.0.trial023.stdout:752: done (0 left) 2026-02-25T08:45:09.872 INFO:tasks.rados.rados.0.trial023.stdout:753: snap_remove snap 96 2026-02-25T08:45:09.918 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:09.996 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:09.993+0000 7fb92175c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:09.999 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:09.997+0000 7fb92175c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:10.015 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:45:10.042 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:10.043 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:10.863 INFO:tasks.rados.rados.0.trial023.stdout:753: done (0 left) 2026-02-25T08:45:10.863 INFO:tasks.rados.rados.0.trial023.stdout:754: delete oid 19 current snap is 99 2026-02-25T08:45:10.890 INFO:tasks.rados.rados.0.trial023.stdout:754: done (0 left) 2026-02-25T08:45:10.890 INFO:tasks.rados.rados.0.trial023.stdout:755: snap_remove snap 97 2026-02-25T08:45:11.866 INFO:tasks.rados.rados.0.trial023.stdout:755: done (0 left) 2026-02-25T08:45:11.866 INFO:tasks.rados.rados.0.trial023.stdout:756: snap_remove snap 95 2026-02-25T08:45:12.044 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:12.121 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:12.118+0000 7f7f656cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:12.123 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:12.121+0000 7f7f656cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:12.139 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:45:12.166 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:12.166 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:12.869 INFO:tasks.rados.rados.0.trial023.stdout:756: done (0 left) 2026-02-25T08:45:12.870 INFO:tasks.rados.rados.0.trial023.stdout:757: delete oid 46 current snap is 99 2026-02-25T08:45:12.899 INFO:tasks.rados.rados.0.trial023.stdout:757: done (0 left) 2026-02-25T08:45:12.899 INFO:tasks.rados.rados.0.trial023.stdout:758: write (excl) oid 37 current snap is 99 2026-02-25T08:45:12.899 INFO:tasks.rados.rados.0.trial023.stdout:758: seq_num 360 ranges {450520=558231,1483047=515125} 2026-02-25T08:45:12.903 INFO:tasks.rados.rados.0.trial023.stdout:758: writing trial02342524-37 from 450520 to 1008751 tid 1 2026-02-25T08:45:12.907 INFO:tasks.rados.rados.0.trial023.stdout:758: writing trial02342524-37 from 1483047 to 1998172 tid 2 2026-02-25T08:45:12.907 INFO:tasks.rados.rados.0.trial023.stdout:759: snap_remove snap 86 2026-02-25T08:45:13.872 INFO:tasks.rados.rados.0.trial023.stdout:759: done (1 left) 2026-02-25T08:45:13.872 INFO:tasks.rados.rados.0.trial023.stdout:760: read oid 16 snap -1 2026-02-25T08:45:13.872 INFO:tasks.rados.rados.0.trial023.stdout:760: expect (ObjNum 310 snap 91 seq_num 310) 2026-02-25T08:45:13.872 INFO:tasks.rados.rados.0.trial023.stdout:761: read oid 12 snap -1 2026-02-25T08:45:13.872 INFO:tasks.rados.rados.0.trial023.stdout:761: expect deleted 2026-02-25T08:45:13.873 INFO:tasks.rados.rados.0.trial023.stdout:762: write (excl) oid 36 current snap is 99 2026-02-25T08:45:13.873 INFO:tasks.rados.rados.0.trial023.stdout:758: finishing write tid 1 to trial02342524-37 2026-02-25T08:45:13.873 INFO:tasks.rados.rados.0.trial023.stdout:762: seq_num 361 ranges {467327=633480,1825176=242151} 2026-02-25T08:45:13.877 INFO:tasks.rados.rados.0.trial023.stdout:762: writing trial02342524-36 from 467327 to 1100807 tid 1 2026-02-25T08:45:13.881 INFO:tasks.rados.rados.0.trial023.stdout:762: writing trial02342524-36 from 1825176 to 2067327 tid 2 2026-02-25T08:45:13.881 INFO:tasks.rados.rados.0.trial023.stdout:763: write oid 39 current snap is 99 2026-02-25T08:45:13.881 INFO:tasks.rados.rados.0.trial023.stdout:758: finishing write tid 2 to trial02342524-37 2026-02-25T08:45:13.881 INFO:tasks.rados.rados.0.trial023.stdout:758: finishing write tid 3 to trial02342524-37 2026-02-25T08:45:13.881 INFO:tasks.rados.rados.0.trial023.stdout:758: finishing write tid 4 to trial02342524-37 2026-02-25T08:45:13.881 INFO:tasks.rados.rados.0.trial023.stdout:758: oid 37 updating version 0 to 158 2026-02-25T08:45:13.881 INFO:tasks.rados.rados.0.trial023.stdout:758: oid 37 updating version 158 to 160 2026-02-25T08:45:13.881 INFO:tasks.rados.rados.0.trial023.stdout:758: oid 37 version 160 is already newer than 160 2026-02-25T08:45:13.881 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 37 v 160 (ObjNum 360 snap 99 seq_num 360) dirty exists 2026-02-25T08:45:13.881 INFO:tasks.rados.rados.0.trial023.stdout:758: left oid 37 (ObjNum 360 snap 99 seq_num 360) 2026-02-25T08:45:13.881 INFO:tasks.rados.rados.0.trial023.stdout:763: seq_num 362 ranges {484134=708729,1883658=200476} 2026-02-25T08:45:13.886 INFO:tasks.rados.rados.0.trial023.stdout:763: writing trial02342524-39 from 484134 to 1192863 tid 1 2026-02-25T08:45:13.889 INFO:tasks.rados.rados.0.trial023.stdout:763: writing trial02342524-39 from 1883658 to 2084134 tid 2 2026-02-25T08:45:13.889 INFO:tasks.rados.rados.0.trial023.stdout:758: done (4 left) 2026-02-25T08:45:13.889 INFO:tasks.rados.rados.0.trial023.stdout:764: read oid 19 snap -1 2026-02-25T08:45:13.889 INFO:tasks.rados.rados.0.trial023.stdout:764: expect deleted 2026-02-25T08:45:13.890 INFO:tasks.rados.rados.0.trial023.stdout:765: write oid 41 current snap is 99 2026-02-25T08:45:13.890 INFO:tasks.rados.rados.0.trial023.stdout:765: seq_num 363 ranges {500941=783978,1942140=158801} 2026-02-25T08:45:13.894 INFO:tasks.rados.rados.0.trial023.stdout:765: writing trial02342524-41 from 500941 to 1284919 tid 1 2026-02-25T08:45:13.897 INFO:tasks.rados.rados.0.trial023.stdout:765: writing trial02342524-41 from 1942140 to 2100941 tid 2 2026-02-25T08:45:13.897 INFO:tasks.rados.rados.0.trial023.stdout:766: delete oid 24 current snap is 99 2026-02-25T08:45:13.941 INFO:tasks.rados.rados.0.trial023.stdout:765: finishing write tid 1 to trial02342524-41 2026-02-25T08:45:13.949 INFO:tasks.rados.rados.0.trial023.stdout:765: finishing write tid 2 to trial02342524-41 2026-02-25T08:45:13.950 INFO:tasks.rados.rados.0.trial023.stdout:762: finishing write tid 1 to trial02342524-36 2026-02-25T08:45:13.951 INFO:tasks.rados.rados.0.trial023.stdout:760: expect (ObjNum 310 snap 91 seq_num 310) 2026-02-25T08:45:13.951 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:14.032 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 304474 2026-02-25T08:45:14.077 INFO:tasks.rados.rados.0.trial023.stdout:762: finishing write tid 2 to trial02342524-36 2026-02-25T08:45:14.077 INFO:tasks.rados.rados.0.trial023.stdout:765: finishing write tid 3 to trial02342524-41 2026-02-25T08:45:14.077 INFO:tasks.rados.rados.0.trial023.stdout:765: finishing write tid 4 to trial02342524-41 2026-02-25T08:45:14.077 INFO:tasks.rados.rados.0.trial023.stdout:765: oid 41 updating version 0 to 119 2026-02-25T08:45:14.077 INFO:tasks.rados.rados.0.trial023.stdout:765: oid 41 version 119 is already newer than 119 2026-02-25T08:45:14.077 INFO:tasks.rados.rados.0.trial023.stdout:765: oid 41 version 119 is already newer than 119 2026-02-25T08:45:14.077 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 41 v 119 (ObjNum 363 snap 99 seq_num 363) dirty exists 2026-02-25T08:45:14.077 INFO:tasks.rados.rados.0.trial023.stdout:765: left oid 41 (ObjNum 363 snap 99 seq_num 363) 2026-02-25T08:45:14.077 INFO:tasks.rados.rados.0.trial023.stdout:762: finishing write tid 3 to trial02342524-36 2026-02-25T08:45:14.077 INFO:tasks.rados.rados.0.trial023.stdout:762: finishing write tid 4 to trial02342524-36 2026-02-25T08:45:14.077 INFO:tasks.rados.rados.0.trial023.stdout:762: oid 36 updating version 0 to 156 2026-02-25T08:45:14.077 INFO:tasks.rados.rados.0.trial023.stdout:762: oid 36 version 156 is already newer than 154 2026-02-25T08:45:14.077 INFO:tasks.rados.rados.0.trial023.stdout:762: oid 36 version 156 is already newer than 156 2026-02-25T08:45:14.077 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 36 v 156 (ObjNum 361 snap 99 seq_num 361) dirty exists 2026-02-25T08:45:14.077 INFO:tasks.rados.rados.0.trial023.stdout:762: left oid 36 (ObjNum 361 snap 99 seq_num 361) 2026-02-25T08:45:14.077 INFO:tasks.rados.rados.0.trial023.stdout:763: finishing write tid 1 to trial02342524-39 2026-02-25T08:45:14.077 INFO:tasks.rados.rados.0.trial023.stdout:763: finishing write tid 2 to trial02342524-39 2026-02-25T08:45:14.077 INFO:tasks.rados.rados.0.trial023.stdout:763: finishing write tid 3 to trial02342524-39 2026-02-25T08:45:14.077 INFO:tasks.rados.rados.0.trial023.stdout:763: finishing write tid 4 to trial02342524-39 2026-02-25T08:45:14.077 INFO:tasks.rados.rados.0.trial023.stdout:763: oid 39 updating version 0 to 144 2026-02-25T08:45:14.077 INFO:tasks.rados.rados.0.trial023.stdout:763: oid 39 version 144 is already newer than 144 2026-02-25T08:45:14.077 INFO:tasks.rados.rados.0.trial023.stdout:763: oid 39 version 144 is already newer than 144 2026-02-25T08:45:14.078 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 39 v 144 (ObjNum 362 snap 99 seq_num 362) dirty exists 2026-02-25T08:45:14.078 INFO:tasks.rados.rados.0.trial023.stdout:763: left oid 39 (ObjNum 362 snap 99 seq_num 362) 2026-02-25T08:45:14.078 INFO:tasks.rados.rados.0.trial023.stdout:760: done (6 left) 2026-02-25T08:45:14.078 INFO:tasks.rados.rados.0.trial023.stdout:761: done (5 left) 2026-02-25T08:45:14.078 INFO:tasks.rados.rados.0.trial023.stdout:762: done (4 left) 2026-02-25T08:45:14.078 INFO:tasks.rados.rados.0.trial023.stdout:763: done (3 left) 2026-02-25T08:45:14.078 INFO:tasks.rados.rados.0.trial023.stdout:764: done (2 left) 2026-02-25T08:45:14.078 INFO:tasks.rados.rados.0.trial023.stdout:765: done (1 left) 2026-02-25T08:45:14.078 INFO:tasks.rados.rados.0.trial023.stdout:766: done (0 left) 2026-02-25T08:45:14.078 INFO:tasks.rados.rados.0.trial023.stdout:767: write oid 41 current snap is 99 2026-02-25T08:45:14.078 INFO:tasks.rados.rados.0.trial023.stdout:767: seq_num 365 ranges {534555=650829,1659104=475451} 2026-02-25T08:45:14.082 INFO:tasks.rados.rados.0.trial023.stdout:767: writing trial02342524-41 from 534555 to 1185384 tid 1 2026-02-25T08:45:14.086 INFO:tasks.rados.rados.0.trial023.stdout:767: writing trial02342524-41 from 1659104 to 2134555 tid 2 2026-02-25T08:45:14.086 INFO:tasks.rados.rados.0.trial023.stdout:768: delete oid 32 current snap is 99 2026-02-25T08:45:14.102 INFO:tasks.rados.rados.0.trial023.stdout:767: finishing write tid 1 to trial02342524-41 2026-02-25T08:45:14.110 INFO:tasks.rados.rados.0.trial023.stdout:767: finishing write tid 2 to trial02342524-41 2026-02-25T08:45:14.116 INFO:tasks.rados.rados.0.trial023.stdout:767: finishing write tid 3 to trial02342524-41 2026-02-25T08:45:14.118 INFO:tasks.rados.rados.0.trial023.stdout:767: finishing write tid 4 to trial02342524-41 2026-02-25T08:45:14.118 INFO:tasks.rados.rados.0.trial023.stdout:767: oid 41 updating version 0 to 124 2026-02-25T08:45:14.118 INFO:tasks.rados.rados.0.trial023.stdout:767: oid 41 version 124 is already newer than 124 2026-02-25T08:45:14.118 INFO:tasks.rados.rados.0.trial023.stdout:767: oid 41 version 124 is already newer than 124 2026-02-25T08:45:14.118 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 41 v 124 (ObjNum 365 snap 99 seq_num 365) dirty exists 2026-02-25T08:45:14.118 INFO:tasks.rados.rados.0.trial023.stdout:767: left oid 41 (ObjNum 365 snap 99 seq_num 365) 2026-02-25T08:45:14.127 INFO:tasks.rados.rados.0.trial023.stdout:767: done (1 left) 2026-02-25T08:45:14.127 INFO:tasks.rados.rados.0.trial023.stdout:768: done (0 left) 2026-02-25T08:45:14.127 INFO:tasks.rados.rados.0.trial023.stdout:769: snap_create 2026-02-25T08:45:14.167 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:14.244 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:14.241+0000 7fc74b81a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:14.246 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:14.244+0000 7fc74b81a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:14.261 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:45:14.287 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:14.288 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:14.844 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:45:14.875 INFO:tasks.rados.rados.0.trial023.stdout:769: done (0 left) 2026-02-25T08:45:14.875 INFO:tasks.rados.rados.0.trial023.stdout:770: write (excl) oid 41 current snap is 100 2026-02-25T08:45:14.876 INFO:tasks.rados.rados.0.trial023.stdout:770: seq_num 367 ranges {568169=401327,1376068=669790} 2026-02-25T08:45:14.879 INFO:tasks.rados.rados.0.trial023.stdout:770: writing trial02342524-41 from 568169 to 969496 tid 1 2026-02-25T08:45:14.884 INFO:tasks.rados.rados.0.trial023.stdout:770: writing trial02342524-41 from 1376068 to 2045858 tid 2 2026-02-25T08:45:14.884 INFO:tasks.rados.rados.0.trial023.stdout:771: read oid 17 snap -1 2026-02-25T08:45:14.884 INFO:tasks.rados.rados.0.trial023.stdout:771: expect deleted 2026-02-25T08:45:14.884 INFO:tasks.rados.rados.0.trial023.stdout:772: snap_remove snap 99 2026-02-25T08:45:14.916 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:45:14.919 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:45:14.920 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:45:15.067 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~215s 2026-02-25T08:45:15.880 INFO:tasks.rados.rados.0.trial023.stdout:772: done (2 left) 2026-02-25T08:45:15.880 INFO:tasks.rados.rados.0.trial023.stdout:773: write oid 7 current snap is 100 2026-02-25T08:45:15.880 INFO:tasks.rados.rados.0.trial023.stdout:773: seq_num 368 ranges {584976=476576,1834550=350426} 2026-02-25T08:45:15.884 INFO:tasks.rados.rados.0.trial023.stdout:773: writing trial02342524-7 from 584976 to 1061552 tid 1 2026-02-25T08:45:15.888 INFO:tasks.rados.rados.0.trial023.stdout:773: writing trial02342524-7 from 1834550 to 2184976 tid 2 2026-02-25T08:45:15.888 INFO:tasks.rados.rados.0.trial023.stdout:774: write oid 15 current snap is 100 2026-02-25T08:45:15.888 INFO:tasks.rados.rados.0.trial023.stdout:774: seq_num 369 ranges {601783=551825,1776679=425104} 2026-02-25T08:45:15.892 INFO:tasks.rados.rados.0.trial023.stdout:774: writing trial02342524-15 from 601783 to 1153608 tid 1 2026-02-25T08:45:15.896 INFO:tasks.rados.rados.0.trial023.stdout:774: writing trial02342524-15 from 1776679 to 2201783 tid 2 2026-02-25T08:45:15.896 INFO:tasks.rados.rados.0.trial023.stdout:775: snap_remove snap 85 2026-02-25T08:45:16.289 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:16.366 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:16.363+0000 7f4bb23e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:16.369 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:16.367+0000 7f4bb23e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:16.384 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:45:16.411 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:16.412 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:16.886 INFO:tasks.rados.rados.0.trial023.stdout:775: done (4 left) 2026-02-25T08:45:16.886 INFO:tasks.rados.rados.0.trial023.stdout:776: read oid 13 snap -1 2026-02-25T08:45:16.886 INFO:tasks.rados.rados.0.trial023.stdout:776: expect (ObjNum 350 snap 99 seq_num 350) 2026-02-25T08:45:16.886 INFO:tasks.rados.rados.0.trial023.stdout:777: write (excl) oid 6 current snap is 100 2026-02-25T08:45:16.886 INFO:tasks.rados.rados.0.trial023.stdout:770: finishing write tid 1 to trial02342524-41 2026-02-25T08:45:16.886 INFO:tasks.rados.rados.0.trial023.stdout:770: finishing write tid 2 to trial02342524-41 2026-02-25T08:45:16.886 INFO:tasks.rados.rados.0.trial023.stdout:777: seq_num 370 ranges {618590=627074,1835161=383429} 2026-02-25T08:45:16.890 INFO:tasks.rados.rados.0.trial023.stdout:777: writing trial02342524-6 from 618590 to 1245664 tid 1 2026-02-25T08:45:16.894 INFO:tasks.rados.rados.0.trial023.stdout:777: writing trial02342524-6 from 1835161 to 2218590 tid 2 2026-02-25T08:45:16.894 INFO:tasks.rados.rados.0.trial023.stdout:771: done (5 left) 2026-02-25T08:45:16.895 INFO:tasks.rados.rados.0.trial023.stdout:778: write (excl) oid 46 current snap is 100 2026-02-25T08:45:16.895 INFO:tasks.rados.rados.0.trial023.stdout:770: finishing write tid 3 to trial02342524-41 2026-02-25T08:45:16.895 INFO:tasks.rados.rados.0.trial023.stdout:770: finishing write tid 4 to trial02342524-41 2026-02-25T08:45:16.895 INFO:tasks.rados.rados.0.trial023.stdout:770: oid 41 updating version 0 to 127 2026-02-25T08:45:16.895 INFO:tasks.rados.rados.0.trial023.stdout:770: oid 41 updating version 127 to 129 2026-02-25T08:45:16.895 INFO:tasks.rados.rados.0.trial023.stdout:770: oid 41 version 129 is already newer than 129 2026-02-25T08:45:16.895 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 41 v 129 (ObjNum 367 snap 100 seq_num 367) dirty exists 2026-02-25T08:45:16.895 INFO:tasks.rados.rados.0.trial023.stdout:770: left oid 41 (ObjNum 367 snap 100 seq_num 367) 2026-02-25T08:45:16.895 INFO:tasks.rados.rados.0.trial023.stdout:778: seq_num 371 ranges {635397=702323,1893643=341754} 2026-02-25T08:45:16.899 INFO:tasks.rados.rados.0.trial023.stdout:778: writing trial02342524-46 from 635397 to 1337720 tid 1 2026-02-25T08:45:16.904 INFO:tasks.rados.rados.0.trial023.stdout:778: writing trial02342524-46 from 1893643 to 2235397 tid 2 2026-02-25T08:45:16.904 INFO:tasks.rados.rados.0.trial023.stdout:770: done (5 left) 2026-02-25T08:45:16.905 INFO:tasks.rados.rados.0.trial023.stdout:779: read oid 36 snap -1 2026-02-25T08:45:16.905 INFO:tasks.rados.rados.0.trial023.stdout:779: expect (ObjNum 361 snap 99 seq_num 361) 2026-02-25T08:45:16.905 INFO:tasks.rados.rados.0.trial023.stdout:780: read oid 33 snap -1 2026-02-25T08:45:16.905 INFO:tasks.rados.rados.0.trial023.stdout:780: expect (ObjNum 323 snap 93 seq_num 323) 2026-02-25T08:45:16.905 INFO:tasks.rados.rados.0.trial023.stdout:781: read oid 3 snap -1 2026-02-25T08:45:16.905 INFO:tasks.rados.rados.0.trial023.stdout:781: expect (ObjNum 352 snap 99 seq_num 352) 2026-02-25T08:45:16.905 INFO:tasks.rados.rados.0.trial023.stdout:782: delete oid 18 current snap is 100 2026-02-25T08:45:16.905 INFO:tasks.rados.rados.0.trial023.stdout:773: finishing write tid 1 to trial02342524-7 2026-02-25T08:45:16.905 INFO:tasks.rados.rados.0.trial023.stdout:773: finishing write tid 2 to trial02342524-7 2026-02-25T08:45:16.905 INFO:tasks.rados.rados.0.trial023.stdout:774: finishing write tid 1 to trial02342524-15 2026-02-25T08:45:16.905 INFO:tasks.rados.rados.0.trial023.stdout:773: finishing write tid 3 to trial02342524-7 2026-02-25T08:45:16.905 INFO:tasks.rados.rados.0.trial023.stdout:773: finishing write tid 4 to trial02342524-7 2026-02-25T08:45:16.906 INFO:tasks.rados.rados.0.trial023.stdout:773: oid 7 updating version 0 to 132 2026-02-25T08:45:16.906 INFO:tasks.rados.rados.0.trial023.stdout:773: oid 7 version 132 is already newer than 132 2026-02-25T08:45:16.906 INFO:tasks.rados.rados.0.trial023.stdout:773: oid 7 version 132 is already newer than 132 2026-02-25T08:45:16.906 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 7 v 132 (ObjNum 368 snap 100 seq_num 368) dirty exists 2026-02-25T08:45:16.906 INFO:tasks.rados.rados.0.trial023.stdout:773: left oid 7 (ObjNum 368 snap 100 seq_num 368) 2026-02-25T08:45:16.906 INFO:tasks.rados.rados.0.trial023.stdout:774: finishing write tid 2 to trial02342524-15 2026-02-25T08:45:16.906 INFO:tasks.rados.rados.0.trial023.stdout:774: finishing write tid 3 to trial02342524-15 2026-02-25T08:45:16.906 INFO:tasks.rados.rados.0.trial023.stdout:774: finishing write tid 4 to trial02342524-15 2026-02-25T08:45:16.906 INFO:tasks.rados.rados.0.trial023.stdout:774: oid 15 updating version 0 to 160 2026-02-25T08:45:16.906 INFO:tasks.rados.rados.0.trial023.stdout:774: oid 15 version 160 is already newer than 160 2026-02-25T08:45:16.906 INFO:tasks.rados.rados.0.trial023.stdout:774: oid 15 version 160 is already newer than 160 2026-02-25T08:45:16.906 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 15 v 160 (ObjNum 369 snap 100 seq_num 369) dirty exists 2026-02-25T08:45:16.907 INFO:tasks.rados.rados.0.trial023.stdout:774: left oid 15 (ObjNum 369 snap 100 seq_num 369) 2026-02-25T08:45:16.926 INFO:tasks.rados.rados.0.trial023.stdout:777: finishing write tid 1 to trial02342524-6 2026-02-25T08:45:16.939 INFO:tasks.rados.rados.0.trial023.stdout:780: expect (ObjNum 323 snap 93 seq_num 323) 2026-02-25T08:45:17.104 INFO:tasks.rados.rados.0.trial023.stdout:781: expect (ObjNum 352 snap 99 seq_num 352) 2026-02-25T08:45:17.150 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:17.195 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:17.242 INFO:tasks.rados.rados.0.trial023.stdout:778: finishing write tid 1 to trial02342524-46 2026-02-25T08:45:17.242 INFO:tasks.rados.rados.0.trial023.stdout:773: done (8 left) 2026-02-25T08:45:17.242 INFO:tasks.rados.rados.0.trial023.stdout:774: done (7 left) 2026-02-25T08:45:17.242 INFO:tasks.rados.rados.0.trial023.stdout:780: done (6 left) 2026-02-25T08:45:17.242 INFO:tasks.rados.rados.0.trial023.stdout:781: done (5 left) 2026-02-25T08:45:17.242 INFO:tasks.rados.rados.0.trial023.stdout:782: done (4 left) 2026-02-25T08:45:17.242 INFO:tasks.rados.rados.0.trial023.stdout:783: write (excl) oid 16 current snap is 100 2026-02-25T08:45:17.242 INFO:tasks.rados.rados.0.trial023.stdout:776: expect (ObjNum 350 snap 99 seq_num 350) 2026-02-25T08:45:17.261 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1776355 2026-02-25T08:45:17.307 INFO:tasks.rados.rados.0.trial023.stdout:777: finishing write tid 2 to trial02342524-6 2026-02-25T08:45:17.307 INFO:tasks.rados.rados.0.trial023.stdout:777: finishing write tid 3 to trial02342524-6 2026-02-25T08:45:17.307 INFO:tasks.rados.rados.0.trial023.stdout:777: finishing write tid 4 to trial02342524-6 2026-02-25T08:45:17.307 INFO:tasks.rados.rados.0.trial023.stdout:777: oid 6 updating version 0 to 49 2026-02-25T08:45:17.307 INFO:tasks.rados.rados.0.trial023.stdout:777: oid 6 version 49 is already newer than 47 2026-02-25T08:45:17.307 INFO:tasks.rados.rados.0.trial023.stdout:777: oid 6 version 49 is already newer than 49 2026-02-25T08:45:17.308 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 6 v 49 (ObjNum 370 snap 100 seq_num 370) dirty exists 2026-02-25T08:45:17.308 INFO:tasks.rados.rados.0.trial023.stdout:777: left oid 6 (ObjNum 370 snap 100 seq_num 370) 2026-02-25T08:45:17.308 INFO:tasks.rados.rados.0.trial023.stdout:778: finishing write tid 2 to trial02342524-46 2026-02-25T08:45:17.308 INFO:tasks.rados.rados.0.trial023.stdout:778: finishing write tid 3 to trial02342524-46 2026-02-25T08:45:17.308 INFO:tasks.rados.rados.0.trial023.stdout:778: finishing write tid 4 to trial02342524-46 2026-02-25T08:45:17.308 INFO:tasks.rados.rados.0.trial023.stdout:778: oid 46 updating version 0 to 225 2026-02-25T08:45:17.308 INFO:tasks.rados.rados.0.trial023.stdout:778: oid 46 updating version 225 to 227 2026-02-25T08:45:17.308 INFO:tasks.rados.rados.0.trial023.stdout:778: oid 46 version 227 is already newer than 227 2026-02-25T08:45:17.308 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 46 v 227 (ObjNum 371 snap 100 seq_num 371) dirty exists 2026-02-25T08:45:17.308 INFO:tasks.rados.rados.0.trial023.stdout:778: left oid 46 (ObjNum 371 snap 100 seq_num 371) 2026-02-25T08:45:17.308 INFO:tasks.rados.rados.0.trial023.stdout:779: expect (ObjNum 361 snap 99 seq_num 361) 2026-02-25T08:45:17.309 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:17.374 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1935385 2026-02-25T08:45:17.415 INFO:tasks.rados.rados.0.trial023.stdout:783: seq_num 373 ranges {669011=569174,2010607=258404} 2026-02-25T08:45:17.420 INFO:tasks.rados.rados.0.trial023.stdout:783: writing trial02342524-16 from 669011 to 1238185 tid 1 2026-02-25T08:45:17.424 INFO:tasks.rados.rados.0.trial023.stdout:783: writing trial02342524-16 from 2010607 to 2269011 tid 2 2026-02-25T08:45:17.425 INFO:tasks.rados.rados.0.trial023.stdout:776: done (4 left) 2026-02-25T08:45:17.425 INFO:tasks.rados.rados.0.trial023.stdout:777: done (3 left) 2026-02-25T08:45:17.425 INFO:tasks.rados.rados.0.trial023.stdout:778: done (2 left) 2026-02-25T08:45:17.425 INFO:tasks.rados.rados.0.trial023.stdout:779: done (1 left) 2026-02-25T08:45:17.425 INFO:tasks.rados.rados.0.trial023.stdout:784: read oid 25 snap -1 2026-02-25T08:45:17.425 INFO:tasks.rados.rados.0.trial023.stdout:784: expect (ObjNum 335 snap 94 seq_num 335) 2026-02-25T08:45:17.425 INFO:tasks.rados.rados.0.trial023.stdout:785: write (excl) oid 47 current snap is 100 2026-02-25T08:45:17.425 INFO:tasks.rados.rados.0.trial023.stdout:785: seq_num 374 ranges {685818=644423,2069089=216729} 2026-02-25T08:45:17.429 INFO:tasks.rados.rados.0.trial023.stdout:785: writing trial02342524-47 from 685818 to 1330241 tid 1 2026-02-25T08:45:17.433 INFO:tasks.rados.rados.0.trial023.stdout:785: writing trial02342524-47 from 2069089 to 2285818 tid 2 2026-02-25T08:45:17.433 INFO:tasks.rados.rados.0.trial023.stdout:786: read oid 28 snap -1 2026-02-25T08:45:17.433 INFO:tasks.rados.rados.0.trial023.stdout:786: expect (ObjNum 354 snap 99 seq_num 354) 2026-02-25T08:45:17.434 INFO:tasks.rados.rados.0.trial023.stdout:787: read oid 46 snap -1 2026-02-25T08:45:17.434 INFO:tasks.rados.rados.0.trial023.stdout:787: expect (ObjNum 371 snap 100 seq_num 371) 2026-02-25T08:45:17.434 INFO:tasks.rados.rados.0.trial023.stdout:788: read oid 11 snap -1 2026-02-25T08:45:17.434 INFO:tasks.rados.rados.0.trial023.stdout:788: expect deleted 2026-02-25T08:45:17.434 INFO:tasks.rados.rados.0.trial023.stdout:789: snap_create 2026-02-25T08:45:17.446 INFO:tasks.rados.rados.0.trial023.stdout:783: finishing write tid 1 to trial02342524-16 2026-02-25T08:45:17.450 INFO:tasks.rados.rados.0.trial023.stdout:784: expect (ObjNum 335 snap 94 seq_num 335) 2026-02-25T08:45:17.513 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 696799 2026-02-25T08:45:17.571 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 687048 2026-02-25T08:45:17.625 INFO:tasks.rados.rados.0.trial023.stdout:785: finishing write tid 1 to trial02342524-47 2026-02-25T08:45:17.626 INFO:tasks.rados.rados.0.trial023.stdout:787: expect (ObjNum 371 snap 100 seq_num 371) 2026-02-25T08:45:17.626 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 429642 2026-02-25T08:45:17.661 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:17.680 INFO:tasks.rados.rados.0.trial023.stdout:785: finishing write tid 2 to trial02342524-47 2026-02-25T08:45:17.680 INFO:tasks.rados.rados.0.trial023.stdout:783: finishing write tid 2 to trial02342524-16 2026-02-25T08:45:17.680 INFO:tasks.rados.rados.0.trial023.stdout:785: finishing write tid 3 to trial02342524-47 2026-02-25T08:45:17.680 INFO:tasks.rados.rados.0.trial023.stdout:786: expect (ObjNum 354 snap 99 seq_num 354) 2026-02-25T08:45:17.680 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:17.799 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:17.863 INFO:tasks.rados.rados.0.trial023.stdout:785: finishing write tid 4 to trial02342524-47 2026-02-25T08:45:17.863 INFO:tasks.rados.rados.0.trial023.stdout:785: oid 47 updating version 0 to 52 2026-02-25T08:45:17.863 INFO:tasks.rados.rados.0.trial023.stdout:785: oid 47 version 52 is already newer than 52 2026-02-25T08:45:17.863 INFO:tasks.rados.rados.0.trial023.stdout:785: oid 47 version 52 is already newer than 50 2026-02-25T08:45:17.863 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 47 v 52 (ObjNum 374 snap 100 seq_num 374) dirty exists 2026-02-25T08:45:17.864 INFO:tasks.rados.rados.0.trial023.stdout:785: left oid 47 (ObjNum 374 snap 100 seq_num 374) 2026-02-25T08:45:17.864 INFO:tasks.rados.rados.0.trial023.stdout:783: finishing write tid 3 to trial02342524-16 2026-02-25T08:45:17.864 INFO:tasks.rados.rados.0.trial023.stdout:783: finishing write tid 4 to trial02342524-16 2026-02-25T08:45:17.864 INFO:tasks.rados.rados.0.trial023.stdout:783: oid 16 updating version 0 to 146 2026-02-25T08:45:17.864 INFO:tasks.rados.rados.0.trial023.stdout:783: oid 16 updating version 146 to 148 2026-02-25T08:45:17.864 INFO:tasks.rados.rados.0.trial023.stdout:783: oid 16 version 148 is already newer than 148 2026-02-25T08:45:17.864 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 16 v 148 (ObjNum 373 snap 100 seq_num 373) dirty exists 2026-02-25T08:45:17.864 INFO:tasks.rados.rados.0.trial023.stdout:783: left oid 16 (ObjNum 373 snap 100 seq_num 373) 2026-02-25T08:45:17.886 INFO:tasks.rados.rados.0.trial023.stdout:783: done (6 left) 2026-02-25T08:45:17.886 INFO:tasks.rados.rados.0.trial023.stdout:784: done (5 left) 2026-02-25T08:45:17.887 INFO:tasks.rados.rados.0.trial023.stdout:785: done (4 left) 2026-02-25T08:45:17.887 INFO:tasks.rados.rados.0.trial023.stdout:786: done (3 left) 2026-02-25T08:45:17.887 INFO:tasks.rados.rados.0.trial023.stdout:787: done (2 left) 2026-02-25T08:45:17.887 INFO:tasks.rados.rados.0.trial023.stdout:788: done (1 left) 2026-02-25T08:45:17.887 INFO:tasks.rados.rados.0.trial023.stdout:789: done (0 left) 2026-02-25T08:45:17.887 INFO:tasks.rados.rados.0.trial023.stdout:790: snap_create 2026-02-25T08:45:18.413 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:18.489 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:18.486+0000 7f05ab50b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:18.491 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:18.489+0000 7f05ab50b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:18.507 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:45:18.535 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:18.535 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:18.890 INFO:tasks.rados.rados.0.trial023.stdout:790: done (0 left) 2026-02-25T08:45:18.891 INFO:tasks.rados.rados.0.trial023.stdout:791: delete oid 1 current snap is 102 2026-02-25T08:45:18.912 INFO:tasks.rados.rados.0.trial023.stdout:791: done (0 left) 2026-02-25T08:45:18.912 INFO:tasks.rados.rados.0.trial023.stdout:792: write oid 25 current snap is 102 2026-02-25T08:45:18.912 INFO:tasks.rados.rados.0.trial023.stdout:792: seq_num 376 ranges {719432=794921,2186053=133379} 2026-02-25T08:45:18.918 INFO:tasks.rados.rados.0.trial023.stdout:792: writing trial02342524-25 from 719432 to 1514353 tid 1 2026-02-25T08:45:18.921 INFO:tasks.rados.rados.0.trial023.stdout:792: writing trial02342524-25 from 2186053 to 2319432 tid 2 2026-02-25T08:45:18.921 INFO:tasks.rados.rados.0.trial023.stdout:793: write oid 37 current snap is 102 2026-02-25T08:45:18.921 INFO:tasks.rados.rados.0.trial023.stdout:793: seq_num 377 ranges {736239=470170,1728182=571782} 2026-02-25T08:45:18.925 INFO:tasks.rados.rados.0.trial023.stdout:793: writing trial02342524-37 from 736239 to 1206409 tid 1 2026-02-25T08:45:18.930 INFO:tasks.rados.rados.0.trial023.stdout:793: writing trial02342524-37 from 1728182 to 2299964 tid 2 2026-02-25T08:45:18.930 INFO:tasks.rados.rados.0.trial023.stdout:794: snap_create 2026-02-25T08:45:18.953 INFO:tasks.rados.rados.0.trial023.stdout:792: finishing write tid 1 to trial02342524-25 2026-02-25T08:45:18.957 INFO:tasks.rados.rados.0.trial023.stdout:793: finishing write tid 1 to trial02342524-37 2026-02-25T08:45:18.961 INFO:tasks.rados.rados.0.trial023.stdout:792: finishing write tid 2 to trial02342524-25 2026-02-25T08:45:18.966 INFO:tasks.rados.rados.0.trial023.stdout:793: finishing write tid 2 to trial02342524-37 2026-02-25T08:45:18.969 INFO:tasks.rados.rados.0.trial023.stdout:792: finishing write tid 3 to trial02342524-25 2026-02-25T08:45:18.970 INFO:tasks.rados.rados.0.trial023.stdout:793: finishing write tid 3 to trial02342524-37 2026-02-25T08:45:18.972 INFO:tasks.rados.rados.0.trial023.stdout:792: finishing write tid 4 to trial02342524-25 2026-02-25T08:45:18.972 INFO:tasks.rados.rados.0.trial023.stdout:792: oid 25 updating version 0 to 191 2026-02-25T08:45:18.972 INFO:tasks.rados.rados.0.trial023.stdout:792: oid 25 version 191 is already newer than 191 2026-02-25T08:45:18.972 INFO:tasks.rados.rados.0.trial023.stdout:792: oid 25 version 191 is already newer than 191 2026-02-25T08:45:18.972 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 25 v 191 (ObjNum 376 snap 102 seq_num 376) dirty exists 2026-02-25T08:45:18.972 INFO:tasks.rados.rados.0.trial023.stdout:792: left oid 25 (ObjNum 376 snap 102 seq_num 376) 2026-02-25T08:45:18.973 INFO:tasks.rados.rados.0.trial023.stdout:793: finishing write tid 4 to trial02342524-37 2026-02-25T08:45:18.973 INFO:tasks.rados.rados.0.trial023.stdout:793: oid 37 updating version 0 to 164 2026-02-25T08:45:18.973 INFO:tasks.rados.rados.0.trial023.stdout:793: oid 37 version 164 is already newer than 164 2026-02-25T08:45:18.973 INFO:tasks.rados.rados.0.trial023.stdout:793: oid 37 version 164 is already newer than 164 2026-02-25T08:45:18.973 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 37 v 164 (ObjNum 377 snap 102 seq_num 377) dirty exists 2026-02-25T08:45:18.973 INFO:tasks.rados.rados.0.trial023.stdout:793: left oid 37 (ObjNum 377 snap 102 seq_num 377) 2026-02-25T08:45:19.831 INFO:tasks.rados.rados.0.trial023.stdout:792: done (2 left) 2026-02-25T08:45:19.832 INFO:tasks.rados.rados.0.trial023.stdout:793: done (1 left) 2026-02-25T08:45:19.832 INFO:tasks.rados.rados.0.trial023.stdout:794: done (0 left) 2026-02-25T08:45:19.832 INFO:tasks.rados.rados.0.trial023.stdout:795: write oid 23 current snap is 103 2026-02-25T08:45:19.832 INFO:tasks.rados.rados.0.trial023.stdout:795: seq_num 378 ranges {753046=545419,1786664=431793} 2026-02-25T08:45:19.836 INFO:tasks.rados.rados.0.trial023.stdout:795: writing trial02342524-23 from 753046 to 1298465 tid 1 2026-02-25T08:45:19.840 INFO:tasks.rados.rados.0.trial023.stdout:795: writing trial02342524-23 from 1786664 to 2218457 tid 2 2026-02-25T08:45:19.840 INFO:tasks.rados.rados.0.trial023.stdout:796: snap_create 2026-02-25T08:45:19.861 INFO:tasks.rados.rados.0.trial023.stdout:795: finishing write tid 1 to trial02342524-23 2026-02-25T08:45:19.867 INFO:tasks.rados.rados.0.trial023.stdout:795: finishing write tid 2 to trial02342524-23 2026-02-25T08:45:19.872 INFO:tasks.rados.rados.0.trial023.stdout:795: finishing write tid 3 to trial02342524-23 2026-02-25T08:45:19.874 INFO:tasks.rados.rados.0.trial023.stdout:795: finishing write tid 4 to trial02342524-23 2026-02-25T08:45:19.874 INFO:tasks.rados.rados.0.trial023.stdout:795: oid 23 updating version 0 to 167 2026-02-25T08:45:19.874 INFO:tasks.rados.rados.0.trial023.stdout:795: oid 23 version 167 is already newer than 167 2026-02-25T08:45:19.875 INFO:tasks.rados.rados.0.trial023.stdout:795: oid 23 version 167 is already newer than 167 2026-02-25T08:45:19.875 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 23 v 167 (ObjNum 378 snap 103 seq_num 378) dirty exists 2026-02-25T08:45:19.875 INFO:tasks.rados.rados.0.trial023.stdout:795: left oid 23 (ObjNum 378 snap 103 seq_num 378) 2026-02-25T08:45:19.920 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:45:19.989 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:45:19.992 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:45:19.993 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:45:20.536 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:20.570 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~220s 2026-02-25T08:45:20.614 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:20.611+0000 7fe74250b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:20.616 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:20.614+0000 7fe74250b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:20.632 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:45:20.659 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:20.659 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:20.836 INFO:tasks.rados.rados.0.trial023.stdout:795: done (1 left) 2026-02-25T08:45:20.836 INFO:tasks.rados.rados.0.trial023.stdout:796: done (0 left) 2026-02-25T08:45:20.836 INFO:tasks.rados.rados.0.trial023.stdout:797: read oid 17 snap -1 2026-02-25T08:45:20.836 INFO:tasks.rados.rados.0.trial023.stdout:797: expect deleted 2026-02-25T08:45:20.836 INFO:tasks.rados.rados.0.trial023.stdout:798: read oid 43 snap -1 2026-02-25T08:45:20.836 INFO:tasks.rados.rados.0.trial023.stdout:798: expect deleted 2026-02-25T08:45:20.837 INFO:tasks.rados.rados.0.trial023.stdout:799: read oid 6 snap -1 2026-02-25T08:45:20.837 INFO:tasks.rados.rados.0.trial023.stdout:799: expect (ObjNum 370 snap 100 seq_num 370) 2026-02-25T08:45:20.837 INFO:tasks.rados.rados.0.trial023.stdout:800: read oid 11 snap -1 2026-02-25T08:45:20.837 INFO:tasks.rados.rados.0.trial023.stdout:800: expect deleted 2026-02-25T08:45:20.837 INFO:tasks.rados.rados.0.trial023.stdout:801: delete oid 19 current snap is 104 2026-02-25T08:45:20.854 INFO:tasks.rados.rados.0.trial023.stdout:801: done (4 left) 2026-02-25T08:45:20.854 INFO:tasks.rados.rados.0.trial023.stdout:802: write oid 15 current snap is 104 2026-02-25T08:45:20.854 INFO:tasks.rados.rados.0.trial023.stdout:802: seq_num 380 ranges {786660=695917,1903628=435462} 2026-02-25T08:45:20.859 INFO:tasks.rados.rados.0.trial023.stdout:802: writing trial02342524-15 from 786660 to 1482577 tid 1 2026-02-25T08:45:20.863 INFO:tasks.rados.rados.0.trial023.stdout:802: writing trial02342524-15 from 1903628 to 2339090 tid 2 2026-02-25T08:45:20.863 INFO:tasks.rados.rados.0.trial023.stdout:803: read oid 36 snap -1 2026-02-25T08:45:20.863 INFO:tasks.rados.rados.0.trial023.stdout:803: expect (ObjNum 361 snap 99 seq_num 361) 2026-02-25T08:45:20.863 INFO:tasks.rados.rados.0.trial023.stdout:804: delete oid 14 current snap is 104 2026-02-25T08:45:20.865 INFO:tasks.rados.rados.0.trial023.stdout:799: expect (ObjNum 370 snap 100 seq_num 370) 2026-02-25T08:45:20.865 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:20.903 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:20.922 INFO:tasks.rados.rados.0.trial023.stdout:797: done (6 left) 2026-02-25T08:45:20.922 INFO:tasks.rados.rados.0.trial023.stdout:798: done (5 left) 2026-02-25T08:45:20.922 INFO:tasks.rados.rados.0.trial023.stdout:799: done (4 left) 2026-02-25T08:45:20.922 INFO:tasks.rados.rados.0.trial023.stdout:800: done (3 left) 2026-02-25T08:45:20.922 INFO:tasks.rados.rados.0.trial023.stdout:804: done (2 left) 2026-02-25T08:45:20.922 INFO:tasks.rados.rados.0.trial023.stdout:805: write oid 12 current snap is 104 2026-02-25T08:45:20.922 INFO:tasks.rados.rados.0.trial023.stdout:802: finishing write tid 1 to trial02342524-15 2026-02-25T08:45:20.922 INFO:tasks.rados.rados.0.trial023.stdout:805: seq_num 382 ranges {420274=562768,1620592=439131} 2026-02-25T08:45:20.925 INFO:tasks.rados.rados.0.trial023.stdout:805: writing trial02342524-12 from 420274 to 983042 tid 1 2026-02-25T08:45:20.929 INFO:tasks.rados.rados.0.trial023.stdout:805: writing trial02342524-12 from 1620592 to 2059723 tid 2 2026-02-25T08:45:20.929 INFO:tasks.rados.rados.0.trial023.stdout:806: write (excl) oid 40 current snap is 104 2026-02-25T08:45:20.930 INFO:tasks.rados.rados.0.trial023.stdout:806: seq_num 383 ranges {437081=638017,1679074=582789} 2026-02-25T08:45:20.933 INFO:tasks.rados.rados.0.trial023.stdout:806: writing trial02342524-40 from 437081 to 1075098 tid 1 2026-02-25T08:45:20.938 INFO:tasks.rados.rados.0.trial023.stdout:806: writing trial02342524-40 from 1679074 to 2261863 tid 2 2026-02-25T08:45:20.938 INFO:tasks.rados.rados.0.trial023.stdout:807: read oid 9 snap 100 2026-02-25T08:45:20.938 INFO:tasks.rados.rados.0.trial023.stdout:807: expect deleted 2026-02-25T08:45:20.939 INFO:tasks.rados.rados.0.trial023.stdout:808: snap_create 2026-02-25T08:45:20.939 INFO:tasks.rados.rados.0.trial023.stdout:802: finishing write tid 2 to trial02342524-15 2026-02-25T08:45:20.939 INFO:tasks.rados.rados.0.trial023.stdout:803: expect (ObjNum 361 snap 99 seq_num 361) 2026-02-25T08:45:20.972 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:21.011 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:21.051 INFO:tasks.rados.rados.0.trial023.stdout:802: finishing write tid 3 to trial02342524-15 2026-02-25T08:45:21.051 INFO:tasks.rados.rados.0.trial023.stdout:802: finishing write tid 4 to trial02342524-15 2026-02-25T08:45:21.051 INFO:tasks.rados.rados.0.trial023.stdout:802: oid 15 updating version 0 to 170 2026-02-25T08:45:21.051 INFO:tasks.rados.rados.0.trial023.stdout:802: oid 15 version 170 is already newer than 170 2026-02-25T08:45:21.051 INFO:tasks.rados.rados.0.trial023.stdout:802: oid 15 version 170 is already newer than 170 2026-02-25T08:45:21.051 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 15 v 170 (ObjNum 380 snap 104 seq_num 380) dirty exists 2026-02-25T08:45:21.052 INFO:tasks.rados.rados.0.trial023.stdout:802: left oid 15 (ObjNum 380 snap 104 seq_num 380) 2026-02-25T08:45:21.052 INFO:tasks.rados.rados.0.trial023.stdout:805: finishing write tid 1 to trial02342524-12 2026-02-25T08:45:21.052 INFO:tasks.rados.rados.0.trial023.stdout:805: finishing write tid 2 to trial02342524-12 2026-02-25T08:45:21.052 INFO:tasks.rados.rados.0.trial023.stdout:805: finishing write tid 3 to trial02342524-12 2026-02-25T08:45:21.052 INFO:tasks.rados.rados.0.trial023.stdout:805: finishing write tid 4 to trial02342524-12 2026-02-25T08:45:21.052 INFO:tasks.rados.rados.0.trial023.stdout:805: oid 12 updating version 0 to 143 2026-02-25T08:45:21.052 INFO:tasks.rados.rados.0.trial023.stdout:805: oid 12 version 143 is already newer than 143 2026-02-25T08:45:21.052 INFO:tasks.rados.rados.0.trial023.stdout:805: oid 12 version 143 is already newer than 143 2026-02-25T08:45:21.052 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 12 v 143 (ObjNum 382 snap 104 seq_num 382) dirty exists 2026-02-25T08:45:21.052 INFO:tasks.rados.rados.0.trial023.stdout:805: left oid 12 (ObjNum 382 snap 104 seq_num 382) 2026-02-25T08:45:21.052 INFO:tasks.rados.rados.0.trial023.stdout:806: finishing write tid 1 to trial02342524-40 2026-02-25T08:45:21.052 INFO:tasks.rados.rados.0.trial023.stdout:806: finishing write tid 2 to trial02342524-40 2026-02-25T08:45:21.053 INFO:tasks.rados.rados.0.trial023.stdout:806: finishing write tid 3 to trial02342524-40 2026-02-25T08:45:21.053 INFO:tasks.rados.rados.0.trial023.stdout:806: finishing write tid 4 to trial02342524-40 2026-02-25T08:45:21.053 INFO:tasks.rados.rados.0.trial023.stdout:806: oid 40 updating version 0 to 174 2026-02-25T08:45:21.053 INFO:tasks.rados.rados.0.trial023.stdout:806: oid 40 version 174 is already newer than 172 2026-02-25T08:45:21.053 INFO:tasks.rados.rados.0.trial023.stdout:806: oid 40 version 174 is already newer than 174 2026-02-25T08:45:21.053 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 40 v 174 (ObjNum 383 snap 104 seq_num 383) dirty exists 2026-02-25T08:45:21.053 INFO:tasks.rados.rados.0.trial023.stdout:806: left oid 40 (ObjNum 383 snap 104 seq_num 383) 2026-02-25T08:45:21.900 INFO:tasks.rados.rados.0.trial023.stdout:802: done (5 left) 2026-02-25T08:45:21.900 INFO:tasks.rados.rados.0.trial023.stdout:803: done (4 left) 2026-02-25T08:45:21.901 INFO:tasks.rados.rados.0.trial023.stdout:805: done (3 left) 2026-02-25T08:45:21.901 INFO:tasks.rados.rados.0.trial023.stdout:806: done (2 left) 2026-02-25T08:45:21.901 INFO:tasks.rados.rados.0.trial023.stdout:807: done (1 left) 2026-02-25T08:45:21.901 INFO:tasks.rados.rados.0.trial023.stdout:808: done (0 left) 2026-02-25T08:45:21.901 INFO:tasks.rados.rados.0.trial023.stdout:809: read oid 48 snap -1 2026-02-25T08:45:21.901 INFO:tasks.rados.rados.0.trial023.stdout:809: expect (ObjNum 344 snap 98 seq_num 344) 2026-02-25T08:45:21.901 INFO:tasks.rados.rados.0.trial023.stdout:810: snap_remove snap 100 2026-02-25T08:45:22.661 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:22.737 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:22.735+0000 7fb78bf0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:22.740 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:22.739+0000 7fb78bf0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:22.755 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:45:22.782 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:22.783 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:22.902 INFO:tasks.rados.rados.0.trial023.stdout:810: done (1 left) 2026-02-25T08:45:22.902 INFO:tasks.rados.rados.0.trial023.stdout:811: delete oid 2 current snap is 105 2026-02-25T08:45:22.902 INFO:tasks.rados.rados.0.trial023.stdout:809: expect (ObjNum 344 snap 98 seq_num 344) 2026-02-25T08:45:22.902 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:22.962 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:22.994 INFO:tasks.rados.rados.0.trial023.stdout:809: done (1 left) 2026-02-25T08:45:22.995 INFO:tasks.rados.rados.0.trial023.stdout:811: done (0 left) 2026-02-25T08:45:22.995 INFO:tasks.rados.rados.0.trial023.stdout:812: read oid 44 snap -1 2026-02-25T08:45:22.995 INFO:tasks.rados.rados.0.trial023.stdout:812: expect deleted 2026-02-25T08:45:22.995 INFO:tasks.rados.rados.0.trial023.stdout:813: read oid 22 snap -1 2026-02-25T08:45:22.995 INFO:tasks.rados.rados.0.trial023.stdout:813: expect (ObjNum 317 snap 93 seq_num 317) 2026-02-25T08:45:22.995 INFO:tasks.rados.rados.0.trial023.stdout:814: write oid 23 current snap is 105 2026-02-25T08:45:22.995 INFO:tasks.rados.rados.0.trial023.stdout:814: seq_num 385 ranges {470695=788515,1679685=586458} 2026-02-25T08:45:22.999 INFO:tasks.rados.rados.0.trial023.stdout:814: writing trial02342524-23 from 470695 to 1259210 tid 1 2026-02-25T08:45:23.003 INFO:tasks.rados.rados.0.trial023.stdout:814: writing trial02342524-23 from 1679685 to 2266143 tid 2 2026-02-25T08:45:23.003 INFO:tasks.rados.rados.0.trial023.stdout:815: read oid 2 snap 103 2026-02-25T08:45:23.003 INFO:tasks.rados.rados.0.trial023.stdout:815: expect (ObjNum 327 snap 93 seq_num 327) 2026-02-25T08:45:23.004 INFO:tasks.rados.rados.0.trial023.stdout:816: write (excl) oid 46 current snap is 105 2026-02-25T08:45:23.004 INFO:tasks.rados.rados.0.trial023.stdout:816: seq_num 386 ranges {487502=463764,1738167=446469} 2026-02-25T08:45:23.006 INFO:tasks.rados.rados.0.trial023.stdout:816: writing trial02342524-46 from 487502 to 951266 tid 1 2026-02-25T08:45:23.011 INFO:tasks.rados.rados.0.trial023.stdout:816: writing trial02342524-46 from 1738167 to 2184636 tid 2 2026-02-25T08:45:23.011 INFO:tasks.rados.rados.0.trial023.stdout:817: snap_remove snap 101 2026-02-25T08:45:23.907 INFO:tasks.rados.rados.0.trial023.stdout:817: done (5 left) 2026-02-25T08:45:23.907 INFO:tasks.rados.rados.0.trial023.stdout:818: read oid 8 snap 103 2026-02-25T08:45:23.907 INFO:tasks.rados.rados.0.trial023.stdout:818: expect (ObjNum 295 snap 87 seq_num 295) 2026-02-25T08:45:23.907 INFO:tasks.rados.rados.0.trial023.stdout:819: read oid 31 snap -1 2026-02-25T08:45:23.907 INFO:tasks.rados.rados.0.trial023.stdout:819: expect deleted 2026-02-25T08:45:23.907 INFO:tasks.rados.rados.0.trial023.stdout:820: read oid 12 snap 103 2026-02-25T08:45:23.907 INFO:tasks.rados.rados.0.trial023.stdout:820: expect deleted 2026-02-25T08:45:23.907 INFO:tasks.rados.rados.0.trial023.stdout:821: read oid 7 snap -1 2026-02-25T08:45:23.908 INFO:tasks.rados.rados.0.trial023.stdout:821: expect (ObjNum 368 snap 100 seq_num 368) 2026-02-25T08:45:23.908 INFO:tasks.rados.rados.0.trial023.stdout:822: snap_remove snap 104 2026-02-25T08:45:24.784 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:24.860 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:24.858+0000 7f445b15c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:24.862 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:24.861+0000 7f445b15c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:24.877 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:45:24.905 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:24.906 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:24.908 INFO:tasks.rados.rados.0.trial023.stdout:822: done (9 left) 2026-02-25T08:45:24.909 INFO:tasks.rados.rados.0.trial023.stdout:823: read oid 26 snap 102 2026-02-25T08:45:24.909 INFO:tasks.rados.rados.0.trial023.stdout:823: expect (ObjNum 330 snap 93 seq_num 330) 2026-02-25T08:45:24.909 INFO:tasks.rados.rados.0.trial023.stdout:824: write (excl) oid 6 current snap is 105 2026-02-25T08:45:24.909 INFO:tasks.rados.rados.0.trial023.stdout:824: seq_num 387 ranges {504309=539013,1796649=590127} 2026-02-25T08:45:24.912 INFO:tasks.rados.rados.0.trial023.stdout:824: writing trial02342524-6 from 504309 to 1043322 tid 1 2026-02-25T08:45:24.918 INFO:tasks.rados.rados.0.trial023.stdout:824: writing trial02342524-6 from 1796649 to 2386776 tid 2 2026-02-25T08:45:24.918 INFO:tasks.rados.rados.0.trial023.stdout:825: read oid 33 snap 102 2026-02-25T08:45:24.918 INFO:tasks.rados.rados.0.trial023.stdout:825: expect (ObjNum 323 snap 93 seq_num 323) 2026-02-25T08:45:24.918 INFO:tasks.rados.rados.0.trial023.stdout:826: snap_create 2026-02-25T08:45:24.918 INFO:tasks.rados.rados.0.trial023.stdout:813: expect (ObjNum 317 snap 93 seq_num 317) 2026-02-25T08:45:24.994 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:45:25.001 INFO:tasks.rados.rados.0.trial023.stdout:815: expect (ObjNum 327 snap 93 seq_num 327) 2026-02-25T08:45:25.002 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:25.044 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 667692 2026-02-25T08:45:25.064 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:45:25.067 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:45:25.068 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:45:25.078 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:25.118 INFO:tasks.rados.rados.0.trial023.stdout:814: finishing write tid 1 to trial02342524-23 2026-02-25T08:45:25.118 INFO:tasks.rados.rados.0.trial023.stdout:816: finishing write tid 1 to trial02342524-46 2026-02-25T08:45:25.118 INFO:tasks.rados.rados.0.trial023.stdout:814: finishing write tid 2 to trial02342524-23 2026-02-25T08:45:25.119 INFO:tasks.rados.rados.0.trial023.stdout:816: finishing write tid 2 to trial02342524-46 2026-02-25T08:45:25.119 INFO:tasks.rados.rados.0.trial023.stdout:814: finishing write tid 3 to trial02342524-23 2026-02-25T08:45:25.119 INFO:tasks.rados.rados.0.trial023.stdout:814: finishing write tid 4 to trial02342524-23 2026-02-25T08:45:25.119 INFO:tasks.rados.rados.0.trial023.stdout:814: oid 23 updating version 0 to 171 2026-02-25T08:45:25.119 INFO:tasks.rados.rados.0.trial023.stdout:814: oid 23 version 171 is already newer than 171 2026-02-25T08:45:25.119 INFO:tasks.rados.rados.0.trial023.stdout:814: oid 23 version 171 is already newer than 171 2026-02-25T08:45:25.119 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 23 v 171 (ObjNum 385 snap 105 seq_num 385) dirty exists 2026-02-25T08:45:25.119 INFO:tasks.rados.rados.0.trial023.stdout:814: left oid 23 (ObjNum 385 snap 105 seq_num 385) 2026-02-25T08:45:25.119 INFO:tasks.rados.rados.0.trial023.stdout:816: finishing write tid 3 to trial02342524-46 2026-02-25T08:45:25.119 INFO:tasks.rados.rados.0.trial023.stdout:816: finishing write tid 4 to trial02342524-46 2026-02-25T08:45:25.119 INFO:tasks.rados.rados.0.trial023.stdout:816: oid 46 updating version 0 to 235 2026-02-25T08:45:25.120 INFO:tasks.rados.rados.0.trial023.stdout:816: oid 46 updating version 235 to 237 2026-02-25T08:45:25.120 INFO:tasks.rados.rados.0.trial023.stdout:816: oid 46 version 237 is already newer than 237 2026-02-25T08:45:25.120 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 46 v 237 (ObjNum 386 snap 105 seq_num 386) dirty exists 2026-02-25T08:45:25.120 INFO:tasks.rados.rados.0.trial023.stdout:816: left oid 46 (ObjNum 386 snap 105 seq_num 386) 2026-02-25T08:45:25.120 INFO:tasks.rados.rados.0.trial023.stdout:821: expect (ObjNum 368 snap 100 seq_num 368) 2026-02-25T08:45:25.192 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1847548 2026-02-25T08:45:25.195 INFO:tasks.rados.rados.0.trial023.stdout:818: expect (ObjNum 295 snap 87 seq_num 295) 2026-02-25T08:45:25.205 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1660291 2026-02-25T08:45:25.220 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 23720 2026-02-25T08:45:25.285 INFO:tasks.rados.rados.0.trial023.stdout:825: expect (ObjNum 323 snap 93 seq_num 323) 2026-02-25T08:45:25.286 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:25.378 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 3061259 2026-02-25T08:45:25.391 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 699489 2026-02-25T08:45:25.436 INFO:tasks.rados.rados.0.trial023.stdout:824: finishing write tid 1 to trial02342524-6 2026-02-25T08:45:25.436 INFO:tasks.rados.rados.0.trial023.stdout:823: expect (ObjNum 330 snap 93 seq_num 330) 2026-02-25T08:45:25.438 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 506952 2026-02-25T08:45:25.473 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 313054 2026-02-25T08:45:25.511 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:25.550 INFO:tasks.rados.rados.0.trial023.stdout:824: finishing write tid 2 to trial02342524-6 2026-02-25T08:45:25.550 INFO:tasks.rados.rados.0.trial023.stdout:824: finishing write tid 3 to trial02342524-6 2026-02-25T08:45:25.550 INFO:tasks.rados.rados.0.trial023.stdout:824: finishing write tid 4 to trial02342524-6 2026-02-25T08:45:25.551 INFO:tasks.rados.rados.0.trial023.stdout:824: oid 6 updating version 0 to 56 2026-02-25T08:45:25.551 INFO:tasks.rados.rados.0.trial023.stdout:824: oid 6 version 56 is already newer than 56 2026-02-25T08:45:25.551 INFO:tasks.rados.rados.0.trial023.stdout:824: oid 6 version 56 is already newer than 54 2026-02-25T08:45:25.551 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 6 v 56 (ObjNum 387 snap 105 seq_num 387) dirty exists 2026-02-25T08:45:25.551 INFO:tasks.rados.rados.0.trial023.stdout:824: left oid 6 (ObjNum 387 snap 105 seq_num 387) 2026-02-25T08:45:25.914 INFO:tasks.rados.rados.0.trial023.stdout:812: done (12 left) 2026-02-25T08:45:25.915 INFO:tasks.rados.rados.0.trial023.stdout:813: done (11 left) 2026-02-25T08:45:25.915 INFO:tasks.rados.rados.0.trial023.stdout:814: done (10 left) 2026-02-25T08:45:25.915 INFO:tasks.rados.rados.0.trial023.stdout:815: done (9 left) 2026-02-25T08:45:25.915 INFO:tasks.rados.rados.0.trial023.stdout:816: done (8 left) 2026-02-25T08:45:25.915 INFO:tasks.rados.rados.0.trial023.stdout:818: done (7 left) 2026-02-25T08:45:25.915 INFO:tasks.rados.rados.0.trial023.stdout:819: done (6 left) 2026-02-25T08:45:25.915 INFO:tasks.rados.rados.0.trial023.stdout:820: done (5 left) 2026-02-25T08:45:25.915 INFO:tasks.rados.rados.0.trial023.stdout:821: done (4 left) 2026-02-25T08:45:25.915 INFO:tasks.rados.rados.0.trial023.stdout:823: done (3 left) 2026-02-25T08:45:25.916 INFO:tasks.rados.rados.0.trial023.stdout:824: done (2 left) 2026-02-25T08:45:25.916 INFO:tasks.rados.rados.0.trial023.stdout:825: done (1 left) 2026-02-25T08:45:25.916 INFO:tasks.rados.rados.0.trial023.stdout:826: done (0 left) 2026-02-25T08:45:25.916 INFO:tasks.rados.rados.0.trial023.stdout:827: delete oid 17 current snap is 106 2026-02-25T08:45:25.933 INFO:tasks.rados.rados.0.trial023.stdout:827: done (0 left) 2026-02-25T08:45:25.933 INFO:tasks.rados.rados.0.trial023.stdout:828: write oid 1 current snap is 106 2026-02-25T08:45:25.933 INFO:tasks.rados.rados.0.trial023.stdout:828: seq_num 389 ranges {537923=405864,1513613=593796} 2026-02-25T08:45:25.936 INFO:tasks.rados.rados.0.trial023.stdout:828: writing trial02342524-1 from 537923 to 943787 tid 1 2026-02-25T08:45:25.941 INFO:tasks.rados.rados.0.trial023.stdout:828: writing trial02342524-1 from 1513613 to 2107409 tid 2 2026-02-25T08:45:25.941 INFO:tasks.rados.rados.0.trial023.stdout:829: snap_remove snap 105 2026-02-25T08:45:26.074 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~226s 2026-02-25T08:45:26.907 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:26.918 INFO:tasks.rados.rados.0.trial023.stdout:829: done (1 left) 2026-02-25T08:45:26.918 INFO:tasks.rados.rados.0.trial023.stdout:830: read oid 7 snap 102 2026-02-25T08:45:26.918 INFO:tasks.rados.rados.0.trial023.stdout:830: expect (ObjNum 368 snap 100 seq_num 368) 2026-02-25T08:45:26.918 INFO:tasks.rados.rados.0.trial023.stdout:831: delete oid 50 current snap is 106 2026-02-25T08:45:26.919 INFO:tasks.rados.rados.0.trial023.stdout:828: finishing write tid 1 to trial02342524-1 2026-02-25T08:45:26.919 INFO:tasks.rados.rados.0.trial023.stdout:828: finishing write tid 2 to trial02342524-1 2026-02-25T08:45:26.919 INFO:tasks.rados.rados.0.trial023.stdout:828: finishing write tid 3 to trial02342524-1 2026-02-25T08:45:26.919 INFO:tasks.rados.rados.0.trial023.stdout:828: finishing write tid 4 to trial02342524-1 2026-02-25T08:45:26.919 INFO:tasks.rados.rados.0.trial023.stdout:828: oid 1 updating version 0 to 146 2026-02-25T08:45:26.919 INFO:tasks.rados.rados.0.trial023.stdout:828: oid 1 version 146 is already newer than 146 2026-02-25T08:45:26.919 INFO:tasks.rados.rados.0.trial023.stdout:828: oid 1 version 146 is already newer than 146 2026-02-25T08:45:26.919 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 1 v 146 (ObjNum 389 snap 106 seq_num 389) dirty exists 2026-02-25T08:45:26.919 INFO:tasks.rados.rados.0.trial023.stdout:828: left oid 1 (ObjNum 389 snap 106 seq_num 389) 2026-02-25T08:45:26.940 INFO:tasks.rados.rados.0.trial023.stdout:828: done (2 left) 2026-02-25T08:45:26.940 INFO:tasks.rados.rados.0.trial023.stdout:831: done (1 left) 2026-02-25T08:45:26.940 INFO:tasks.rados.rados.0.trial023.stdout:832: read oid 22 snap -1 2026-02-25T08:45:26.941 INFO:tasks.rados.rados.0.trial023.stdout:832: expect (ObjNum 317 snap 93 seq_num 317) 2026-02-25T08:45:26.941 INFO:tasks.rados.rados.0.trial023.stdout:833: read oid 21 snap -1 2026-02-25T08:45:26.941 INFO:tasks.rados.rados.0.trial023.stdout:833: expect (ObjNum 356 snap 99 seq_num 356) 2026-02-25T08:45:26.941 INFO:tasks.rados.rados.0.trial023.stdout:834: delete oid 19 current snap is 106 2026-02-25T08:45:26.954 INFO:tasks.rados.rados.0.trial023.stdout:832: expect (ObjNum 317 snap 93 seq_num 317) 2026-02-25T08:45:26.986 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:26.984+0000 7f7ec005a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:26.988 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:26.987+0000 7f7ec005a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:27.003 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:45:27.022 INFO:tasks.rados.rados.0.trial023.stdout:832: done (3 left) 2026-02-25T08:45:27.023 INFO:tasks.rados.rados.0.trial023.stdout:834: done (2 left) 2026-02-25T08:45:27.023 INFO:tasks.rados.rados.0.trial023.stdout:835: write (excl) oid 2 current snap is 106 2026-02-25T08:45:27.023 INFO:tasks.rados.rados.0.trial023.stdout:830: expect (ObjNum 368 snap 100 seq_num 368) 2026-02-25T08:45:27.029 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:27.030 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:27.053 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 416742 2026-02-25T08:45:27.100 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1967476 2026-02-25T08:45:27.102 INFO:tasks.rados.rados.0.trial023.stdout:833: expect (ObjNum 356 snap 99 seq_num 356) 2026-02-25T08:45:27.102 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:27.214 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:27.270 INFO:tasks.rados.rados.0.trial023.stdout:835: seq_num 392 ranges {588344=631611,1689059=741123} 2026-02-25T08:45:27.275 INFO:tasks.rados.rados.0.trial023.stdout:835: writing trial02342524-2 from 588344 to 1219955 tid 1 2026-02-25T08:45:27.279 INFO:tasks.rados.rados.0.trial023.stdout:835: writing trial02342524-2 from 1689059 to 2430182 tid 2 2026-02-25T08:45:27.280 INFO:tasks.rados.rados.0.trial023.stdout:830: done (2 left) 2026-02-25T08:45:27.280 INFO:tasks.rados.rados.0.trial023.stdout:833: done (1 left) 2026-02-25T08:45:27.280 INFO:tasks.rados.rados.0.trial023.stdout:836: write (excl) oid 19 current snap is 106 2026-02-25T08:45:27.280 INFO:tasks.rados.rados.0.trial023.stdout:836: seq_num 393 ranges {605151=706860,2031188=573963} 2026-02-25T08:45:27.284 INFO:tasks.rados.rados.0.trial023.stdout:836: writing trial02342524-19 from 605151 to 1312011 tid 1 2026-02-25T08:45:27.290 INFO:tasks.rados.rados.0.trial023.stdout:836: writing trial02342524-19 from 2031188 to 2605151 tid 2 2026-02-25T08:45:27.290 INFO:tasks.rados.rados.0.trial023.stdout:837: read oid 47 snap -1 2026-02-25T08:45:27.290 INFO:tasks.rados.rados.0.trial023.stdout:837: expect (ObjNum 374 snap 100 seq_num 374) 2026-02-25T08:45:27.290 INFO:tasks.rados.rados.0.trial023.stdout:838: read oid 25 snap -1 2026-02-25T08:45:27.290 INFO:tasks.rados.rados.0.trial023.stdout:838: expect (ObjNum 376 snap 102 seq_num 376) 2026-02-25T08:45:27.290 INFO:tasks.rados.rados.0.trial023.stdout:839: write oid 35 current snap is 106 2026-02-25T08:45:27.291 INFO:tasks.rados.rados.0.trial023.stdout:839: seq_num 394 ranges {621958=782109,2089670=532288} 2026-02-25T08:45:27.295 INFO:tasks.rados.rados.0.trial023.stdout:839: writing trial02342524-35 from 621958 to 1404067 tid 1 2026-02-25T08:45:27.299 INFO:tasks.rados.rados.0.trial023.stdout:839: writing trial02342524-35 from 2089670 to 2621958 tid 2 2026-02-25T08:45:27.300 INFO:tasks.rados.rados.0.trial023.stdout:840: write oid 8 current snap is 106 2026-02-25T08:45:27.300 INFO:tasks.rados.rados.0.trial023.stdout:840: seq_num 395 ranges {638765=457358,1748152=604803} 2026-02-25T08:45:27.303 INFO:tasks.rados.rados.0.trial023.stdout:840: writing trial02342524-8 from 638765 to 1096123 tid 1 2026-02-25T08:45:27.309 INFO:tasks.rados.rados.0.trial023.stdout:840: writing trial02342524-8 from 1748152 to 2352955 tid 2 2026-02-25T08:45:27.309 INFO:tasks.rados.rados.0.trial023.stdout:841: write (excl) oid 42 current snap is 106 2026-02-25T08:45:27.309 INFO:tasks.rados.rados.0.trial023.stdout:835: finishing write tid 1 to trial02342524-2 2026-02-25T08:45:27.309 INFO:tasks.rados.rados.0.trial023.stdout:841: seq_num 396 ranges {655572=648960,1922987=732585} 2026-02-25T08:45:27.313 INFO:tasks.rados.rados.0.trial023.stdout:841: writing trial02342524-42 from 655572 to 1304532 tid 1 2026-02-25T08:45:27.319 INFO:tasks.rados.rados.0.trial023.stdout:841: writing trial02342524-42 from 1922987 to 2655572 tid 2 2026-02-25T08:45:27.319 INFO:tasks.rados.rados.0.trial023.stdout:842: snap_create 2026-02-25T08:45:27.319 INFO:tasks.rados.rados.0.trial023.stdout:837: expect (ObjNum 374 snap 100 seq_num 374) 2026-02-25T08:45:27.319 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:27.350 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:27.413 INFO:tasks.rados.rados.0.trial023.stdout:835: finishing write tid 2 to trial02342524-2 2026-02-25T08:45:27.413 INFO:tasks.rados.rados.0.trial023.stdout:838: expect (ObjNum 376 snap 102 seq_num 376) 2026-02-25T08:45:27.424 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2159417 2026-02-25T08:45:27.461 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1521375 2026-02-25T08:45:27.477 INFO:tasks.rados.rados.0.trial023.stdout:836: finishing write tid 1 to trial02342524-19 2026-02-25T08:45:27.477 INFO:tasks.rados.rados.0.trial023.stdout:835: finishing write tid 3 to trial02342524-2 2026-02-25T08:45:27.477 INFO:tasks.rados.rados.0.trial023.stdout:835: finishing write tid 4 to trial02342524-2 2026-02-25T08:45:27.477 INFO:tasks.rados.rados.0.trial023.stdout:835: oid 2 updating version 0 to 64 2026-02-25T08:45:27.478 INFO:tasks.rados.rados.0.trial023.stdout:835: oid 2 version 64 is already newer than 62 2026-02-25T08:45:27.478 INFO:tasks.rados.rados.0.trial023.stdout:835: oid 2 version 64 is already newer than 64 2026-02-25T08:45:27.478 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 2 v 64 (ObjNum 392 snap 106 seq_num 392) dirty exists 2026-02-25T08:45:27.478 INFO:tasks.rados.rados.0.trial023.stdout:835: left oid 2 (ObjNum 392 snap 106 seq_num 392) 2026-02-25T08:45:27.478 INFO:tasks.rados.rados.0.trial023.stdout:836: finishing write tid 2 to trial02342524-19 2026-02-25T08:45:27.478 INFO:tasks.rados.rados.0.trial023.stdout:836: finishing write tid 3 to trial02342524-19 2026-02-25T08:45:27.478 INFO:tasks.rados.rados.0.trial023.stdout:836: finishing write tid 4 to trial02342524-19 2026-02-25T08:45:27.478 INFO:tasks.rados.rados.0.trial023.stdout:836: oid 19 updating version 0 to 151 2026-02-25T08:45:27.478 INFO:tasks.rados.rados.0.trial023.stdout:836: oid 19 version 151 is already newer than 149 2026-02-25T08:45:27.478 INFO:tasks.rados.rados.0.trial023.stdout:836: oid 19 version 151 is already newer than 151 2026-02-25T08:45:27.479 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 19 v 151 (ObjNum 393 snap 106 seq_num 393) dirty exists 2026-02-25T08:45:27.479 INFO:tasks.rados.rados.0.trial023.stdout:836: left oid 19 (ObjNum 393 snap 106 seq_num 393) 2026-02-25T08:45:27.479 INFO:tasks.rados.rados.0.trial023.stdout:839: finishing write tid 1 to trial02342524-35 2026-02-25T08:45:27.479 INFO:tasks.rados.rados.0.trial023.stdout:839: finishing write tid 2 to trial02342524-35 2026-02-25T08:45:27.479 INFO:tasks.rados.rados.0.trial023.stdout:839: finishing write tid 3 to trial02342524-35 2026-02-25T08:45:27.479 INFO:tasks.rados.rados.0.trial023.stdout:839: finishing write tid 4 to trial02342524-35 2026-02-25T08:45:27.479 INFO:tasks.rados.rados.0.trial023.stdout:839: oid 35 updating version 0 to 245 2026-02-25T08:45:27.479 INFO:tasks.rados.rados.0.trial023.stdout:839: oid 35 version 245 is already newer than 245 2026-02-25T08:45:27.479 INFO:tasks.rados.rados.0.trial023.stdout:839: oid 35 version 245 is already newer than 245 2026-02-25T08:45:27.479 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 35 v 245 (ObjNum 394 snap 106 seq_num 394) dirty exists 2026-02-25T08:45:27.479 INFO:tasks.rados.rados.0.trial023.stdout:839: left oid 35 (ObjNum 394 snap 106 seq_num 394) 2026-02-25T08:45:27.480 INFO:tasks.rados.rados.0.trial023.stdout:841: finishing write tid 1 to trial02342524-42 2026-02-25T08:45:27.480 INFO:tasks.rados.rados.0.trial023.stdout:841: finishing write tid 2 to trial02342524-42 2026-02-25T08:45:27.480 INFO:tasks.rados.rados.0.trial023.stdout:841: finishing write tid 3 to trial02342524-42 2026-02-25T08:45:27.480 INFO:tasks.rados.rados.0.trial023.stdout:841: finishing write tid 4 to trial02342524-42 2026-02-25T08:45:27.480 INFO:tasks.rados.rados.0.trial023.stdout:841: oid 42 updating version 0 to 248 2026-02-25T08:45:27.480 INFO:tasks.rados.rados.0.trial023.stdout:841: oid 42 version 248 is already newer than 246 2026-02-25T08:45:27.480 INFO:tasks.rados.rados.0.trial023.stdout:841: oid 42 version 248 is already newer than 248 2026-02-25T08:45:27.480 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 42 v 248 (ObjNum 396 snap 106 seq_num 396) dirty exists 2026-02-25T08:45:27.480 INFO:tasks.rados.rados.0.trial023.stdout:841: left oid 42 (ObjNum 396 snap 106 seq_num 396) 2026-02-25T08:45:27.480 INFO:tasks.rados.rados.0.trial023.stdout:840: finishing write tid 1 to trial02342524-8 2026-02-25T08:45:27.480 INFO:tasks.rados.rados.0.trial023.stdout:840: finishing write tid 2 to trial02342524-8 2026-02-25T08:45:27.481 INFO:tasks.rados.rados.0.trial023.stdout:840: finishing write tid 3 to trial02342524-8 2026-02-25T08:45:27.481 INFO:tasks.rados.rados.0.trial023.stdout:840: finishing write tid 4 to trial02342524-8 2026-02-25T08:45:27.481 INFO:tasks.rados.rados.0.trial023.stdout:840: oid 8 updating version 0 to 46 2026-02-25T08:45:27.481 INFO:tasks.rados.rados.0.trial023.stdout:840: oid 8 version 46 is already newer than 46 2026-02-25T08:45:27.481 INFO:tasks.rados.rados.0.trial023.stdout:840: oid 8 version 46 is already newer than 46 2026-02-25T08:45:27.481 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 8 v 46 (ObjNum 395 snap 106 seq_num 395) dirty exists 2026-02-25T08:45:27.481 INFO:tasks.rados.rados.0.trial023.stdout:840: left oid 8 (ObjNum 395 snap 106 seq_num 395) 2026-02-25T08:45:27.922 INFO:tasks.rados.rados.0.trial023.stdout:835: done (7 left) 2026-02-25T08:45:27.922 INFO:tasks.rados.rados.0.trial023.stdout:836: done (6 left) 2026-02-25T08:45:27.922 INFO:tasks.rados.rados.0.trial023.stdout:837: done (5 left) 2026-02-25T08:45:27.922 INFO:tasks.rados.rados.0.trial023.stdout:838: done (4 left) 2026-02-25T08:45:27.922 INFO:tasks.rados.rados.0.trial023.stdout:839: done (3 left) 2026-02-25T08:45:27.922 INFO:tasks.rados.rados.0.trial023.stdout:840: done (2 left) 2026-02-25T08:45:27.922 INFO:tasks.rados.rados.0.trial023.stdout:841: done (1 left) 2026-02-25T08:45:27.922 INFO:tasks.rados.rados.0.trial023.stdout:842: done (0 left) 2026-02-25T08:45:27.923 INFO:tasks.rados.rados.0.trial023.stdout:843: write oid 2 current snap is 107 2026-02-25T08:45:27.923 INFO:tasks.rados.rados.0.trial023.stdout:843: seq_num 397 ranges {672379=724209,1865116=608472} 2026-02-25T08:45:27.926 INFO:tasks.rados.rados.0.trial023.stdout:843: writing trial02342524-2 from 672379 to 1396588 tid 1 2026-02-25T08:45:27.931 INFO:tasks.rados.rados.0.trial023.stdout:843: writing trial02342524-2 from 1865116 to 2473588 tid 2 2026-02-25T08:45:27.931 INFO:tasks.rados.rados.0.trial023.stdout:844: read oid 6 snap -1 2026-02-25T08:45:27.931 INFO:tasks.rados.rados.0.trial023.stdout:844: expect (ObjNum 387 snap 105 seq_num 387) 2026-02-25T08:45:27.931 INFO:tasks.rados.rados.0.trial023.stdout:845: snap_remove snap 106 2026-02-25T08:45:28.926 INFO:tasks.rados.rados.0.trial023.stdout:845: done (2 left) 2026-02-25T08:45:28.926 INFO:tasks.rados.rados.0.trial023.stdout:846: snap_remove snap 103 2026-02-25T08:45:29.031 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:29.107 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:29.105+0000 7f777f1f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:29.110 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:29.108+0000 7f777f1f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:29.124 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:45:29.151 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:29.151 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:29.833 INFO:tasks.rados.rados.0.trial023.stdout:846: done (2 left) 2026-02-25T08:45:29.833 INFO:tasks.rados.rados.0.trial023.stdout:847: snap_create 2026-02-25T08:45:29.833 INFO:tasks.rados.rados.0.trial023.stdout:844: expect (ObjNum 387 snap 105 seq_num 387) 2026-02-25T08:45:29.921 INFO:tasks.rados.rados.0.trial023.stdout:843: finishing write tid 1 to trial02342524-2 2026-02-25T08:45:29.922 INFO:tasks.rados.rados.0.trial023.stdout:843: finishing write tid 2 to trial02342524-2 2026-02-25T08:45:29.922 INFO:tasks.rados.rados.0.trial023.stdout:843: finishing write tid 3 to trial02342524-2 2026-02-25T08:45:29.922 INFO:tasks.rados.rados.0.trial023.stdout:843: finishing write tid 4 to trial02342524-2 2026-02-25T08:45:29.922 INFO:tasks.rados.rados.0.trial023.stdout:843: oid 2 updating version 0 to 68 2026-02-25T08:45:29.922 INFO:tasks.rados.rados.0.trial023.stdout:843: oid 2 version 68 is already newer than 68 2026-02-25T08:45:29.922 INFO:tasks.rados.rados.0.trial023.stdout:843: oid 2 version 68 is already newer than 68 2026-02-25T08:45:29.922 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 2 v 68 (ObjNum 397 snap 107 seq_num 397) dirty exists 2026-02-25T08:45:29.922 INFO:tasks.rados.rados.0.trial023.stdout:843: left oid 2 (ObjNum 397 snap 107 seq_num 397) 2026-02-25T08:45:30.069 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:45:30.140 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:45:30.143 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:45:30.144 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:45:30.834 INFO:tasks.rados.rados.0.trial023.stdout:843: done (2 left) 2026-02-25T08:45:30.834 INFO:tasks.rados.rados.0.trial023.stdout:844: done (1 left) 2026-02-25T08:45:30.834 INFO:tasks.rados.rados.0.trial023.stdout:847: done (0 left) 2026-02-25T08:45:30.834 INFO:tasks.rados.rados.0.trial023.stdout:848: write (excl) oid 40 current snap is 108 2026-02-25T08:45:30.834 INFO:tasks.rados.rados.0.trial023.stdout:848: seq_num 398 ranges {689186=799458,1923598=752130} 2026-02-25T08:45:30.839 INFO:tasks.rados.rados.0.trial023.stdout:848: writing trial02342524-40 from 689186 to 1488644 tid 1 2026-02-25T08:45:30.844 INFO:tasks.rados.rados.0.trial023.stdout:848: writing trial02342524-40 from 1923598 to 2675728 tid 2 2026-02-25T08:45:30.844 INFO:tasks.rados.rados.0.trial023.stdout:849: write (excl) oid 48 current snap is 108 2026-02-25T08:45:30.844 INFO:tasks.rados.rados.0.trial023.stdout:849: seq_num 399 ranges {705993=474707,1582080=612141} 2026-02-25T08:45:30.848 INFO:tasks.rados.rados.0.trial023.stdout:849: writing trial02342524-48 from 705993 to 1180700 tid 1 2026-02-25T08:45:30.852 INFO:tasks.rados.rados.0.trial023.stdout:849: writing trial02342524-48 from 1582080 to 2194221 tid 2 2026-02-25T08:45:30.852 INFO:tasks.rados.rados.0.trial023.stdout:850: write (excl) oid 13 current snap is 108 2026-02-25T08:45:30.852 INFO:tasks.rados.rados.0.trial023.stdout:850: seq_num 400 ranges {722800=549956,2040562=682238} 2026-02-25T08:45:30.857 INFO:tasks.rados.rados.0.trial023.stdout:850: writing trial02342524-13 from 722800 to 1272756 tid 1 2026-02-25T08:45:30.863 INFO:tasks.rados.rados.0.trial023.stdout:850: writing trial02342524-13 from 2040562 to 2722800 tid 2 2026-02-25T08:45:30.863 INFO:tasks.rados.rados.0.trial023.stdout:851: write (excl) oid 43 current snap is 108 2026-02-25T08:45:30.863 INFO:tasks.rados.rados.0.trial023.stdout:851: seq_num 401 ranges {739607=625205,1982691=499457} 2026-02-25T08:45:30.867 INFO:tasks.rados.rados.0.trial023.stdout:851: writing trial02342524-43 from 739607 to 1364812 tid 1 2026-02-25T08:45:30.872 INFO:tasks.rados.rados.0.trial023.stdout:851: writing trial02342524-43 from 1982691 to 2482148 tid 2 2026-02-25T08:45:30.872 INFO:tasks.rados.rados.0.trial023.stdout:852: snap_remove snap 102 2026-02-25T08:45:31.153 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:31.229 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:31.227+0000 7fb443b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:31.232 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:31.231+0000 7fb443b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:31.249 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:45:31.277 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:31.277 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:31.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~231s 2026-02-25T08:45:31.837 INFO:tasks.rados.rados.0.trial023.stdout:852: done (4 left) 2026-02-25T08:45:31.837 INFO:tasks.rados.rados.0.trial023.stdout:853: write oid 3 current snap is 108 2026-02-25T08:45:31.837 INFO:tasks.rados.rados.0.trial023.stdout:849: finishing write tid 1 to trial02342524-48 2026-02-25T08:45:31.837 INFO:tasks.rados.rados.0.trial023.stdout:853: seq_num 402 ranges {756414=700454,2041173=715241} 2026-02-25T08:45:31.842 INFO:tasks.rados.rados.0.trial023.stdout:853: writing trial02342524-3 from 756414 to 1456868 tid 1 2026-02-25T08:45:31.848 INFO:tasks.rados.rados.0.trial023.stdout:853: writing trial02342524-3 from 2041173 to 2756414 tid 2 2026-02-25T08:45:31.848 INFO:tasks.rados.rados.0.trial023.stdout:854: snap_create 2026-02-25T08:45:31.848 INFO:tasks.rados.rados.0.trial023.stdout:850: finishing write tid 1 to trial02342524-13 2026-02-25T08:45:31.849 INFO:tasks.rados.rados.0.trial023.stdout:851: finishing write tid 1 to trial02342524-43 2026-02-25T08:45:31.849 INFO:tasks.rados.rados.0.trial023.stdout:849: finishing write tid 2 to trial02342524-48 2026-02-25T08:45:31.849 INFO:tasks.rados.rados.0.trial023.stdout:850: finishing write tid 2 to trial02342524-13 2026-02-25T08:45:31.849 INFO:tasks.rados.rados.0.trial023.stdout:849: finishing write tid 3 to trial02342524-48 2026-02-25T08:45:31.849 INFO:tasks.rados.rados.0.trial023.stdout:849: finishing write tid 4 to trial02342524-48 2026-02-25T08:45:31.849 INFO:tasks.rados.rados.0.trial023.stdout:849: oid 48 updating version 0 to 150 2026-02-25T08:45:31.849 INFO:tasks.rados.rados.0.trial023.stdout:849: oid 48 updating version 150 to 152 2026-02-25T08:45:31.849 INFO:tasks.rados.rados.0.trial023.stdout:849: oid 48 version 152 is already newer than 152 2026-02-25T08:45:31.849 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 48 v 152 (ObjNum 399 snap 108 seq_num 399) dirty exists 2026-02-25T08:45:31.849 INFO:tasks.rados.rados.0.trial023.stdout:849: left oid 48 (ObjNum 399 snap 108 seq_num 399) 2026-02-25T08:45:31.849 INFO:tasks.rados.rados.0.trial023.stdout:850: finishing write tid 3 to trial02342524-13 2026-02-25T08:45:31.849 INFO:tasks.rados.rados.0.trial023.stdout:850: finishing write tid 4 to trial02342524-13 2026-02-25T08:45:31.849 INFO:tasks.rados.rados.0.trial023.stdout:850: oid 13 updating version 0 to 173 2026-02-25T08:45:31.849 INFO:tasks.rados.rados.0.trial023.stdout:850: oid 13 updating version 173 to 175 2026-02-25T08:45:31.849 INFO:tasks.rados.rados.0.trial023.stdout:850: oid 13 version 175 is already newer than 175 2026-02-25T08:45:31.849 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 13 v 175 (ObjNum 400 snap 108 seq_num 400) dirty exists 2026-02-25T08:45:31.850 INFO:tasks.rados.rados.0.trial023.stdout:850: left oid 13 (ObjNum 400 snap 108 seq_num 400) 2026-02-25T08:45:31.850 INFO:tasks.rados.rados.0.trial023.stdout:851: finishing write tid 2 to trial02342524-43 2026-02-25T08:45:31.850 INFO:tasks.rados.rados.0.trial023.stdout:851: finishing write tid 3 to trial02342524-43 2026-02-25T08:45:31.850 INFO:tasks.rados.rados.0.trial023.stdout:851: finishing write tid 4 to trial02342524-43 2026-02-25T08:45:31.850 INFO:tasks.rados.rados.0.trial023.stdout:851: oid 43 updating version 0 to 27 2026-02-25T08:45:31.850 INFO:tasks.rados.rados.0.trial023.stdout:851: oid 43 version 27 is already newer than 27 2026-02-25T08:45:31.850 INFO:tasks.rados.rados.0.trial023.stdout:851: oid 43 version 27 is already newer than 25 2026-02-25T08:45:31.850 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 43 v 27 (ObjNum 401 snap 108 seq_num 401) dirty exists 2026-02-25T08:45:31.850 INFO:tasks.rados.rados.0.trial023.stdout:851: left oid 43 (ObjNum 401 snap 108 seq_num 401) 2026-02-25T08:45:31.850 INFO:tasks.rados.rados.0.trial023.stdout:848: finishing write tid 1 to trial02342524-40 2026-02-25T08:45:31.850 INFO:tasks.rados.rados.0.trial023.stdout:848: finishing write tid 2 to trial02342524-40 2026-02-25T08:45:31.850 INFO:tasks.rados.rados.0.trial023.stdout:848: finishing write tid 3 to trial02342524-40 2026-02-25T08:45:31.850 INFO:tasks.rados.rados.0.trial023.stdout:848: finishing write tid 4 to trial02342524-40 2026-02-25T08:45:31.850 INFO:tasks.rados.rados.0.trial023.stdout:848: oid 40 updating version 0 to 188 2026-02-25T08:45:31.850 INFO:tasks.rados.rados.0.trial023.stdout:848: oid 40 updating version 188 to 190 2026-02-25T08:45:31.850 INFO:tasks.rados.rados.0.trial023.stdout:848: oid 40 version 190 is already newer than 190 2026-02-25T08:45:31.850 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 40 v 190 (ObjNum 398 snap 108 seq_num 398) dirty exists 2026-02-25T08:45:31.850 INFO:tasks.rados.rados.0.trial023.stdout:848: left oid 40 (ObjNum 398 snap 108 seq_num 398) 2026-02-25T08:45:31.887 INFO:tasks.rados.rados.0.trial023.stdout:853: finishing write tid 1 to trial02342524-3 2026-02-25T08:45:31.896 INFO:tasks.rados.rados.0.trial023.stdout:853: finishing write tid 2 to trial02342524-3 2026-02-25T08:45:31.908 INFO:tasks.rados.rados.0.trial023.stdout:853: finishing write tid 3 to trial02342524-3 2026-02-25T08:45:31.910 INFO:tasks.rados.rados.0.trial023.stdout:853: finishing write tid 4 to trial02342524-3 2026-02-25T08:45:31.911 INFO:tasks.rados.rados.0.trial023.stdout:853: oid 3 updating version 0 to 195 2026-02-25T08:45:31.911 INFO:tasks.rados.rados.0.trial023.stdout:853: oid 3 version 195 is already newer than 195 2026-02-25T08:45:31.911 INFO:tasks.rados.rados.0.trial023.stdout:853: oid 3 version 195 is already newer than 195 2026-02-25T08:45:31.911 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 3 v 195 (ObjNum 402 snap 108 seq_num 402) dirty exists 2026-02-25T08:45:31.911 INFO:tasks.rados.rados.0.trial023.stdout:853: left oid 3 (ObjNum 402 snap 108 seq_num 402) 2026-02-25T08:45:32.840 INFO:tasks.rados.rados.0.trial023.stdout:848: done (5 left) 2026-02-25T08:45:32.840 INFO:tasks.rados.rados.0.trial023.stdout:849: done (4 left) 2026-02-25T08:45:32.840 INFO:tasks.rados.rados.0.trial023.stdout:850: done (3 left) 2026-02-25T08:45:32.841 INFO:tasks.rados.rados.0.trial023.stdout:851: done (2 left) 2026-02-25T08:45:32.841 INFO:tasks.rados.rados.0.trial023.stdout:853: done (1 left) 2026-02-25T08:45:32.841 INFO:tasks.rados.rados.0.trial023.stdout:854: done (0 left) 2026-02-25T08:45:32.841 INFO:tasks.rados.rados.0.trial023.stdout:855: snap_create 2026-02-25T08:45:33.279 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:33.356 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:33.354+0000 7f6af2f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:33.359 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:33.358+0000 7f6af2f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:33.373 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:45:33.400 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:33.401 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:33.843 INFO:tasks.rados.rados.0.trial023.stdout:855: done (0 left) 2026-02-25T08:45:33.843 INFO:tasks.rados.rados.0.trial023.stdout:856: read oid 17 snap -1 2026-02-25T08:45:33.843 INFO:tasks.rados.rados.0.trial023.stdout:856: expect deleted 2026-02-25T08:45:33.843 INFO:tasks.rados.rados.0.trial023.stdout:857: read oid 16 snap 108 2026-02-25T08:45:33.844 INFO:tasks.rados.rados.0.trial023.stdout:857: expect (ObjNum 373 snap 100 seq_num 373) 2026-02-25T08:45:33.844 INFO:tasks.rados.rados.0.trial023.stdout:858: snap_create 2026-02-25T08:45:33.889 INFO:tasks.rados.rados.0.trial023.stdout:857: expect (ObjNum 373 snap 100 seq_num 373) 2026-02-25T08:45:33.969 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 217986 2026-02-25T08:45:34.848 INFO:tasks.rados.rados.0.trial023.stdout:856: done (2 left) 2026-02-25T08:45:34.848 INFO:tasks.rados.rados.0.trial023.stdout:857: done (1 left) 2026-02-25T08:45:34.848 INFO:tasks.rados.rados.0.trial023.stdout:858: done (0 left) 2026-02-25T08:45:34.848 INFO:tasks.rados.rados.0.trial023.stdout:859: read oid 46 snap -1 2026-02-25T08:45:34.848 INFO:tasks.rados.rados.0.trial023.stdout:859: expect (ObjNum 386 snap 105 seq_num 386) 2026-02-25T08:45:34.848 INFO:tasks.rados.rados.0.trial023.stdout:860: write oid 22 current snap is 111 2026-02-25T08:45:34.849 INFO:tasks.rados.rados.0.trial023.stdout:860: seq_num 403 ranges {773221=492056,1816008=503126} 2026-02-25T08:45:34.852 INFO:tasks.rados.rados.0.trial023.stdout:860: writing trial02342524-22 from 773221 to 1265277 tid 1 2026-02-25T08:45:34.856 INFO:tasks.rados.rados.0.trial023.stdout:860: writing trial02342524-22 from 1816008 to 2319134 tid 2 2026-02-25T08:45:34.856 INFO:tasks.rados.rados.0.trial023.stdout:861: snap_remove snap 110 2026-02-25T08:45:35.145 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:45:35.216 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:45:35.219 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:45:35.219 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:45:35.402 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:35.478 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:35.476+0000 7f9ea4a51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:35.481 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:35.480+0000 7f9ea4a51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:35.495 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:45:35.522 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:35.523 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:35.851 INFO:tasks.rados.rados.0.trial023.stdout:861: done (2 left) 2026-02-25T08:45:35.852 INFO:tasks.rados.rados.0.trial023.stdout:862: read oid 17 snap 109 2026-02-25T08:45:35.852 INFO:tasks.rados.rados.0.trial023.stdout:862: expect deleted 2026-02-25T08:45:35.852 INFO:tasks.rados.rados.0.trial023.stdout:863: write oid 33 current snap is 111 2026-02-25T08:45:35.852 INFO:tasks.rados.rados.0.trial023.stdout:863: seq_num 404 ranges {790028=567305,1874490=763137} 2026-02-25T08:45:35.856 INFO:tasks.rados.rados.0.trial023.stdout:863: writing trial02342524-33 from 790028 to 1357333 tid 1 2026-02-25T08:45:35.861 INFO:tasks.rados.rados.0.trial023.stdout:863: writing trial02342524-33 from 1874490 to 2637627 tid 2 2026-02-25T08:45:35.861 INFO:tasks.rados.rados.0.trial023.stdout:864: snap_remove snap 108 2026-02-25T08:45:36.855 INFO:tasks.rados.rados.0.trial023.stdout:864: done (4 left) 2026-02-25T08:45:36.856 INFO:tasks.rados.rados.0.trial023.stdout:865: write (excl) oid 50 current snap is 111 2026-02-25T08:45:36.856 INFO:tasks.rados.rados.0.trial023.stdout:865: seq_num 405 ranges {406835=642554,1532972=506795,2772312=34523} 2026-02-25T08:45:36.859 INFO:tasks.rados.rados.0.trial023.stdout:865: writing trial02342524-50 from 406835 to 1049389 tid 1 2026-02-25T08:45:36.863 INFO:tasks.rados.rados.0.trial023.stdout:865: writing trial02342524-50 from 1532972 to 2039767 tid 2 2026-02-25T08:45:36.866 INFO:tasks.rados.rados.0.trial023.stdout:865: writing trial02342524-50 from 2772312 to 2806835 tid 3 2026-02-25T08:45:36.866 INFO:tasks.rados.rados.0.trial023.stdout:866: read oid 13 snap 107 2026-02-25T08:45:36.866 INFO:tasks.rados.rados.0.trial023.stdout:866: expect (ObjNum 350 snap 99 seq_num 350) 2026-02-25T08:45:36.867 INFO:tasks.rados.rados.0.trial023.stdout:867: delete oid 6 current snap is 111 2026-02-25T08:45:36.867 INFO:tasks.rados.rados.0.trial023.stdout:859: expect (ObjNum 386 snap 105 seq_num 386) 2026-02-25T08:45:36.948 INFO:tasks.rados.rados.0.trial023.stdout:860: finishing write tid 1 to trial02342524-22 2026-02-25T08:45:36.948 INFO:tasks.rados.rados.0.trial023.stdout:860: finishing write tid 2 to trial02342524-22 2026-02-25T08:45:36.948 INFO:tasks.rados.rados.0.trial023.stdout:860: finishing write tid 3 to trial02342524-22 2026-02-25T08:45:36.948 INFO:tasks.rados.rados.0.trial023.stdout:860: finishing write tid 4 to trial02342524-22 2026-02-25T08:45:36.948 INFO:tasks.rados.rados.0.trial023.stdout:860: oid 22 updating version 0 to 59 2026-02-25T08:45:36.948 INFO:tasks.rados.rados.0.trial023.stdout:860: oid 22 version 59 is already newer than 59 2026-02-25T08:45:36.948 INFO:tasks.rados.rados.0.trial023.stdout:860: oid 22 version 59 is already newer than 59 2026-02-25T08:45:36.948 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 22 v 59 (ObjNum 403 snap 111 seq_num 403) dirty exists 2026-02-25T08:45:36.949 INFO:tasks.rados.rados.0.trial023.stdout:860: left oid 22 (ObjNum 403 snap 111 seq_num 403) 2026-02-25T08:45:36.949 INFO:tasks.rados.rados.0.trial023.stdout:859: done (6 left) 2026-02-25T08:45:36.949 INFO:tasks.rados.rados.0.trial023.stdout:860: done (5 left) 2026-02-25T08:45:36.949 INFO:tasks.rados.rados.0.trial023.stdout:867: done (4 left) 2026-02-25T08:45:36.949 INFO:tasks.rados.rados.0.trial023.stdout:868: snap_create 2026-02-25T08:45:36.949 INFO:tasks.rados.rados.0.trial023.stdout:863: finishing write tid 1 to trial02342524-33 2026-02-25T08:45:36.949 INFO:tasks.rados.rados.0.trial023.stdout:863: finishing write tid 2 to trial02342524-33 2026-02-25T08:45:36.949 INFO:tasks.rados.rados.0.trial023.stdout:863: finishing write tid 3 to trial02342524-33 2026-02-25T08:45:36.949 INFO:tasks.rados.rados.0.trial023.stdout:863: finishing write tid 4 to trial02342524-33 2026-02-25T08:45:36.949 INFO:tasks.rados.rados.0.trial023.stdout:863: oid 33 updating version 0 to 39 2026-02-25T08:45:36.949 INFO:tasks.rados.rados.0.trial023.stdout:863: oid 33 version 39 is already newer than 39 2026-02-25T08:45:36.949 INFO:tasks.rados.rados.0.trial023.stdout:863: oid 33 version 39 is already newer than 39 2026-02-25T08:45:36.950 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 33 v 39 (ObjNum 404 snap 111 seq_num 404) dirty exists 2026-02-25T08:45:36.950 INFO:tasks.rados.rados.0.trial023.stdout:863: left oid 33 (ObjNum 404 snap 111 seq_num 404) 2026-02-25T08:45:36.950 INFO:tasks.rados.rados.0.trial023.stdout:865: finishing write tid 1 to trial02342524-50 2026-02-25T08:45:36.950 INFO:tasks.rados.rados.0.trial023.stdout:865: finishing write tid 2 to trial02342524-50 2026-02-25T08:45:36.950 INFO:tasks.rados.rados.0.trial023.stdout:866: expect (ObjNum 350 snap 99 seq_num 350) 2026-02-25T08:45:37.003 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2032979 2026-02-25T08:45:37.018 INFO:tasks.rados.rados.0.trial023.stdout:865: finishing write tid 3 to trial02342524-50 2026-02-25T08:45:37.018 INFO:tasks.rados.rados.0.trial023.stdout:865: finishing write tid 4 to trial02342524-50 2026-02-25T08:45:37.018 INFO:tasks.rados.rados.0.trial023.stdout:865: finishing write tid 5 to trial02342524-50 2026-02-25T08:45:37.018 INFO:tasks.rados.rados.0.trial023.stdout:865: oid 50 updating version 0 to 199 2026-02-25T08:45:37.019 INFO:tasks.rados.rados.0.trial023.stdout:865: oid 50 version 199 is already newer than 196 2026-02-25T08:45:37.019 INFO:tasks.rados.rados.0.trial023.stdout:865: oid 50 version 199 is already newer than 199 2026-02-25T08:45:37.019 INFO:tasks.rados.rados.0.trial023.stdout:865: oid 50 version 199 is already newer than 199 2026-02-25T08:45:37.019 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 50 v 199 (ObjNum 405 snap 111 seq_num 405) dirty exists 2026-02-25T08:45:37.019 INFO:tasks.rados.rados.0.trial023.stdout:865: left oid 50 (ObjNum 405 snap 111 seq_num 405) 2026-02-25T08:45:37.080 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~237s 2026-02-25T08:45:37.524 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:37.602 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:37.600+0000 7f5319e3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:37.605 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:37.604+0000 7f5319e3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:37.619 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:45:37.646 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:37.647 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:37.858 INFO:tasks.rados.rados.0.trial023.stdout:862: done (4 left) 2026-02-25T08:45:37.858 INFO:tasks.rados.rados.0.trial023.stdout:863: done (3 left) 2026-02-25T08:45:37.859 INFO:tasks.rados.rados.0.trial023.stdout:865: done (2 left) 2026-02-25T08:45:37.859 INFO:tasks.rados.rados.0.trial023.stdout:866: done (1 left) 2026-02-25T08:45:37.859 INFO:tasks.rados.rados.0.trial023.stdout:868: done (0 left) 2026-02-25T08:45:37.859 INFO:tasks.rados.rados.0.trial023.stdout:869: snap_create 2026-02-25T08:45:38.863 INFO:tasks.rados.rados.0.trial023.stdout:869: done (0 left) 2026-02-25T08:45:38.863 INFO:tasks.rados.rados.0.trial023.stdout:870: snap_remove snap 109 2026-02-25T08:45:39.648 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:39.728 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:39.726+0000 7ff253d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:39.731 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:39.730+0000 7ff253d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:39.746 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:45:39.772 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:39.773 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:39.833 INFO:tasks.rados.rados.0.trial023.stdout:870: done (0 left) 2026-02-25T08:45:39.834 INFO:tasks.rados.rados.0.trial023.stdout:871: read oid 44 snap -1 2026-02-25T08:45:39.834 INFO:tasks.rados.rados.0.trial023.stdout:871: expect deleted 2026-02-25T08:45:39.834 INFO:tasks.rados.rados.0.trial023.stdout:872: read oid 31 snap -1 2026-02-25T08:45:39.834 INFO:tasks.rados.rados.0.trial023.stdout:872: expect deleted 2026-02-25T08:45:39.834 INFO:tasks.rados.rados.0.trial023.stdout:873: read oid 2 snap -1 2026-02-25T08:45:39.834 INFO:tasks.rados.rados.0.trial023.stdout:873: expect (ObjNum 397 snap 107 seq_num 397) 2026-02-25T08:45:39.834 INFO:tasks.rados.rados.0.trial023.stdout:874: write (excl) oid 15 current snap is 113 2026-02-25T08:45:39.834 INFO:tasks.rados.rados.0.trial023.stdout:874: seq_num 407 ranges {440449=793052,1933583=510464} 2026-02-25T08:45:39.838 INFO:tasks.rados.rados.0.trial023.stdout:874: writing trial02342524-15 from 440449 to 1233501 tid 1 2026-02-25T08:45:39.843 INFO:tasks.rados.rados.0.trial023.stdout:874: writing trial02342524-15 from 1933583 to 2444047 tid 2 2026-02-25T08:45:39.843 INFO:tasks.rados.rados.0.trial023.stdout:875: read oid 40 snap -1 2026-02-25T08:45:39.843 INFO:tasks.rados.rados.0.trial023.stdout:875: expect (ObjNum 398 snap 108 seq_num 398) 2026-02-25T08:45:39.843 INFO:tasks.rados.rados.0.trial023.stdout:876: write (excl) oid 13 current snap is 113 2026-02-25T08:45:39.843 INFO:tasks.rados.rados.0.trial023.stdout:876: seq_num 408 ranges {457256=468301,1592065=770475,2854581=2675} 2026-02-25T08:45:39.846 INFO:tasks.rados.rados.0.trial023.stdout:876: writing trial02342524-13 from 457256 to 925557 tid 1 2026-02-25T08:45:39.852 INFO:tasks.rados.rados.0.trial023.stdout:876: writing trial02342524-13 from 1592065 to 2362540 tid 2 2026-02-25T08:45:39.854 INFO:tasks.rados.rados.0.trial023.stdout:876: writing trial02342524-13 from 2854581 to 2857256 tid 3 2026-02-25T08:45:39.854 INFO:tasks.rados.rados.0.trial023.stdout:877: write oid 27 current snap is 113 2026-02-25T08:45:39.854 INFO:tasks.rados.rados.0.trial023.stdout:877: seq_num 409 ranges {474063=543550,1650547=514133,2765651=108412} 2026-02-25T08:45:39.857 INFO:tasks.rados.rados.0.trial023.stdout:877: writing trial02342524-27 from 474063 to 1017613 tid 1 2026-02-25T08:45:39.862 INFO:tasks.rados.rados.0.trial023.stdout:877: writing trial02342524-27 from 1650547 to 2164680 tid 2 2026-02-25T08:45:39.864 INFO:tasks.rados.rados.0.trial023.stdout:877: writing trial02342524-27 from 2765651 to 2874063 tid 3 2026-02-25T08:45:39.865 INFO:tasks.rados.rados.0.trial023.stdout:878: write (excl) oid 20 current snap is 113 2026-02-25T08:45:39.865 INFO:tasks.rados.rados.0.trial023.stdout:878: seq_num 410 ranges {490870=618799,1592676=774144,2793074=97796} 2026-02-25T08:45:39.868 INFO:tasks.rados.rados.0.trial023.stdout:878: writing trial02342524-20 from 490870 to 1109669 tid 1 2026-02-25T08:45:39.873 INFO:tasks.rados.rados.0.trial023.stdout:878: writing trial02342524-20 from 1592676 to 2366820 tid 2 2026-02-25T08:45:39.875 INFO:tasks.rados.rados.0.trial023.stdout:878: writing trial02342524-20 from 2793074 to 2890870 tid 3 2026-02-25T08:45:39.876 INFO:tasks.rados.rados.0.trial023.stdout:879: write (excl) oid 50 current snap is 113 2026-02-25T08:45:39.876 INFO:tasks.rados.rados.0.trial023.stdout:879: seq_num 411 ranges {507677=410401,1367511=517802,2420497=487180} 2026-02-25T08:45:39.879 INFO:tasks.rados.rados.0.trial023.stdout:879: writing trial02342524-50 from 507677 to 918078 tid 1 2026-02-25T08:45:39.882 INFO:tasks.rados.rados.0.trial023.stdout:879: writing trial02342524-50 from 1367511 to 1885313 tid 2 2026-02-25T08:45:39.887 INFO:tasks.rados.rados.0.trial023.stdout:879: writing trial02342524-50 from 2420497 to 2907677 tid 3 2026-02-25T08:45:39.887 INFO:tasks.rados.rados.0.trial023.stdout:880: write oid 48 current snap is 113 2026-02-25T08:45:39.887 INFO:tasks.rados.rados.0.trial023.stdout:880: seq_num 412 ranges {524484=485650,1425993=661460,2847920=76564} 2026-02-25T08:45:39.890 INFO:tasks.rados.rados.0.trial023.stdout:880: writing trial02342524-48 from 524484 to 1010134 tid 1 2026-02-25T08:45:39.894 INFO:tasks.rados.rados.0.trial023.stdout:880: writing trial02342524-48 from 1425993 to 2087453 tid 2 2026-02-25T08:45:39.898 INFO:tasks.rados.rados.0.trial023.stdout:880: writing trial02342524-48 from 2847920 to 2924484 tid 3 2026-02-25T08:45:39.898 INFO:tasks.rados.rados.0.trial023.stdout:881: write oid 7 current snap is 113 2026-02-25T08:45:39.898 INFO:tasks.rados.rados.0.trial023.stdout:881: seq_num 413 ranges {541291=560899,1884475=521471} 2026-02-25T08:45:39.902 INFO:tasks.rados.rados.0.trial023.stdout:881: writing trial02342524-7 from 541291 to 1102190 tid 1 2026-02-25T08:45:39.907 INFO:tasks.rados.rados.0.trial023.stdout:881: writing trial02342524-7 from 1884475 to 2405946 tid 2 2026-02-25T08:45:39.907 INFO:tasks.rados.rados.0.trial023.stdout:882: read oid 23 snap -1 2026-02-25T08:45:39.907 INFO:tasks.rados.rados.0.trial023.stdout:882: expect (ObjNum 385 snap 105 seq_num 385) 2026-02-25T08:45:39.907 INFO:tasks.rados.rados.0.trial023.stdout:883: delete oid 34 current snap is 113 2026-02-25T08:45:39.907 INFO:tasks.rados.rados.0.trial023.stdout:873: expect (ObjNum 397 snap 107 seq_num 397) 2026-02-25T08:45:39.921 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2222009 2026-02-25T08:45:39.989 INFO:tasks.rados.rados.0.trial023.stdout:876: finishing write tid 1 to trial02342524-13 2026-02-25T08:45:39.989 INFO:tasks.rados.rados.0.trial023.stdout:878: finishing write tid 1 to trial02342524-20 2026-02-25T08:45:39.989 INFO:tasks.rados.rados.0.trial023.stdout:877: finishing write tid 1 to trial02342524-27 2026-02-25T08:45:39.989 INFO:tasks.rados.rados.0.trial023.stdout:876: finishing write tid 2 to trial02342524-13 2026-02-25T08:45:39.990 INFO:tasks.rados.rados.0.trial023.stdout:876: finishing write tid 3 to trial02342524-13 2026-02-25T08:45:39.990 INFO:tasks.rados.rados.0.trial023.stdout:877: finishing write tid 2 to trial02342524-27 2026-02-25T08:45:39.990 INFO:tasks.rados.rados.0.trial023.stdout:876: finishing write tid 4 to trial02342524-13 2026-02-25T08:45:39.990 INFO:tasks.rados.rados.0.trial023.stdout:876: finishing write tid 5 to trial02342524-13 2026-02-25T08:45:39.990 INFO:tasks.rados.rados.0.trial023.stdout:876: oid 13 updating version 0 to 177 2026-02-25T08:45:39.990 INFO:tasks.rados.rados.0.trial023.stdout:876: oid 13 updating version 177 to 180 2026-02-25T08:45:39.990 INFO:tasks.rados.rados.0.trial023.stdout:876: oid 13 version 180 is already newer than 180 2026-02-25T08:45:39.990 INFO:tasks.rados.rados.0.trial023.stdout:876: oid 13 version 180 is already newer than 180 2026-02-25T08:45:39.990 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 13 v 180 (ObjNum 408 snap 113 seq_num 408) dirty exists 2026-02-25T08:45:39.990 INFO:tasks.rados.rados.0.trial023.stdout:876: left oid 13 (ObjNum 408 snap 113 seq_num 408) 2026-02-25T08:45:39.990 INFO:tasks.rados.rados.0.trial023.stdout:878: finishing write tid 2 to trial02342524-20 2026-02-25T08:45:39.990 INFO:tasks.rados.rados.0.trial023.stdout:877: finishing write tid 3 to trial02342524-27 2026-02-25T08:45:39.991 INFO:tasks.rados.rados.0.trial023.stdout:882: expect (ObjNum 385 snap 105 seq_num 385) 2026-02-25T08:45:40.037 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:40.068 INFO:tasks.rados.rados.0.trial023.stdout:878: finishing write tid 3 to trial02342524-20 2026-02-25T08:45:40.068 INFO:tasks.rados.rados.0.trial023.stdout:877: finishing write tid 4 to trial02342524-27 2026-02-25T08:45:40.068 INFO:tasks.rados.rados.0.trial023.stdout:877: finishing write tid 5 to trial02342524-27 2026-02-25T08:45:40.068 INFO:tasks.rados.rados.0.trial023.stdout:877: oid 27 updating version 0 to 51 2026-02-25T08:45:40.068 INFO:tasks.rados.rados.0.trial023.stdout:877: oid 27 version 51 is already newer than 51 2026-02-25T08:45:40.068 INFO:tasks.rados.rados.0.trial023.stdout:877: oid 27 version 51 is already newer than 51 2026-02-25T08:45:40.068 INFO:tasks.rados.rados.0.trial023.stdout:877: oid 27 version 51 is already newer than 51 2026-02-25T08:45:40.068 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 27 v 51 (ObjNum 409 snap 113 seq_num 409) dirty exists 2026-02-25T08:45:40.068 INFO:tasks.rados.rados.0.trial023.stdout:877: left oid 27 (ObjNum 409 snap 113 seq_num 409) 2026-02-25T08:45:40.068 INFO:tasks.rados.rados.0.trial023.stdout:878: finishing write tid 4 to trial02342524-20 2026-02-25T08:45:40.069 INFO:tasks.rados.rados.0.trial023.stdout:878: finishing write tid 5 to trial02342524-20 2026-02-25T08:45:40.069 INFO:tasks.rados.rados.0.trial023.stdout:878: oid 20 updating version 0 to 204 2026-02-25T08:45:40.069 INFO:tasks.rados.rados.0.trial023.stdout:878: oid 20 version 204 is already newer than 201 2026-02-25T08:45:40.069 INFO:tasks.rados.rados.0.trial023.stdout:878: oid 20 version 204 is already newer than 204 2026-02-25T08:45:40.069 INFO:tasks.rados.rados.0.trial023.stdout:878: oid 20 version 204 is already newer than 204 2026-02-25T08:45:40.069 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 20 v 204 (ObjNum 410 snap 113 seq_num 410) dirty exists 2026-02-25T08:45:40.069 INFO:tasks.rados.rados.0.trial023.stdout:878: left oid 20 (ObjNum 410 snap 113 seq_num 410) 2026-02-25T08:45:40.069 INFO:tasks.rados.rados.0.trial023.stdout:874: finishing write tid 1 to trial02342524-15 2026-02-25T08:45:40.069 INFO:tasks.rados.rados.0.trial023.stdout:881: finishing write tid 1 to trial02342524-7 2026-02-25T08:45:40.069 INFO:tasks.rados.rados.0.trial023.stdout:879: finishing write tid 1 to trial02342524-50 2026-02-25T08:45:40.069 INFO:tasks.rados.rados.0.trial023.stdout:871: done (12 left) 2026-02-25T08:45:40.069 INFO:tasks.rados.rados.0.trial023.stdout:872: done (11 left) 2026-02-25T08:45:40.070 INFO:tasks.rados.rados.0.trial023.stdout:873: done (10 left) 2026-02-25T08:45:40.070 INFO:tasks.rados.rados.0.trial023.stdout:876: done (9 left) 2026-02-25T08:45:40.070 INFO:tasks.rados.rados.0.trial023.stdout:877: done (8 left) 2026-02-25T08:45:40.070 INFO:tasks.rados.rados.0.trial023.stdout:878: done (7 left) 2026-02-25T08:45:40.070 INFO:tasks.rados.rados.0.trial023.stdout:882: done (6 left) 2026-02-25T08:45:40.070 INFO:tasks.rados.rados.0.trial023.stdout:883: done (5 left) 2026-02-25T08:45:40.070 INFO:tasks.rados.rados.0.trial023.stdout:884: write oid 38 current snap is 113 2026-02-25T08:45:40.070 INFO:tasks.rados.rados.0.trial023.stdout:881: finishing write tid 2 to trial02342524-7 2026-02-25T08:45:40.070 INFO:tasks.rados.rados.0.trial023.stdout:884: seq_num 415 ranges {574905=711397,1885086=525140,2930189=44716} 2026-02-25T08:45:40.071 INFO:tasks.rados.rados.0.trial023.stdout:884: writing trial02342524-38 from 574905 to 1286302 tid 1 2026-02-25T08:45:40.076 INFO:tasks.rados.rados.0.trial023.stdout:884: writing trial02342524-38 from 1885086 to 2410226 tid 2 2026-02-25T08:45:40.079 INFO:tasks.rados.rados.0.trial023.stdout:884: writing trial02342524-38 from 2930189 to 2974905 tid 3 2026-02-25T08:45:40.079 INFO:tasks.rados.rados.0.trial023.stdout:885: read oid 4 snap 112 2026-02-25T08:45:40.079 INFO:tasks.rados.rados.0.trial023.stdout:885: expect (ObjNum 315 snap 93 seq_num 315) 2026-02-25T08:45:40.079 INFO:tasks.rados.rados.0.trial023.stdout:886: read oid 26 snap -1 2026-02-25T08:45:40.079 INFO:tasks.rados.rados.0.trial023.stdout:886: expect (ObjNum 330 snap 93 seq_num 330) 2026-02-25T08:45:40.079 INFO:tasks.rados.rados.0.trial023.stdout:887: snap_create 2026-02-25T08:45:40.079 INFO:tasks.rados.rados.0.trial023.stdout:881: finishing write tid 3 to trial02342524-7 2026-02-25T08:45:40.079 INFO:tasks.rados.rados.0.trial023.stdout:881: finishing write tid 4 to trial02342524-7 2026-02-25T08:45:40.079 INFO:tasks.rados.rados.0.trial023.stdout:881: oid 7 updating version 0 to 136 2026-02-25T08:45:40.080 INFO:tasks.rados.rados.0.trial023.stdout:881: oid 7 version 136 is already newer than 136 2026-02-25T08:45:40.080 INFO:tasks.rados.rados.0.trial023.stdout:881: oid 7 version 136 is already newer than 136 2026-02-25T08:45:40.080 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 7 v 136 (ObjNum 413 snap 113 seq_num 413) dirty exists 2026-02-25T08:45:40.080 INFO:tasks.rados.rados.0.trial023.stdout:881: left oid 7 (ObjNum 413 snap 113 seq_num 413) 2026-02-25T08:45:40.080 INFO:tasks.rados.rados.0.trial023.stdout:874: finishing write tid 2 to trial02342524-15 2026-02-25T08:45:40.080 INFO:tasks.rados.rados.0.trial023.stdout:879: finishing write tid 2 to trial02342524-50 2026-02-25T08:45:40.080 INFO:tasks.rados.rados.0.trial023.stdout:874: finishing write tid 3 to trial02342524-15 2026-02-25T08:45:40.080 INFO:tasks.rados.rados.0.trial023.stdout:874: finishing write tid 4 to trial02342524-15 2026-02-25T08:45:40.080 INFO:tasks.rados.rados.0.trial023.stdout:874: oid 15 updating version 0 to 196 2026-02-25T08:45:40.080 INFO:tasks.rados.rados.0.trial023.stdout:874: oid 15 updating version 196 to 198 2026-02-25T08:45:40.080 INFO:tasks.rados.rados.0.trial023.stdout:874: oid 15 version 198 is already newer than 198 2026-02-25T08:45:40.080 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 15 v 198 (ObjNum 407 snap 113 seq_num 407) dirty exists 2026-02-25T08:45:40.080 INFO:tasks.rados.rados.0.trial023.stdout:874: left oid 15 (ObjNum 407 snap 113 seq_num 407) 2026-02-25T08:45:40.081 INFO:tasks.rados.rados.0.trial023.stdout:875: expect (ObjNum 398 snap 108 seq_num 398) 2026-02-25T08:45:40.094 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2656656 2026-02-25T08:45:40.129 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 867716 2026-02-25T08:45:40.151 INFO:tasks.rados.rados.0.trial023.stdout:879: finishing write tid 3 to trial02342524-50 2026-02-25T08:45:40.152 INFO:tasks.rados.rados.0.trial023.stdout:879: finishing write tid 4 to trial02342524-50 2026-02-25T08:45:40.152 INFO:tasks.rados.rados.0.trial023.stdout:879: finishing write tid 5 to trial02342524-50 2026-02-25T08:45:40.152 INFO:tasks.rados.rados.0.trial023.stdout:879: oid 50 updating version 0 to 209 2026-02-25T08:45:40.152 INFO:tasks.rados.rados.0.trial023.stdout:879: oid 50 version 209 is already newer than 209 2026-02-25T08:45:40.152 INFO:tasks.rados.rados.0.trial023.stdout:879: oid 50 version 209 is already newer than 206 2026-02-25T08:45:40.152 INFO:tasks.rados.rados.0.trial023.stdout:879: oid 50 version 209 is already newer than 209 2026-02-25T08:45:40.152 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 50 v 209 (ObjNum 411 snap 113 seq_num 411) dirty exists 2026-02-25T08:45:40.152 INFO:tasks.rados.rados.0.trial023.stdout:879: left oid 50 (ObjNum 411 snap 113 seq_num 411) 2026-02-25T08:45:40.152 INFO:tasks.rados.rados.0.trial023.stdout:880: finishing write tid 1 to trial02342524-48 2026-02-25T08:45:40.152 INFO:tasks.rados.rados.0.trial023.stdout:880: finishing write tid 2 to trial02342524-48 2026-02-25T08:45:40.152 INFO:tasks.rados.rados.0.trial023.stdout:880: finishing write tid 3 to trial02342524-48 2026-02-25T08:45:40.152 INFO:tasks.rados.rados.0.trial023.stdout:880: finishing write tid 4 to trial02342524-48 2026-02-25T08:45:40.153 INFO:tasks.rados.rados.0.trial023.stdout:880: finishing write tid 5 to trial02342524-48 2026-02-25T08:45:40.153 INFO:tasks.rados.rados.0.trial023.stdout:880: oid 48 updating version 0 to 159 2026-02-25T08:45:40.153 INFO:tasks.rados.rados.0.trial023.stdout:880: oid 48 version 159 is already newer than 159 2026-02-25T08:45:40.153 INFO:tasks.rados.rados.0.trial023.stdout:880: oid 48 version 159 is already newer than 159 2026-02-25T08:45:40.153 INFO:tasks.rados.rados.0.trial023.stdout:880: oid 48 version 159 is already newer than 159 2026-02-25T08:45:40.153 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 48 v 159 (ObjNum 412 snap 113 seq_num 412) dirty exists 2026-02-25T08:45:40.153 INFO:tasks.rados.rados.0.trial023.stdout:880: left oid 48 (ObjNum 412 snap 113 seq_num 412) 2026-02-25T08:45:40.153 INFO:tasks.rados.rados.0.trial023.stdout:886: expect (ObjNum 330 snap 93 seq_num 330) 2026-02-25T08:45:40.171 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 3358857 2026-02-25T08:45:40.207 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:40.220 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:45:40.246 INFO:tasks.rados.rados.0.trial023.stdout:884: finishing write tid 1 to trial02342524-38 2026-02-25T08:45:40.246 INFO:tasks.rados.rados.0.trial023.stdout:885: expect (ObjNum 315 snap 93 seq_num 315) 2026-02-25T08:45:40.259 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1106399 2026-02-25T08:45:40.290 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:45:40.293 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:40.293 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:45:40.293 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:45:40.343 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:40.395 INFO:tasks.rados.rados.0.trial023.stdout:884: finishing write tid 2 to trial02342524-38 2026-02-25T08:45:40.395 INFO:tasks.rados.rados.0.trial023.stdout:884: finishing write tid 3 to trial02342524-38 2026-02-25T08:45:40.395 INFO:tasks.rados.rados.0.trial023.stdout:884: finishing write tid 4 to trial02342524-38 2026-02-25T08:45:40.395 INFO:tasks.rados.rados.0.trial023.stdout:884: finishing write tid 5 to trial02342524-38 2026-02-25T08:45:40.395 INFO:tasks.rados.rados.0.trial023.stdout:884: oid 38 updating version 0 to 156 2026-02-25T08:45:40.395 INFO:tasks.rados.rados.0.trial023.stdout:884: oid 38 version 156 is already newer than 156 2026-02-25T08:45:40.395 INFO:tasks.rados.rados.0.trial023.stdout:884: oid 38 version 156 is already newer than 156 2026-02-25T08:45:40.395 INFO:tasks.rados.rados.0.trial023.stdout:884: oid 38 version 156 is already newer than 156 2026-02-25T08:45:40.395 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 38 v 156 (ObjNum 415 snap 113 seq_num 415) dirty exists 2026-02-25T08:45:40.395 INFO:tasks.rados.rados.0.trial023.stdout:884: left oid 38 (ObjNum 415 snap 113 seq_num 415) 2026-02-25T08:45:40.866 INFO:tasks.rados.rados.0.trial023.stdout:874: done (8 left) 2026-02-25T08:45:40.866 INFO:tasks.rados.rados.0.trial023.stdout:875: done (7 left) 2026-02-25T08:45:40.866 INFO:tasks.rados.rados.0.trial023.stdout:879: done (6 left) 2026-02-25T08:45:40.867 INFO:tasks.rados.rados.0.trial023.stdout:880: done (5 left) 2026-02-25T08:45:40.867 INFO:tasks.rados.rados.0.trial023.stdout:881: done (4 left) 2026-02-25T08:45:40.867 INFO:tasks.rados.rados.0.trial023.stdout:884: done (3 left) 2026-02-25T08:45:40.867 INFO:tasks.rados.rados.0.trial023.stdout:885: done (2 left) 2026-02-25T08:45:40.867 INFO:tasks.rados.rados.0.trial023.stdout:886: done (1 left) 2026-02-25T08:45:40.867 INFO:tasks.rados.rados.0.trial023.stdout:887: done (0 left) 2026-02-25T08:45:40.867 INFO:tasks.rados.rados.0.trial023.stdout:888: write oid 44 current snap is 114 2026-02-25T08:45:40.867 INFO:tasks.rados.rados.0.trial023.stdout:888: seq_num 416 ranges {591712=786646,1943568=668798} 2026-02-25T08:45:40.871 INFO:tasks.rados.rados.0.trial023.stdout:888: writing trial02342524-44 from 591712 to 1378358 tid 1 2026-02-25T08:45:40.876 INFO:tasks.rados.rados.0.trial023.stdout:888: writing trial02342524-44 from 1943568 to 2612366 tid 2 2026-02-25T08:45:40.876 INFO:tasks.rados.rados.0.trial023.stdout:889: read oid 18 snap -1 2026-02-25T08:45:40.876 INFO:tasks.rados.rados.0.trial023.stdout:889: expect deleted 2026-02-25T08:45:40.876 INFO:tasks.rados.rados.0.trial023.stdout:890: read oid 34 snap -1 2026-02-25T08:45:40.877 INFO:tasks.rados.rados.0.trial023.stdout:890: expect deleted 2026-02-25T08:45:40.877 INFO:tasks.rados.rados.0.trial023.stdout:891: read oid 46 snap -1 2026-02-25T08:45:40.877 INFO:tasks.rados.rados.0.trial023.stdout:891: expect (ObjNum 386 snap 105 seq_num 386) 2026-02-25T08:45:40.877 INFO:tasks.rados.rados.0.trial023.stdout:892: read oid 32 snap -1 2026-02-25T08:45:40.877 INFO:tasks.rados.rados.0.trial023.stdout:892: expect deleted 2026-02-25T08:45:40.877 INFO:tasks.rados.rados.0.trial023.stdout:893: read oid 8 snap -1 2026-02-25T08:45:40.877 INFO:tasks.rados.rados.0.trial023.stdout:893: expect (ObjNum 395 snap 106 seq_num 395) 2026-02-25T08:45:40.877 INFO:tasks.rados.rados.0.trial023.stdout:894: write (excl) oid 1 current snap is 114 2026-02-25T08:45:40.877 INFO:tasks.rados.rados.0.trial023.stdout:894: seq_num 417 ranges {608519=461895,1602050=528809,2585035=423484} 2026-02-25T08:45:40.880 INFO:tasks.rados.rados.0.trial023.stdout:894: writing trial02342524-1 from 608519 to 1070414 tid 1 2026-02-25T08:45:40.883 INFO:tasks.rados.rados.0.trial023.stdout:894: writing trial02342524-1 from 1602050 to 2130859 tid 2 2026-02-25T08:45:40.887 INFO:tasks.rados.rados.0.trial023.stdout:894: writing trial02342524-1 from 2585035 to 3008519 tid 3 2026-02-25T08:45:40.888 INFO:tasks.rados.rados.0.trial023.stdout:895: snap_remove snap 111 2026-02-25T08:45:41.774 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:41.851 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:41.848+0000 7ffa6165a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:41.854 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:41.852+0000 7ffa6165a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:41.872 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:45:41.873 INFO:tasks.rados.rados.0.trial023.stdout:895: done (7 left) 2026-02-25T08:45:41.873 INFO:tasks.rados.rados.0.trial023.stdout:896: read oid 45 snap 107 2026-02-25T08:45:41.873 INFO:tasks.rados.rados.0.trial023.stdout:896: expect (ObjNum 341 snap 97 seq_num 341) 2026-02-25T08:45:41.874 INFO:tasks.rados.rados.0.trial023.stdout:897: write (excl) oid 39 current snap is 114 2026-02-25T08:45:41.874 INFO:tasks.rados.rados.0.trial023.stdout:897: seq_num 418 ranges {625326=537144,1944179=672467} 2026-02-25T08:45:41.878 INFO:tasks.rados.rados.0.trial023.stdout:897: writing trial02342524-39 from 625326 to 1162470 tid 1 2026-02-25T08:45:41.884 INFO:tasks.rados.rados.0.trial023.stdout:897: writing trial02342524-39 from 1944179 to 2616646 tid 2 2026-02-25T08:45:41.884 INFO:tasks.rados.rados.0.trial023.stdout:898: delete oid 21 current snap is 114 2026-02-25T08:45:41.884 INFO:tasks.rados.rados.0.trial023.stdout:888: finishing write tid 1 to trial02342524-44 2026-02-25T08:45:41.884 INFO:tasks.rados.rados.0.trial023.stdout:891: expect (ObjNum 386 snap 105 seq_num 386) 2026-02-25T08:45:41.890 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1249101 2026-02-25T08:45:41.907 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:41.924 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:41.924 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:41.949 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2137096 2026-02-25T08:45:41.957 INFO:tasks.rados.rados.0.trial023.stdout:888: finishing write tid 2 to trial02342524-44 2026-02-25T08:45:41.957 INFO:tasks.rados.rados.0.trial023.stdout:893: expect (ObjNum 395 snap 106 seq_num 395) 2026-02-25T08:45:41.957 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:42.031 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:42.073 INFO:tasks.rados.rados.0.trial023.stdout:888: finishing write tid 3 to trial02342524-44 2026-02-25T08:45:42.073 INFO:tasks.rados.rados.0.trial023.stdout:888: finishing write tid 4 to trial02342524-44 2026-02-25T08:45:42.073 INFO:tasks.rados.rados.0.trial023.stdout:888: oid 44 updating version 0 to 28 2026-02-25T08:45:42.073 INFO:tasks.rados.rados.0.trial023.stdout:888: oid 44 version 28 is already newer than 28 2026-02-25T08:45:42.073 INFO:tasks.rados.rados.0.trial023.stdout:888: oid 44 version 28 is already newer than 28 2026-02-25T08:45:42.074 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 44 v 28 (ObjNum 416 snap 114 seq_num 416) dirty exists 2026-02-25T08:45:42.074 INFO:tasks.rados.rados.0.trial023.stdout:888: left oid 44 (ObjNum 416 snap 114 seq_num 416) 2026-02-25T08:45:42.074 INFO:tasks.rados.rados.0.trial023.stdout:894: finishing write tid 1 to trial02342524-1 2026-02-25T08:45:42.074 INFO:tasks.rados.rados.0.trial023.stdout:888: done (9 left) 2026-02-25T08:45:42.074 INFO:tasks.rados.rados.0.trial023.stdout:889: done (8 left) 2026-02-25T08:45:42.074 INFO:tasks.rados.rados.0.trial023.stdout:890: done (7 left) 2026-02-25T08:45:42.074 INFO:tasks.rados.rados.0.trial023.stdout:891: done (6 left) 2026-02-25T08:45:42.074 INFO:tasks.rados.rados.0.trial023.stdout:892: done (5 left) 2026-02-25T08:45:42.074 INFO:tasks.rados.rados.0.trial023.stdout:893: done (4 left) 2026-02-25T08:45:42.074 INFO:tasks.rados.rados.0.trial023.stdout:898: done (3 left) 2026-02-25T08:45:42.074 INFO:tasks.rados.rados.0.trial023.stdout:899: snap_create 2026-02-25T08:45:42.074 INFO:tasks.rados.rados.0.trial023.stdout:894: finishing write tid 2 to trial02342524-1 2026-02-25T08:45:42.074 INFO:tasks.rados.rados.0.trial023.stdout:894: finishing write tid 3 to trial02342524-1 2026-02-25T08:45:42.075 INFO:tasks.rados.rados.0.trial023.stdout:894: finishing write tid 4 to trial02342524-1 2026-02-25T08:45:42.075 INFO:tasks.rados.rados.0.trial023.stdout:894: finishing write tid 5 to trial02342524-1 2026-02-25T08:45:42.075 INFO:tasks.rados.rados.0.trial023.stdout:894: oid 1 updating version 0 to 151 2026-02-25T08:45:42.075 INFO:tasks.rados.rados.0.trial023.stdout:894: oid 1 version 151 is already newer than 151 2026-02-25T08:45:42.075 INFO:tasks.rados.rados.0.trial023.stdout:894: oid 1 version 151 is already newer than 151 2026-02-25T08:45:42.075 INFO:tasks.rados.rados.0.trial023.stdout:894: oid 1 version 151 is already newer than 148 2026-02-25T08:45:42.075 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 1 v 151 (ObjNum 417 snap 114 seq_num 417) dirty exists 2026-02-25T08:45:42.075 INFO:tasks.rados.rados.0.trial023.stdout:894: left oid 1 (ObjNum 417 snap 114 seq_num 417) 2026-02-25T08:45:42.075 INFO:tasks.rados.rados.0.trial023.stdout:896: expect (ObjNum 341 snap 97 seq_num 341) 2026-02-25T08:45:42.167 INFO:tasks.rados.rados.0.trial023.stdout:897: finishing write tid 1 to trial02342524-39 2026-02-25T08:45:42.167 INFO:tasks.rados.rados.0.trial023.stdout:897: finishing write tid 2 to trial02342524-39 2026-02-25T08:45:42.167 INFO:tasks.rados.rados.0.trial023.stdout:897: finishing write tid 3 to trial02342524-39 2026-02-25T08:45:42.167 INFO:tasks.rados.rados.0.trial023.stdout:897: finishing write tid 4 to trial02342524-39 2026-02-25T08:45:42.168 INFO:tasks.rados.rados.0.trial023.stdout:897: oid 39 updating version 0 to 165 2026-02-25T08:45:42.168 INFO:tasks.rados.rados.0.trial023.stdout:897: oid 39 version 165 is already newer than 161 2026-02-25T08:45:42.168 INFO:tasks.rados.rados.0.trial023.stdout:897: oid 39 version 165 is already newer than 165 2026-02-25T08:45:42.168 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 39 v 165 (ObjNum 418 snap 114 seq_num 418) dirty exists 2026-02-25T08:45:42.168 INFO:tasks.rados.rados.0.trial023.stdout:897: left oid 39 (ObjNum 418 snap 114 seq_num 418) 2026-02-25T08:45:42.585 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~242s 2026-02-25T08:45:42.878 INFO:tasks.rados.rados.0.trial023.stdout:894: done (3 left) 2026-02-25T08:45:42.878 INFO:tasks.rados.rados.0.trial023.stdout:896: done (2 left) 2026-02-25T08:45:42.878 INFO:tasks.rados.rados.0.trial023.stdout:897: done (1 left) 2026-02-25T08:45:42.878 INFO:tasks.rados.rados.0.trial023.stdout:899: done (0 left) 2026-02-25T08:45:42.878 INFO:tasks.rados.rados.0.trial023.stdout:900: snap_remove snap 112 2026-02-25T08:45:43.880 INFO:tasks.rados.rados.0.trial023.stdout:900: done (0 left) 2026-02-25T08:45:43.881 INFO:tasks.rados.rados.0.trial023.stdout:901: write (excl) oid 32 current snap is 115 2026-02-25T08:45:43.881 INFO:tasks.rados.rados.0.trial023.stdout:901: seq_num 420 ranges {658940=403995,1777496=676136,2950951=107989} 2026-02-25T08:45:43.884 INFO:tasks.rados.rados.0.trial023.stdout:901: writing trial02342524-32 from 658940 to 1062935 tid 1 2026-02-25T08:45:43.890 INFO:tasks.rados.rados.0.trial023.stdout:901: writing trial02342524-32 from 1777496 to 2453632 tid 2 2026-02-25T08:45:43.892 INFO:tasks.rados.rados.0.trial023.stdout:901: writing trial02342524-32 from 2950951 to 3058940 tid 3 2026-02-25T08:45:43.892 INFO:tasks.rados.rados.0.trial023.stdout:902: snap_remove snap 107 2026-02-25T08:45:43.926 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:44.002 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:43.998+0000 7fc17045a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:44.004 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:44.002+0000 7fc17045a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:44.019 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:45:44.045 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:44.046 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:44.885 INFO:tasks.rados.rados.0.trial023.stdout:902: done (1 left) 2026-02-25T08:45:44.886 INFO:tasks.rados.rados.0.trial023.stdout:903: snap_remove snap 114 2026-02-25T08:45:45.294 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:45:45.363 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:45:45.366 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:45:45.367 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:45:45.887 INFO:tasks.rados.rados.0.trial023.stdout:903: done (1 left) 2026-02-25T08:45:45.887 INFO:tasks.rados.rados.0.trial023.stdout:904: delete oid 37 current snap is 115 2026-02-25T08:45:45.887 INFO:tasks.rados.rados.0.trial023.stdout:901: finishing write tid 1 to trial02342524-32 2026-02-25T08:45:45.887 INFO:tasks.rados.rados.0.trial023.stdout:901: finishing write tid 2 to trial02342524-32 2026-02-25T08:45:45.887 INFO:tasks.rados.rados.0.trial023.stdout:901: finishing write tid 3 to trial02342524-32 2026-02-25T08:45:45.887 INFO:tasks.rados.rados.0.trial023.stdout:901: finishing write tid 4 to trial02342524-32 2026-02-25T08:45:45.887 INFO:tasks.rados.rados.0.trial023.stdout:901: finishing write tid 5 to trial02342524-32 2026-02-25T08:45:45.887 INFO:tasks.rados.rados.0.trial023.stdout:901: oid 32 updating version 0 to 140 2026-02-25T08:45:45.888 INFO:tasks.rados.rados.0.trial023.stdout:901: oid 32 version 140 is already newer than 140 2026-02-25T08:45:45.888 INFO:tasks.rados.rados.0.trial023.stdout:901: oid 32 version 140 is already newer than 140 2026-02-25T08:45:45.888 INFO:tasks.rados.rados.0.trial023.stdout:901: oid 32 version 140 is already newer than 137 2026-02-25T08:45:45.888 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 32 v 140 (ObjNum 420 snap 115 seq_num 420) dirty exists 2026-02-25T08:45:45.888 INFO:tasks.rados.rados.0.trial023.stdout:901: left oid 32 (ObjNum 420 snap 115 seq_num 420) 2026-02-25T08:45:45.927 INFO:tasks.rados.rados.0.trial023.stdout:901: done (1 left) 2026-02-25T08:45:45.927 INFO:tasks.rados.rados.0.trial023.stdout:904: done (0 left) 2026-02-25T08:45:45.928 INFO:tasks.rados.rados.0.trial023.stdout:905: write oid 25 current snap is 115 2026-02-25T08:45:45.928 INFO:tasks.rados.rados.0.trial023.stdout:905: seq_num 422 ranges {692554=554493,1778107=679805,2889444=203110} 2026-02-25T08:45:45.932 INFO:tasks.rados.rados.0.trial023.stdout:905: writing trial02342524-25 from 692554 to 1247047 tid 1 2026-02-25T08:45:45.936 INFO:tasks.rados.rados.0.trial023.stdout:905: writing trial02342524-25 from 1778107 to 2457912 tid 2 2026-02-25T08:45:45.939 INFO:tasks.rados.rados.0.trial023.stdout:905: writing trial02342524-25 from 2889444 to 3092554 tid 3 2026-02-25T08:45:45.939 INFO:tasks.rados.rados.0.trial023.stdout:906: read oid 48 snap -1 2026-02-25T08:45:45.939 INFO:tasks.rados.rados.0.trial023.stdout:906: expect (ObjNum 412 snap 113 seq_num 412) 2026-02-25T08:45:45.939 INFO:tasks.rados.rados.0.trial023.stdout:907: snap_create 2026-02-25T08:45:45.962 INFO:tasks.rados.rados.0.trial023.stdout:906: expect (ObjNum 412 snap 113 seq_num 412) 2026-02-25T08:45:45.964 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 624723 2026-02-25T08:45:46.046 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:46.074 INFO:tasks.rados.rados.0.trial023.stdout:905: finishing write tid 1 to trial02342524-25 2026-02-25T08:45:46.074 INFO:tasks.rados.rados.0.trial023.stdout:905: finishing write tid 2 to trial02342524-25 2026-02-25T08:45:46.074 INFO:tasks.rados.rados.0.trial023.stdout:905: finishing write tid 3 to trial02342524-25 2026-02-25T08:45:46.075 INFO:tasks.rados.rados.0.trial023.stdout:905: finishing write tid 4 to trial02342524-25 2026-02-25T08:45:46.075 INFO:tasks.rados.rados.0.trial023.stdout:905: finishing write tid 5 to trial02342524-25 2026-02-25T08:45:46.075 INFO:tasks.rados.rados.0.trial023.stdout:905: oid 25 updating version 0 to 214 2026-02-25T08:45:46.075 INFO:tasks.rados.rados.0.trial023.stdout:905: oid 25 version 214 is already newer than 214 2026-02-25T08:45:46.075 INFO:tasks.rados.rados.0.trial023.stdout:905: oid 25 version 214 is already newer than 214 2026-02-25T08:45:46.075 INFO:tasks.rados.rados.0.trial023.stdout:905: oid 25 version 214 is already newer than 214 2026-02-25T08:45:46.075 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 25 v 214 (ObjNum 422 snap 115 seq_num 422) dirty exists 2026-02-25T08:45:46.075 INFO:tasks.rados.rados.0.trial023.stdout:905: left oid 25 (ObjNum 422 snap 115 seq_num 422) 2026-02-25T08:45:46.119 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:46.116+0000 7fa16c7b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:46.122 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:46.120+0000 7fa16c7b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:46.137 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:45:46.163 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:46.164 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:46.893 INFO:tasks.rados.rados.0.trial023.stdout:905: done (2 left) 2026-02-25T08:45:46.893 INFO:tasks.rados.rados.0.trial023.stdout:906: done (1 left) 2026-02-25T08:45:46.893 INFO:tasks.rados.rados.0.trial023.stdout:907: done (0 left) 2026-02-25T08:45:46.893 INFO:tasks.rados.rados.0.trial023.stdout:908: read oid 31 snap 113 2026-02-25T08:45:46.893 INFO:tasks.rados.rados.0.trial023.stdout:908: expect deleted 2026-02-25T08:45:46.893 INFO:tasks.rados.rados.0.trial023.stdout:909: write oid 25 current snap is 116 2026-02-25T08:45:46.894 INFO:tasks.rados.rados.0.trial023.stdout:909: seq_num 423 ranges {709361=629742,1836589=539816,3033220=76141} 2026-02-25T08:45:46.897 INFO:tasks.rados.rados.0.trial023.stdout:909: writing trial02342524-25 from 709361 to 1339103 tid 1 2026-02-25T08:45:46.901 INFO:tasks.rados.rados.0.trial023.stdout:909: writing trial02342524-25 from 1836589 to 2376405 tid 2 2026-02-25T08:45:46.904 INFO:tasks.rados.rados.0.trial023.stdout:909: writing trial02342524-25 from 3033220 to 3109361 tid 3 2026-02-25T08:45:46.904 INFO:tasks.rados.rados.0.trial023.stdout:910: write oid 47 current snap is 116 2026-02-25T08:45:46.904 INFO:tasks.rados.rados.0.trial023.stdout:910: seq_num 424 ranges {726168=704991,1895071=683474} 2026-02-25T08:45:46.909 INFO:tasks.rados.rados.0.trial023.stdout:910: writing trial02342524-47 from 726168 to 1431159 tid 1 2026-02-25T08:45:46.914 INFO:tasks.rados.rados.0.trial023.stdout:910: writing trial02342524-47 from 1895071 to 2578545 tid 2 2026-02-25T08:45:46.914 INFO:tasks.rados.rados.0.trial023.stdout:911: snap_remove snap 115 2026-02-25T08:45:47.896 INFO:tasks.rados.rados.0.trial023.stdout:911: done (3 left) 2026-02-25T08:45:47.896 INFO:tasks.rados.rados.0.trial023.stdout:912: read oid 50 snap -1 2026-02-25T08:45:47.896 INFO:tasks.rados.rados.0.trial023.stdout:912: expect (ObjNum 411 snap 113 seq_num 411) 2026-02-25T08:45:47.896 INFO:tasks.rados.rados.0.trial023.stdout:913: read oid 20 snap -1 2026-02-25T08:45:47.896 INFO:tasks.rados.rados.0.trial023.stdout:913: expect (ObjNum 410 snap 113 seq_num 410) 2026-02-25T08:45:47.896 INFO:tasks.rados.rados.0.trial023.stdout:914: write (excl) oid 2 current snap is 116 2026-02-25T08:45:47.896 INFO:tasks.rados.rados.0.trial023.stdout:914: seq_num 425 ranges {742975=780240,1953553=427132,2971713=171262} 2026-02-25T08:45:47.901 INFO:tasks.rados.rados.0.trial023.stdout:914: writing trial02342524-2 from 742975 to 1523215 tid 1 2026-02-25T08:45:47.904 INFO:tasks.rados.rados.0.trial023.stdout:914: writing trial02342524-2 from 1953553 to 2380685 tid 2 2026-02-25T08:45:47.907 INFO:tasks.rados.rados.0.trial023.stdout:914: writing trial02342524-2 from 2971713 to 3142975 tid 3 2026-02-25T08:45:47.907 INFO:tasks.rados.rados.0.trial023.stdout:915: write (excl) oid 10 current snap is 116 2026-02-25T08:45:47.907 INFO:tasks.rados.rados.0.trial023.stdout:915: seq_num 426 ranges {759782=571842,2012035=687143} 2026-02-25T08:45:47.912 INFO:tasks.rados.rados.0.trial023.stdout:915: writing trial02342524-10 from 759782 to 1331624 tid 1 2026-02-25T08:45:47.918 INFO:tasks.rados.rados.0.trial023.stdout:915: writing trial02342524-10 from 2012035 to 2699178 tid 2 2026-02-25T08:45:47.918 INFO:tasks.rados.rados.0.trial023.stdout:916: delete oid 46 current snap is 116 2026-02-25T08:45:47.918 INFO:tasks.rados.rados.0.trial023.stdout:909: finishing write tid 1 to trial02342524-25 2026-02-25T08:45:47.918 INFO:tasks.rados.rados.0.trial023.stdout:909: finishing write tid 2 to trial02342524-25 2026-02-25T08:45:47.918 INFO:tasks.rados.rados.0.trial023.stdout:910: finishing write tid 1 to trial02342524-47 2026-02-25T08:45:47.918 INFO:tasks.rados.rados.0.trial023.stdout:909: finishing write tid 3 to trial02342524-25 2026-02-25T08:45:47.919 INFO:tasks.rados.rados.0.trial023.stdout:910: finishing write tid 2 to trial02342524-47 2026-02-25T08:45:47.919 INFO:tasks.rados.rados.0.trial023.stdout:909: finishing write tid 4 to trial02342524-25 2026-02-25T08:45:47.919 INFO:tasks.rados.rados.0.trial023.stdout:909: finishing write tid 5 to trial02342524-25 2026-02-25T08:45:47.919 INFO:tasks.rados.rados.0.trial023.stdout:909: oid 25 updating version 0 to 219 2026-02-25T08:45:47.919 INFO:tasks.rados.rados.0.trial023.stdout:909: oid 25 version 219 is already newer than 219 2026-02-25T08:45:47.919 INFO:tasks.rados.rados.0.trial023.stdout:909: oid 25 version 219 is already newer than 219 2026-02-25T08:45:47.919 INFO:tasks.rados.rados.0.trial023.stdout:909: oid 25 version 219 is already newer than 219 2026-02-25T08:45:47.919 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 25 v 219 (ObjNum 423 snap 116 seq_num 423) dirty exists 2026-02-25T08:45:47.919 INFO:tasks.rados.rados.0.trial023.stdout:909: left oid 25 (ObjNum 423 snap 116 seq_num 423) 2026-02-25T08:45:47.919 INFO:tasks.rados.rados.0.trial023.stdout:910: finishing write tid 3 to trial02342524-47 2026-02-25T08:45:47.919 INFO:tasks.rados.rados.0.trial023.stdout:910: finishing write tid 4 to trial02342524-47 2026-02-25T08:45:47.919 INFO:tasks.rados.rados.0.trial023.stdout:910: oid 47 updating version 0 to 62 2026-02-25T08:45:47.919 INFO:tasks.rados.rados.0.trial023.stdout:910: oid 47 version 62 is already newer than 62 2026-02-25T08:45:47.919 INFO:tasks.rados.rados.0.trial023.stdout:910: oid 47 version 62 is already newer than 62 2026-02-25T08:45:47.919 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 47 v 62 (ObjNum 424 snap 116 seq_num 424) dirty exists 2026-02-25T08:45:47.919 INFO:tasks.rados.rados.0.trial023.stdout:910: left oid 47 (ObjNum 424 snap 116 seq_num 424) 2026-02-25T08:45:47.947 INFO:tasks.rados.rados.0.trial023.stdout:915: finishing write tid 1 to trial02342524-10 2026-02-25T08:45:47.955 INFO:tasks.rados.rados.0.trial023.stdout:912: expect (ObjNum 411 snap 113 seq_num 411) 2026-02-25T08:45:47.987 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1184275 2026-02-25T08:45:48.039 INFO:tasks.rados.rados.0.trial023.stdout:908: done (7 left) 2026-02-25T08:45:48.039 INFO:tasks.rados.rados.0.trial023.stdout:909: done (6 left) 2026-02-25T08:45:48.039 INFO:tasks.rados.rados.0.trial023.stdout:910: done (5 left) 2026-02-25T08:45:48.039 INFO:tasks.rados.rados.0.trial023.stdout:912: done (4 left) 2026-02-25T08:45:48.039 INFO:tasks.rados.rados.0.trial023.stdout:916: done (3 left) 2026-02-25T08:45:48.039 INFO:tasks.rados.rados.0.trial023.stdout:917: write (excl) oid 36 current snap is 116 2026-02-25T08:45:48.039 INFO:tasks.rados.rados.0.trial023.stdout:914: finishing write tid 1 to trial02342524-2 2026-02-25T08:45:48.039 INFO:tasks.rados.rados.0.trial023.stdout:915: finishing write tid 2 to trial02342524-10 2026-02-25T08:45:48.039 INFO:tasks.rados.rados.0.trial023.stdout:915: finishing write tid 3 to trial02342524-10 2026-02-25T08:45:48.039 INFO:tasks.rados.rados.0.trial023.stdout:915: finishing write tid 4 to trial02342524-10 2026-02-25T08:45:48.039 INFO:tasks.rados.rados.0.trial023.stdout:915: oid 10 updating version 0 to 184 2026-02-25T08:45:48.039 INFO:tasks.rados.rados.0.trial023.stdout:915: oid 10 updating version 184 to 186 2026-02-25T08:45:48.039 INFO:tasks.rados.rados.0.trial023.stdout:915: oid 10 version 186 is already newer than 186 2026-02-25T08:45:48.039 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 10 v 186 (ObjNum 426 snap 116 seq_num 426) dirty exists 2026-02-25T08:45:48.039 INFO:tasks.rados.rados.0.trial023.stdout:915: left oid 10 (ObjNum 426 snap 116 seq_num 426) 2026-02-25T08:45:48.039 INFO:tasks.rados.rados.0.trial023.stdout:913: expect (ObjNum 410 snap 113 seq_num 410) 2026-02-25T08:45:48.075 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:48.089 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~248s 2026-02-25T08:45:48.109 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:48.144 INFO:tasks.rados.rados.0.trial023.stdout:914: finishing write tid 2 to trial02342524-2 2026-02-25T08:45:48.144 INFO:tasks.rados.rados.0.trial023.stdout:914: finishing write tid 3 to trial02342524-2 2026-02-25T08:45:48.144 INFO:tasks.rados.rados.0.trial023.stdout:914: finishing write tid 4 to trial02342524-2 2026-02-25T08:45:48.144 INFO:tasks.rados.rados.0.trial023.stdout:914: finishing write tid 5 to trial02342524-2 2026-02-25T08:45:48.144 INFO:tasks.rados.rados.0.trial023.stdout:914: oid 2 updating version 0 to 73 2026-02-25T08:45:48.144 INFO:tasks.rados.rados.0.trial023.stdout:914: oid 2 version 73 is already newer than 73 2026-02-25T08:45:48.144 INFO:tasks.rados.rados.0.trial023.stdout:914: oid 2 version 73 is already newer than 73 2026-02-25T08:45:48.144 INFO:tasks.rados.rados.0.trial023.stdout:914: oid 2 version 73 is already newer than 70 2026-02-25T08:45:48.144 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 2 v 73 (ObjNum 425 snap 116 seq_num 425) dirty exists 2026-02-25T08:45:48.144 INFO:tasks.rados.rados.0.trial023.stdout:914: left oid 2 (ObjNum 425 snap 116 seq_num 425) 2026-02-25T08:45:48.144 INFO:tasks.rados.rados.0.trial023.stdout:917: seq_num 428 ranges {793396=722340,2128999=690812} 2026-02-25T08:45:48.150 INFO:tasks.rados.rados.0.trial023.stdout:917: writing trial02342524-36 from 793396 to 1515736 tid 1 2026-02-25T08:45:48.155 INFO:tasks.rados.rados.0.trial023.stdout:917: writing trial02342524-36 from 2128999 to 2819811 tid 2 2026-02-25T08:45:48.155 INFO:tasks.rados.rados.0.trial023.stdout:913: done (3 left) 2026-02-25T08:45:48.155 INFO:tasks.rados.rados.0.trial023.stdout:914: done (2 left) 2026-02-25T08:45:48.155 INFO:tasks.rados.rados.0.trial023.stdout:915: done (1 left) 2026-02-25T08:45:48.155 INFO:tasks.rados.rados.0.trial023.stdout:918: delete oid 15 current snap is 116 2026-02-25T08:45:48.164 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:48.184 INFO:tasks.rados.rados.0.trial023.stdout:917: finishing write tid 1 to trial02342524-36 2026-02-25T08:45:48.200 INFO:tasks.rados.rados.0.trial023.stdout:917: finishing write tid 2 to trial02342524-36 2026-02-25T08:45:48.207 INFO:tasks.rados.rados.0.trial023.stdout:917: finishing write tid 3 to trial02342524-36 2026-02-25T08:45:48.210 INFO:tasks.rados.rados.0.trial023.stdout:917: finishing write tid 4 to trial02342524-36 2026-02-25T08:45:48.210 INFO:tasks.rados.rados.0.trial023.stdout:917: oid 36 updating version 0 to 210 2026-02-25T08:45:48.210 INFO:tasks.rados.rados.0.trial023.stdout:917: oid 36 version 210 is already newer than 210 2026-02-25T08:45:48.210 INFO:tasks.rados.rados.0.trial023.stdout:917: oid 36 version 210 is already newer than 208 2026-02-25T08:45:48.210 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 36 v 210 (ObjNum 428 snap 116 seq_num 428) dirty exists 2026-02-25T08:45:48.210 INFO:tasks.rados.rados.0.trial023.stdout:917: left oid 36 (ObjNum 428 snap 116 seq_num 428) 2026-02-25T08:45:48.225 INFO:tasks.rados.rados.0.trial023.stdout:917: done (1 left) 2026-02-25T08:45:48.225 INFO:tasks.rados.rados.0.trial023.stdout:918: done (0 left) 2026-02-25T08:45:48.225 INFO:tasks.rados.rados.0.trial023.stdout:919: write oid 46 current snap is 116 2026-02-25T08:45:48.225 INFO:tasks.rados.rados.0.trial023.stdout:919: seq_num 430 ranges {427010=472838,1329610=694481,2708828=518182} 2026-02-25T08:45:48.229 INFO:tasks.rados.rados.0.trial023.stdout:919: writing trial02342524-46 from 427010 to 899848 tid 1 2026-02-25T08:45:48.234 INFO:tasks.rados.rados.0.trial023.stdout:919: writing trial02342524-46 from 1329610 to 2024091 tid 2 2026-02-25T08:45:48.238 INFO:tasks.rados.rados.0.trial023.stdout:919: writing trial02342524-46 from 2708828 to 3227010 tid 3 2026-02-25T08:45:48.238 INFO:tasks.rados.rados.0.trial023.stdout:920: read oid 1 snap -1 2026-02-25T08:45:48.238 INFO:tasks.rados.rados.0.trial023.stdout:920: expect (ObjNum 417 snap 114 seq_num 417) 2026-02-25T08:45:48.238 INFO:tasks.rados.rados.0.trial023.stdout:921: delete oid 22 current snap is 116 2026-02-25T08:45:48.239 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:48.236+0000 7f1f1941a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:48.242 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:48.240+0000 7f1f1941a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:48.248 INFO:tasks.rados.rados.0.trial023.stdout:919: finishing write tid 1 to trial02342524-46 2026-02-25T08:45:48.261 INFO:tasks.rados.rados.0.trial023.stdout:919: finishing write tid 2 to trial02342524-46 2026-02-25T08:45:48.271 INFO:tasks.rados.rados.0.trial023.stdout:919: finishing write tid 3 to trial02342524-46 2026-02-25T08:45:48.279 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:45:48.280 INFO:tasks.rados.rados.0.trial023.stdout:920: expect (ObjNum 417 snap 114 seq_num 417) 2026-02-25T08:45:48.280 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:48.318 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:48.318 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:48.371 INFO:tasks.rados.rados.0.trial023.stdout:919: finishing write tid 4 to trial02342524-46 2026-02-25T08:45:48.371 INFO:tasks.rados.rados.0.trial023.stdout:919: finishing write tid 5 to trial02342524-46 2026-02-25T08:45:48.371 INFO:tasks.rados.rados.0.trial023.stdout:919: oid 46 updating version 0 to 262 2026-02-25T08:45:48.371 INFO:tasks.rados.rados.0.trial023.stdout:919: oid 46 version 262 is already newer than 262 2026-02-25T08:45:48.371 INFO:tasks.rados.rados.0.trial023.stdout:919: oid 46 version 262 is already newer than 262 2026-02-25T08:45:48.371 INFO:tasks.rados.rados.0.trial023.stdout:919: oid 46 version 262 is already newer than 262 2026-02-25T08:45:48.372 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 46 v 262 (ObjNum 430 snap 116 seq_num 430) dirty exists 2026-02-25T08:45:48.372 INFO:tasks.rados.rados.0.trial023.stdout:919: left oid 46 (ObjNum 430 snap 116 seq_num 430) 2026-02-25T08:45:48.372 INFO:tasks.rados.rados.0.trial023.stdout:919: done (2 left) 2026-02-25T08:45:48.372 INFO:tasks.rados.rados.0.trial023.stdout:920: done (1 left) 2026-02-25T08:45:48.372 INFO:tasks.rados.rados.0.trial023.stdout:921: done (0 left) 2026-02-25T08:45:48.372 INFO:tasks.rados.rados.0.trial023.stdout:922: read oid 19 snap 113 2026-02-25T08:45:48.372 INFO:tasks.rados.rados.0.trial023.stdout:922: expect (ObjNum 393 snap 106 seq_num 393) 2026-02-25T08:45:48.372 INFO:tasks.rados.rados.0.trial023.stdout:923: delete oid 42 current snap is 116 2026-02-25T08:45:48.386 INFO:tasks.rados.rados.0.trial023.stdout:922: expect (ObjNum 393 snap 106 seq_num 393) 2026-02-25T08:45:48.408 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2173785 2026-02-25T08:45:48.411 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:48.433 INFO:tasks.rados.rados.0.trial023.stdout:922: done (1 left) 2026-02-25T08:45:48.433 INFO:tasks.rados.rados.0.trial023.stdout:923: done (0 left) 2026-02-25T08:45:48.433 INFO:tasks.rados.rados.0.trial023.stdout:924: snap_remove snap 113 2026-02-25T08:45:48.900 INFO:tasks.rados.rados.0.trial023.stdout:924: done (0 left) 2026-02-25T08:45:48.900 INFO:tasks.rados.rados.0.trial023.stdout:925: write oid 40 current snap is 116 2026-02-25T08:45:48.900 INFO:tasks.rados.rados.0.trial023.stdout:925: seq_num 433 ranges {477431=698585,1905056=441808,3074744=202687} 2026-02-25T08:45:48.905 INFO:tasks.rados.rados.0.trial023.stdout:925: writing trial02342524-40 from 477431 to 1176016 tid 1 2026-02-25T08:45:48.909 INFO:tasks.rados.rados.0.trial023.stdout:925: writing trial02342524-40 from 1905056 to 2346864 tid 2 2026-02-25T08:45:48.913 INFO:tasks.rados.rados.0.trial023.stdout:925: writing trial02342524-40 from 3074744 to 3277431 tid 3 2026-02-25T08:45:48.913 INFO:tasks.rados.rados.0.trial023.stdout:926: read oid 17 snap -1 2026-02-25T08:45:48.913 INFO:tasks.rados.rados.0.trial023.stdout:926: expect deleted 2026-02-25T08:45:48.913 INFO:tasks.rados.rados.0.trial023.stdout:927: snap_create 2026-02-25T08:45:48.955 INFO:tasks.rados.rados.0.trial023.stdout:925: finishing write tid 1 to trial02342524-40 2026-02-25T08:45:49.008 INFO:tasks.rados.rados.0.trial023.stdout:925: finishing write tid 2 to trial02342524-40 2026-02-25T08:45:49.012 INFO:tasks.rados.rados.0.trial023.stdout:925: finishing write tid 3 to trial02342524-40 2026-02-25T08:45:49.016 INFO:tasks.rados.rados.0.trial023.stdout:925: finishing write tid 4 to trial02342524-40 2026-02-25T08:45:49.017 INFO:tasks.rados.rados.0.trial023.stdout:925: finishing write tid 5 to trial02342524-40 2026-02-25T08:45:49.017 INFO:tasks.rados.rados.0.trial023.stdout:925: oid 40 updating version 0 to 220 2026-02-25T08:45:49.017 INFO:tasks.rados.rados.0.trial023.stdout:925: oid 40 version 220 is already newer than 220 2026-02-25T08:45:49.017 INFO:tasks.rados.rados.0.trial023.stdout:925: oid 40 version 220 is already newer than 220 2026-02-25T08:45:49.017 INFO:tasks.rados.rados.0.trial023.stdout:925: oid 40 version 220 is already newer than 220 2026-02-25T08:45:49.017 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 40 v 220 (ObjNum 433 snap 116 seq_num 433) dirty exists 2026-02-25T08:45:49.017 INFO:tasks.rados.rados.0.trial023.stdout:925: left oid 40 (ObjNum 433 snap 116 seq_num 433) 2026-02-25T08:45:49.837 INFO:tasks.rados.rados.0.trial023.stdout:925: done (2 left) 2026-02-25T08:45:49.837 INFO:tasks.rados.rados.0.trial023.stdout:926: done (1 left) 2026-02-25T08:45:49.837 INFO:tasks.rados.rados.0.trial023.stdout:927: done (0 left) 2026-02-25T08:45:49.837 INFO:tasks.rados.rados.0.trial023.stdout:928: read oid 28 snap 116 2026-02-25T08:45:49.837 INFO:tasks.rados.rados.0.trial023.stdout:928: expect (ObjNum 354 snap 99 seq_num 354) 2026-02-25T08:45:49.838 INFO:tasks.rados.rados.0.trial023.stdout:929: read oid 5 snap -1 2026-02-25T08:45:49.838 INFO:tasks.rados.rados.0.trial023.stdout:929: expect deleted 2026-02-25T08:45:49.838 INFO:tasks.rados.rados.0.trial023.stdout:930: write (excl) oid 13 current snap is 117 2026-02-25T08:45:49.838 INFO:tasks.rados.rados.0.trial023.stdout:930: seq_num 434 ranges {494238=490187,1563538=701819,2818520=475718} 2026-02-25T08:45:49.841 INFO:tasks.rados.rados.0.trial023.stdout:930: writing trial02342524-13 from 494238 to 984425 tid 1 2026-02-25T08:45:49.846 INFO:tasks.rados.rados.0.trial023.stdout:930: writing trial02342524-13 from 1563538 to 2265357 tid 2 2026-02-25T08:45:49.850 INFO:tasks.rados.rados.0.trial023.stdout:930: writing trial02342524-13 from 2818520 to 3294238 tid 3 2026-02-25T08:45:49.851 INFO:tasks.rados.rados.0.trial023.stdout:931: write oid 7 current snap is 117 2026-02-25T08:45:49.851 INFO:tasks.rados.rados.0.trial023.stdout:931: seq_num 435 ranges {511045=565436,1622020=445477,2729590=556855} 2026-02-25T08:45:49.854 INFO:tasks.rados.rados.0.trial023.stdout:931: writing trial02342524-7 from 511045 to 1076481 tid 1 2026-02-25T08:45:49.858 INFO:tasks.rados.rados.0.trial023.stdout:931: writing trial02342524-7 from 1622020 to 2067497 tid 2 2026-02-25T08:45:49.863 INFO:tasks.rados.rados.0.trial023.stdout:931: writing trial02342524-7 from 2729590 to 3286445 tid 3 2026-02-25T08:45:49.863 INFO:tasks.rados.rados.0.trial023.stdout:932: write oid 21 current snap is 117 2026-02-25T08:45:49.863 INFO:tasks.rados.rados.0.trial023.stdout:932: seq_num 436 ranges {527852=640685,1680502=589135,2757013=570839} 2026-02-25T08:45:49.867 INFO:tasks.rados.rados.0.trial023.stdout:932: writing trial02342524-21 from 527852 to 1168537 tid 1 2026-02-25T08:45:49.871 INFO:tasks.rados.rados.0.trial023.stdout:932: writing trial02342524-21 from 1680502 to 2269637 tid 2 2026-02-25T08:45:49.876 INFO:tasks.rados.rados.0.trial023.stdout:932: writing trial02342524-21 from 2757013 to 3327852 tid 3 2026-02-25T08:45:49.876 INFO:tasks.rados.rados.0.trial023.stdout:933: write (excl) oid 14 current snap is 117 2026-02-25T08:45:49.876 INFO:tasks.rados.rados.0.trial023.stdout:933: seq_num 437 ranges {544659=715934,1738984=449146,2784436=560223} 2026-02-25T08:45:49.880 INFO:tasks.rados.rados.0.trial023.stdout:933: writing trial02342524-14 from 544659 to 1260593 tid 1 2026-02-25T08:45:49.884 INFO:tasks.rados.rados.0.trial023.stdout:933: writing trial02342524-14 from 1738984 to 2188130 tid 2 2026-02-25T08:45:49.888 INFO:tasks.rados.rados.0.trial023.stdout:933: writing trial02342524-14 from 2784436 to 3344659 tid 3 2026-02-25T08:45:49.889 INFO:tasks.rados.rados.0.trial023.stdout:934: write oid 8 current snap is 117 2026-02-25T08:45:49.889 INFO:tasks.rados.rados.0.trial023.stdout:934: seq_num 438 ranges {561466=791183,2081113=592804,3095506=265960} 2026-02-25T08:45:49.893 INFO:tasks.rados.rados.0.trial023.stdout:934: writing trial02342524-8 from 561466 to 1352649 tid 1 2026-02-25T08:45:49.898 INFO:tasks.rados.rados.0.trial023.stdout:934: writing trial02342524-8 from 2081113 to 2673917 tid 2 2026-02-25T08:45:49.901 INFO:tasks.rados.rados.0.trial023.stdout:934: writing trial02342524-8 from 3095506 to 3361466 tid 3 2026-02-25T08:45:49.901 INFO:tasks.rados.rados.0.trial023.stdout:935: write oid 50 current snap is 117 2026-02-25T08:45:49.901 INFO:tasks.rados.rados.0.trial023.stdout:935: seq_num 439 ranges {578273=466432,1739595=452815,2722929=655344} 2026-02-25T08:45:49.905 INFO:tasks.rados.rados.0.trial023.stdout:935: writing trial02342524-50 from 578273 to 1044705 tid 1 2026-02-25T08:45:49.909 INFO:tasks.rados.rados.0.trial023.stdout:935: writing trial02342524-50 from 1739595 to 2192410 tid 2 2026-02-25T08:45:49.915 INFO:tasks.rados.rados.0.trial023.stdout:935: writing trial02342524-50 from 2722929 to 3378273 tid 3 2026-02-25T08:45:49.915 INFO:tasks.rados.rados.0.trial023.stdout:936: snap_create 2026-02-25T08:45:49.915 INFO:tasks.rados.rados.0.trial023.stdout:928: expect (ObjNum 354 snap 99 seq_num 354) 2026-02-25T08:45:49.931 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2030293 2026-02-25T08:45:50.034 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 670639 2026-02-25T08:45:50.092 INFO:tasks.rados.rados.0.trial023.stdout:930: finishing write tid 1 to trial02342524-13 2026-02-25T08:45:50.092 INFO:tasks.rados.rados.0.trial023.stdout:932: finishing write tid 1 to trial02342524-21 2026-02-25T08:45:50.092 INFO:tasks.rados.rados.0.trial023.stdout:932: finishing write tid 2 to trial02342524-21 2026-02-25T08:45:50.093 INFO:tasks.rados.rados.0.trial023.stdout:930: finishing write tid 2 to trial02342524-13 2026-02-25T08:45:50.093 INFO:tasks.rados.rados.0.trial023.stdout:933: finishing write tid 1 to trial02342524-14 2026-02-25T08:45:50.093 INFO:tasks.rados.rados.0.trial023.stdout:930: finishing write tid 3 to trial02342524-13 2026-02-25T08:45:50.093 INFO:tasks.rados.rados.0.trial023.stdout:930: finishing write tid 4 to trial02342524-13 2026-02-25T08:45:50.093 INFO:tasks.rados.rados.0.trial023.stdout:930: finishing write tid 5 to trial02342524-13 2026-02-25T08:45:50.093 INFO:tasks.rados.rados.0.trial023.stdout:930: oid 13 updating version 0 to 191 2026-02-25T08:45:50.093 INFO:tasks.rados.rados.0.trial023.stdout:930: oid 13 version 191 is already newer than 191 2026-02-25T08:45:50.093 INFO:tasks.rados.rados.0.trial023.stdout:930: oid 13 version 191 is already newer than 191 2026-02-25T08:45:50.094 INFO:tasks.rados.rados.0.trial023.stdout:930: oid 13 version 191 is already newer than 188 2026-02-25T08:45:50.094 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 13 v 191 (ObjNum 434 snap 117 seq_num 434) dirty exists 2026-02-25T08:45:50.094 INFO:tasks.rados.rados.0.trial023.stdout:930: left oid 13 (ObjNum 434 snap 117 seq_num 434) 2026-02-25T08:45:50.094 INFO:tasks.rados.rados.0.trial023.stdout:932: finishing write tid 3 to trial02342524-21 2026-02-25T08:45:50.094 INFO:tasks.rados.rados.0.trial023.stdout:932: finishing write tid 4 to trial02342524-21 2026-02-25T08:45:50.094 INFO:tasks.rados.rados.0.trial023.stdout:932: finishing write tid 5 to trial02342524-21 2026-02-25T08:45:50.094 INFO:tasks.rados.rados.0.trial023.stdout:932: oid 21 updating version 0 to 162 2026-02-25T08:45:50.094 INFO:tasks.rados.rados.0.trial023.stdout:932: oid 21 version 162 is already newer than 162 2026-02-25T08:45:50.094 INFO:tasks.rados.rados.0.trial023.stdout:932: oid 21 version 162 is already newer than 162 2026-02-25T08:45:50.094 INFO:tasks.rados.rados.0.trial023.stdout:932: oid 21 version 162 is already newer than 162 2026-02-25T08:45:50.094 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 21 v 162 (ObjNum 436 snap 117 seq_num 436) dirty exists 2026-02-25T08:45:50.094 INFO:tasks.rados.rados.0.trial023.stdout:932: left oid 21 (ObjNum 436 snap 117 seq_num 436) 2026-02-25T08:45:50.095 INFO:tasks.rados.rados.0.trial023.stdout:933: finishing write tid 2 to trial02342524-14 2026-02-25T08:45:50.095 INFO:tasks.rados.rados.0.trial023.stdout:934: finishing write tid 1 to trial02342524-8 2026-02-25T08:45:50.095 INFO:tasks.rados.rados.0.trial023.stdout:935: finishing write tid 1 to trial02342524-50 2026-02-25T08:45:50.095 INFO:tasks.rados.rados.0.trial023.stdout:934: finishing write tid 2 to trial02342524-8 2026-02-25T08:45:50.095 INFO:tasks.rados.rados.0.trial023.stdout:933: finishing write tid 3 to trial02342524-14 2026-02-25T08:45:50.095 INFO:tasks.rados.rados.0.trial023.stdout:935: finishing write tid 2 to trial02342524-50 2026-02-25T08:45:50.095 INFO:tasks.rados.rados.0.trial023.stdout:934: finishing write tid 3 to trial02342524-8 2026-02-25T08:45:50.095 INFO:tasks.rados.rados.0.trial023.stdout:935: finishing write tid 3 to trial02342524-50 2026-02-25T08:45:50.095 INFO:tasks.rados.rados.0.trial023.stdout:934: finishing write tid 4 to trial02342524-8 2026-02-25T08:45:50.095 INFO:tasks.rados.rados.0.trial023.stdout:934: finishing write tid 5 to trial02342524-8 2026-02-25T08:45:50.095 INFO:tasks.rados.rados.0.trial023.stdout:934: oid 8 updating version 0 to 56 2026-02-25T08:45:50.095 INFO:tasks.rados.rados.0.trial023.stdout:934: oid 8 version 56 is already newer than 56 2026-02-25T08:45:50.095 INFO:tasks.rados.rados.0.trial023.stdout:934: oid 8 version 56 is already newer than 56 2026-02-25T08:45:50.096 INFO:tasks.rados.rados.0.trial023.stdout:934: oid 8 version 56 is already newer than 56 2026-02-25T08:45:50.096 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 8 v 56 (ObjNum 438 snap 117 seq_num 438) dirty exists 2026-02-25T08:45:50.096 INFO:tasks.rados.rados.0.trial023.stdout:934: left oid 8 (ObjNum 438 snap 117 seq_num 438) 2026-02-25T08:45:50.096 INFO:tasks.rados.rados.0.trial023.stdout:935: finishing write tid 4 to trial02342524-50 2026-02-25T08:45:50.096 INFO:tasks.rados.rados.0.trial023.stdout:933: finishing write tid 4 to trial02342524-14 2026-02-25T08:45:50.096 INFO:tasks.rados.rados.0.trial023.stdout:935: finishing write tid 5 to trial02342524-50 2026-02-25T08:45:50.096 INFO:tasks.rados.rados.0.trial023.stdout:935: oid 50 updating version 0 to 224 2026-02-25T08:45:50.096 INFO:tasks.rados.rados.0.trial023.stdout:935: oid 50 version 224 is already newer than 224 2026-02-25T08:45:50.096 INFO:tasks.rados.rados.0.trial023.stdout:935: oid 50 version 224 is already newer than 224 2026-02-25T08:45:50.096 INFO:tasks.rados.rados.0.trial023.stdout:935: oid 50 version 224 is already newer than 224 2026-02-25T08:45:50.096 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 50 v 224 (ObjNum 439 snap 117 seq_num 439) dirty exists 2026-02-25T08:45:50.096 INFO:tasks.rados.rados.0.trial023.stdout:935: left oid 50 (ObjNum 439 snap 117 seq_num 439) 2026-02-25T08:45:50.096 INFO:tasks.rados.rados.0.trial023.stdout:933: finishing write tid 5 to trial02342524-14 2026-02-25T08:45:50.097 INFO:tasks.rados.rados.0.trial023.stdout:933: oid 14 updating version 0 to 77 2026-02-25T08:45:50.097 INFO:tasks.rados.rados.0.trial023.stdout:933: oid 14 version 77 is already newer than 74 2026-02-25T08:45:50.097 INFO:tasks.rados.rados.0.trial023.stdout:933: oid 14 version 77 is already newer than 77 2026-02-25T08:45:50.097 INFO:tasks.rados.rados.0.trial023.stdout:933: oid 14 version 77 is already newer than 77 2026-02-25T08:45:50.097 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 14 v 77 (ObjNum 437 snap 117 seq_num 437) dirty exists 2026-02-25T08:45:50.097 INFO:tasks.rados.rados.0.trial023.stdout:933: left oid 14 (ObjNum 437 snap 117 seq_num 437) 2026-02-25T08:45:50.097 INFO:tasks.rados.rados.0.trial023.stdout:931: finishing write tid 1 to trial02342524-7 2026-02-25T08:45:50.097 INFO:tasks.rados.rados.0.trial023.stdout:931: finishing write tid 2 to trial02342524-7 2026-02-25T08:45:50.097 INFO:tasks.rados.rados.0.trial023.stdout:931: finishing write tid 3 to trial02342524-7 2026-02-25T08:45:50.097 INFO:tasks.rados.rados.0.trial023.stdout:931: finishing write tid 4 to trial02342524-7 2026-02-25T08:45:50.097 INFO:tasks.rados.rados.0.trial023.stdout:931: finishing write tid 5 to trial02342524-7 2026-02-25T08:45:50.097 INFO:tasks.rados.rados.0.trial023.stdout:931: oid 7 updating version 0 to 145 2026-02-25T08:45:50.097 INFO:tasks.rados.rados.0.trial023.stdout:931: oid 7 version 145 is already newer than 145 2026-02-25T08:45:50.098 INFO:tasks.rados.rados.0.trial023.stdout:931: oid 7 version 145 is already newer than 145 2026-02-25T08:45:50.098 INFO:tasks.rados.rados.0.trial023.stdout:931: oid 7 version 145 is already newer than 145 2026-02-25T08:45:50.098 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 7 v 145 (ObjNum 435 snap 117 seq_num 435) dirty exists 2026-02-25T08:45:50.098 INFO:tasks.rados.rados.0.trial023.stdout:931: left oid 7 (ObjNum 435 snap 117 seq_num 435) 2026-02-25T08:45:50.318 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:50.368 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:45:50.396 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:50.393+0000 7f4395435640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:50.399 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:50.397+0000 7f4395435640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:50.414 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:45:50.438 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:45:50.441 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:45:50.441 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:45:50.442 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:50.443 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:50.904 INFO:tasks.rados.rados.0.trial023.stdout:928: done (8 left) 2026-02-25T08:45:50.904 INFO:tasks.rados.rados.0.trial023.stdout:929: done (7 left) 2026-02-25T08:45:50.904 INFO:tasks.rados.rados.0.trial023.stdout:930: done (6 left) 2026-02-25T08:45:50.904 INFO:tasks.rados.rados.0.trial023.stdout:931: done (5 left) 2026-02-25T08:45:50.904 INFO:tasks.rados.rados.0.trial023.stdout:932: done (4 left) 2026-02-25T08:45:50.904 INFO:tasks.rados.rados.0.trial023.stdout:933: done (3 left) 2026-02-25T08:45:50.904 INFO:tasks.rados.rados.0.trial023.stdout:934: done (2 left) 2026-02-25T08:45:50.904 INFO:tasks.rados.rados.0.trial023.stdout:935: done (1 left) 2026-02-25T08:45:50.904 INFO:tasks.rados.rados.0.trial023.stdout:936: done (0 left) 2026-02-25T08:45:50.904 INFO:tasks.rados.rados.0.trial023.stdout:937: write oid 32 current snap is 118 2026-02-25T08:45:50.904 INFO:tasks.rados.rados.0.trial023.stdout:937: seq_num 440 ranges {595080=541681,1798077=596473,3150352=244728} 2026-02-25T08:45:50.909 INFO:tasks.rados.rados.0.trial023.stdout:937: writing trial02342524-32 from 595080 to 1136761 tid 1 2026-02-25T08:45:50.914 INFO:tasks.rados.rados.0.trial023.stdout:937: writing trial02342524-32 from 1798077 to 2394550 tid 2 2026-02-25T08:45:50.918 INFO:tasks.rados.rados.0.trial023.stdout:937: writing trial02342524-32 from 3150352 to 3395080 tid 3 2026-02-25T08:45:50.918 INFO:tasks.rados.rados.0.trial023.stdout:938: delete oid 34 current snap is 118 2026-02-25T08:45:50.927 INFO:tasks.rados.rados.0.trial023.stdout:938: done (1 left) 2026-02-25T08:45:50.927 INFO:tasks.rados.rados.0.trial023.stdout:939: read oid 20 snap -1 2026-02-25T08:45:50.928 INFO:tasks.rados.rados.0.trial023.stdout:939: expect (ObjNum 410 snap 113 seq_num 410) 2026-02-25T08:45:50.928 INFO:tasks.rados.rados.0.trial023.stdout:940: snap_create 2026-02-25T08:45:50.948 INFO:tasks.rados.rados.0.trial023.stdout:937: finishing write tid 1 to trial02342524-32 2026-02-25T08:45:50.949 INFO:tasks.rados.rados.0.trial023.stdout:939: expect (ObjNum 410 snap 113 seq_num 410) 2026-02-25T08:45:50.949 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:50.993 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1496264 2026-02-25T08:45:51.016 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 625727 2026-02-25T08:45:51.043 INFO:tasks.rados.rados.0.trial023.stdout:937: finishing write tid 2 to trial02342524-32 2026-02-25T08:45:51.044 INFO:tasks.rados.rados.0.trial023.stdout:937: finishing write tid 3 to trial02342524-32 2026-02-25T08:45:51.044 INFO:tasks.rados.rados.0.trial023.stdout:937: finishing write tid 4 to trial02342524-32 2026-02-25T08:45:51.044 INFO:tasks.rados.rados.0.trial023.stdout:937: finishing write tid 5 to trial02342524-32 2026-02-25T08:45:51.044 INFO:tasks.rados.rados.0.trial023.stdout:937: oid 32 updating version 0 to 150 2026-02-25T08:45:51.044 INFO:tasks.rados.rados.0.trial023.stdout:937: oid 32 version 150 is already newer than 150 2026-02-25T08:45:51.044 INFO:tasks.rados.rados.0.trial023.stdout:937: oid 32 version 150 is already newer than 150 2026-02-25T08:45:51.044 INFO:tasks.rados.rados.0.trial023.stdout:937: oid 32 version 150 is already newer than 150 2026-02-25T08:45:51.044 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 32 v 150 (ObjNum 440 snap 118 seq_num 440) dirty exists 2026-02-25T08:45:51.044 INFO:tasks.rados.rados.0.trial023.stdout:937: left oid 32 (ObjNum 440 snap 118 seq_num 440) 2026-02-25T08:45:51.907 INFO:tasks.rados.rados.0.trial023.stdout:937: done (2 left) 2026-02-25T08:45:51.908 INFO:tasks.rados.rados.0.trial023.stdout:939: done (1 left) 2026-02-25T08:45:51.908 INFO:tasks.rados.rados.0.trial023.stdout:940: done (0 left) 2026-02-25T08:45:51.908 INFO:tasks.rados.rados.0.trial023.stdout:941: delete oid 7 current snap is 119 2026-02-25T08:45:51.946 INFO:tasks.rados.rados.0.trial023.stdout:941: done (0 left) 2026-02-25T08:45:51.947 INFO:tasks.rados.rados.0.trial023.stdout:942: read oid 22 snap -1 2026-02-25T08:45:51.947 INFO:tasks.rados.rados.0.trial023.stdout:942: expect deleted 2026-02-25T08:45:51.947 INFO:tasks.rados.rados.0.trial023.stdout:943: read oid 34 snap -1 2026-02-25T08:45:51.947 INFO:tasks.rados.rados.0.trial023.stdout:943: expect deleted 2026-02-25T08:45:51.947 INFO:tasks.rados.rados.0.trial023.stdout:944: snap_remove snap 116 2026-02-25T08:45:52.444 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:52.521 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:52.518+0000 7fb43aa3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:52.523 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:52.521+0000 7fb43aa3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:52.539 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:45:52.571 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:52.571 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:52.913 INFO:tasks.rados.rados.0.trial023.stdout:944: done (2 left) 2026-02-25T08:45:52.913 INFO:tasks.rados.rados.0.trial023.stdout:945: delete oid 23 current snap is 119 2026-02-25T08:45:52.960 INFO:tasks.rados.rados.0.trial023.stdout:942: done (2 left) 2026-02-25T08:45:52.960 INFO:tasks.rados.rados.0.trial023.stdout:943: done (1 left) 2026-02-25T08:45:52.960 INFO:tasks.rados.rados.0.trial023.stdout:945: done (0 left) 2026-02-25T08:45:52.960 INFO:tasks.rados.rados.0.trial023.stdout:946: read oid 16 snap -1 2026-02-25T08:45:52.960 INFO:tasks.rados.rados.0.trial023.stdout:946: expect (ObjNum 373 snap 100 seq_num 373) 2026-02-25T08:45:52.960 INFO:tasks.rados.rados.0.trial023.stdout:947: write oid 10 current snap is 119 2026-02-25T08:45:52.960 INFO:tasks.rados.rados.0.trial023.stdout:947: seq_num 444 ranges {662308=559030,1632005=603811,2860044=602264} 2026-02-25T08:45:52.964 INFO:tasks.rados.rados.0.trial023.stdout:947: writing trial02342524-10 from 662308 to 1221338 tid 1 2026-02-25T08:45:52.968 INFO:tasks.rados.rados.0.trial023.stdout:947: writing trial02342524-10 from 1632005 to 2235816 tid 2 2026-02-25T08:45:52.973 INFO:tasks.rados.rados.0.trial023.stdout:947: writing trial02342524-10 from 2860044 to 3462308 tid 3 2026-02-25T08:45:52.973 INFO:tasks.rados.rados.0.trial023.stdout:948: snap_remove snap 117 2026-02-25T08:45:53.592 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~253s 2026-02-25T08:45:53.914 INFO:tasks.rados.rados.0.trial023.stdout:948: done (2 left) 2026-02-25T08:45:53.914 INFO:tasks.rados.rados.0.trial023.stdout:949: write oid 33 current snap is 119 2026-02-25T08:45:53.914 INFO:tasks.rados.rados.0.trial023.stdout:949: seq_num 445 ranges {679115=634279,2090487=463822,3003820=475295} 2026-02-25T08:45:53.919 INFO:tasks.rados.rados.0.trial023.stdout:949: writing trial02342524-33 from 679115 to 1313394 tid 1 2026-02-25T08:45:53.923 INFO:tasks.rados.rados.0.trial023.stdout:949: writing trial02342524-33 from 2090487 to 2554309 tid 2 2026-02-25T08:45:53.927 INFO:tasks.rados.rados.0.trial023.stdout:949: writing trial02342524-33 from 3003820 to 3479115 tid 3 2026-02-25T08:45:53.927 INFO:tasks.rados.rados.0.trial023.stdout:950: read oid 44 snap -1 2026-02-25T08:45:53.928 INFO:tasks.rados.rados.0.trial023.stdout:950: expect (ObjNum 416 snap 114 seq_num 416) 2026-02-25T08:45:53.928 INFO:tasks.rados.rados.0.trial023.stdout:951: snap_create 2026-02-25T08:45:53.928 INFO:tasks.rados.rados.0.trial023.stdout:946: expect (ObjNum 373 snap 100 seq_num 373) 2026-02-25T08:45:53.936 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1385102 2026-02-25T08:45:53.962 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:54.041 INFO:tasks.rados.rados.0.trial023.stdout:947: finishing write tid 1 to trial02342524-10 2026-02-25T08:45:54.041 INFO:tasks.rados.rados.0.trial023.stdout:947: finishing write tid 2 to trial02342524-10 2026-02-25T08:45:54.041 INFO:tasks.rados.rados.0.trial023.stdout:947: finishing write tid 3 to trial02342524-10 2026-02-25T08:45:54.041 INFO:tasks.rados.rados.0.trial023.stdout:947: finishing write tid 4 to trial02342524-10 2026-02-25T08:45:54.041 INFO:tasks.rados.rados.0.trial023.stdout:947: finishing write tid 5 to trial02342524-10 2026-02-25T08:45:54.041 INFO:tasks.rados.rados.0.trial023.stdout:947: oid 10 updating version 0 to 198 2026-02-25T08:45:54.042 INFO:tasks.rados.rados.0.trial023.stdout:947: oid 10 version 198 is already newer than 198 2026-02-25T08:45:54.042 INFO:tasks.rados.rados.0.trial023.stdout:947: oid 10 version 198 is already newer than 198 2026-02-25T08:45:54.042 INFO:tasks.rados.rados.0.trial023.stdout:947: oid 10 version 198 is already newer than 198 2026-02-25T08:45:54.042 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 10 v 198 (ObjNum 444 snap 119 seq_num 444) dirty exists 2026-02-25T08:45:54.042 INFO:tasks.rados.rados.0.trial023.stdout:947: left oid 10 (ObjNum 444 snap 119 seq_num 444) 2026-02-25T08:45:54.042 INFO:tasks.rados.rados.0.trial023.stdout:950: expect (ObjNum 416 snap 114 seq_num 416) 2026-02-25T08:45:54.078 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:54.103 INFO:tasks.rados.rados.0.trial023.stdout:949: finishing write tid 1 to trial02342524-33 2026-02-25T08:45:54.103 INFO:tasks.rados.rados.0.trial023.stdout:949: finishing write tid 2 to trial02342524-33 2026-02-25T08:45:54.103 INFO:tasks.rados.rados.0.trial023.stdout:949: finishing write tid 3 to trial02342524-33 2026-02-25T08:45:54.103 INFO:tasks.rados.rados.0.trial023.stdout:949: finishing write tid 4 to trial02342524-33 2026-02-25T08:45:54.103 INFO:tasks.rados.rados.0.trial023.stdout:949: finishing write tid 5 to trial02342524-33 2026-02-25T08:45:54.103 INFO:tasks.rados.rados.0.trial023.stdout:949: oid 33 updating version 0 to 44 2026-02-25T08:45:54.104 INFO:tasks.rados.rados.0.trial023.stdout:949: oid 33 version 44 is already newer than 44 2026-02-25T08:45:54.104 INFO:tasks.rados.rados.0.trial023.stdout:949: oid 33 version 44 is already newer than 44 2026-02-25T08:45:54.104 INFO:tasks.rados.rados.0.trial023.stdout:949: oid 33 version 44 is already newer than 44 2026-02-25T08:45:54.104 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 33 v 44 (ObjNum 445 snap 119 seq_num 445) dirty exists 2026-02-25T08:45:54.104 INFO:tasks.rados.rados.0.trial023.stdout:949: left oid 33 (ObjNum 445 snap 119 seq_num 445) 2026-02-25T08:45:54.572 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:54.650 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:54.647+0000 7faa87acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:54.652 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:54.650+0000 7faa87acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:54.669 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:45:54.697 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:54.698 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:54.920 INFO:tasks.rados.rados.0.trial023.stdout:946: done (4 left) 2026-02-25T08:45:54.920 INFO:tasks.rados.rados.0.trial023.stdout:947: done (3 left) 2026-02-25T08:45:54.920 INFO:tasks.rados.rados.0.trial023.stdout:949: done (2 left) 2026-02-25T08:45:54.920 INFO:tasks.rados.rados.0.trial023.stdout:950: done (1 left) 2026-02-25T08:45:54.920 INFO:tasks.rados.rados.0.trial023.stdout:951: done (0 left) 2026-02-25T08:45:54.921 INFO:tasks.rados.rados.0.trial023.stdout:952: snap_remove snap 118 2026-02-25T08:45:55.443 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:45:55.514 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:45:55.517 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:45:55.517 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:45:55.922 INFO:tasks.rados.rados.0.trial023.stdout:952: done (0 left) 2026-02-25T08:45:55.922 INFO:tasks.rados.rados.0.trial023.stdout:953: delete oid 3 current snap is 120 2026-02-25T08:45:55.962 INFO:tasks.rados.rados.0.trial023.stdout:953: done (0 left) 2026-02-25T08:45:55.962 INFO:tasks.rados.rados.0.trial023.stdout:954: write oid 14 current snap is 120 2026-02-25T08:45:55.962 INFO:tasks.rados.rados.0.trial023.stdout:954: seq_num 447 ranges {712729=784777,2091098=467491,3342313=170416} 2026-02-25T08:45:55.967 INFO:tasks.rados.rados.0.trial023.stdout:954: writing trial02342524-14 from 712729 to 1497506 tid 1 2026-02-25T08:45:55.971 INFO:tasks.rados.rados.0.trial023.stdout:954: writing trial02342524-14 from 2091098 to 2558589 tid 2 2026-02-25T08:45:55.975 INFO:tasks.rados.rados.0.trial023.stdout:954: writing trial02342524-14 from 3342313 to 3512729 tid 3 2026-02-25T08:45:55.975 INFO:tasks.rados.rados.0.trial023.stdout:955: read oid 27 snap 119 2026-02-25T08:45:55.975 INFO:tasks.rados.rados.0.trial023.stdout:955: expect (ObjNum 409 snap 113 seq_num 409) 2026-02-25T08:45:55.975 INFO:tasks.rados.rados.0.trial023.stdout:956: write (excl) oid 21 current snap is 120 2026-02-25T08:45:55.975 INFO:tasks.rados.rados.0.trial023.stdout:956: seq_num 448 ranges {729536=460026,1749580=611149,2853383=452450} 2026-02-25T08:45:55.979 INFO:tasks.rados.rados.0.trial023.stdout:956: writing trial02342524-21 from 729536 to 1189562 tid 1 2026-02-25T08:45:55.984 INFO:tasks.rados.rados.0.trial023.stdout:956: writing trial02342524-21 from 1749580 to 2360729 tid 2 2026-02-25T08:45:55.987 INFO:tasks.rados.rados.0.trial023.stdout:956: writing trial02342524-21 from 2853383 to 3305833 tid 3 2026-02-25T08:45:55.988 INFO:tasks.rados.rados.0.trial023.stdout:957: write (excl) oid 17 current snap is 120 2026-02-25T08:45:55.988 INFO:tasks.rados.rados.0.trial023.stdout:957: seq_num 449 ranges {746343=651628,1924415=754807,3397159=149184} 2026-02-25T08:45:55.992 INFO:tasks.rados.rados.0.trial023.stdout:957: writing trial02342524-17 from 746343 to 1397971 tid 1 2026-02-25T08:45:55.997 INFO:tasks.rados.rados.0.trial023.stdout:957: writing trial02342524-17 from 1924415 to 2679222 tid 2 2026-02-25T08:45:56.001 INFO:tasks.rados.rados.0.trial023.stdout:957: writing trial02342524-17 from 3397159 to 3546343 tid 3 2026-02-25T08:45:56.001 INFO:tasks.rados.rados.0.trial023.stdout:958: snap_create 2026-02-25T08:45:56.003 INFO:tasks.rados.rados.0.trial023.stdout:955: expect (ObjNum 409 snap 113 seq_num 409) 2026-02-25T08:45:56.003 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:56.098 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1126393 2026-02-25T08:45:56.139 INFO:tasks.rados.rados.0.trial023.stdout:954: finishing write tid 1 to trial02342524-14 2026-02-25T08:45:56.139 INFO:tasks.rados.rados.0.trial023.stdout:957: finishing write tid 1 to trial02342524-17 2026-02-25T08:45:56.139 INFO:tasks.rados.rados.0.trial023.stdout:954: finishing write tid 2 to trial02342524-14 2026-02-25T08:45:56.139 INFO:tasks.rados.rados.0.trial023.stdout:956: finishing write tid 1 to trial02342524-21 2026-02-25T08:45:56.139 INFO:tasks.rados.rados.0.trial023.stdout:954: finishing write tid 3 to trial02342524-14 2026-02-25T08:45:56.139 INFO:tasks.rados.rados.0.trial023.stdout:954: finishing write tid 4 to trial02342524-14 2026-02-25T08:45:56.139 INFO:tasks.rados.rados.0.trial023.stdout:954: finishing write tid 5 to trial02342524-14 2026-02-25T08:45:56.139 INFO:tasks.rados.rados.0.trial023.stdout:954: oid 14 updating version 0 to 82 2026-02-25T08:45:56.139 INFO:tasks.rados.rados.0.trial023.stdout:954: oid 14 version 82 is already newer than 82 2026-02-25T08:45:56.140 INFO:tasks.rados.rados.0.trial023.stdout:954: oid 14 version 82 is already newer than 82 2026-02-25T08:45:56.140 INFO:tasks.rados.rados.0.trial023.stdout:954: oid 14 version 82 is already newer than 82 2026-02-25T08:45:56.140 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 14 v 82 (ObjNum 447 snap 120 seq_num 447) dirty exists 2026-02-25T08:45:56.140 INFO:tasks.rados.rados.0.trial023.stdout:954: left oid 14 (ObjNum 447 snap 120 seq_num 447) 2026-02-25T08:45:56.140 INFO:tasks.rados.rados.0.trial023.stdout:956: finishing write tid 2 to trial02342524-21 2026-02-25T08:45:56.140 INFO:tasks.rados.rados.0.trial023.stdout:957: finishing write tid 2 to trial02342524-17 2026-02-25T08:45:56.140 INFO:tasks.rados.rados.0.trial023.stdout:956: finishing write tid 3 to trial02342524-21 2026-02-25T08:45:56.140 INFO:tasks.rados.rados.0.trial023.stdout:957: finishing write tid 3 to trial02342524-17 2026-02-25T08:45:56.140 INFO:tasks.rados.rados.0.trial023.stdout:956: finishing write tid 4 to trial02342524-21 2026-02-25T08:45:56.140 INFO:tasks.rados.rados.0.trial023.stdout:956: finishing write tid 5 to trial02342524-21 2026-02-25T08:45:56.140 INFO:tasks.rados.rados.0.trial023.stdout:956: oid 21 updating version 0 to 167 2026-02-25T08:45:56.140 INFO:tasks.rados.rados.0.trial023.stdout:956: oid 21 version 167 is already newer than 167 2026-02-25T08:45:56.141 INFO:tasks.rados.rados.0.trial023.stdout:956: oid 21 version 167 is already newer than 164 2026-02-25T08:45:56.141 INFO:tasks.rados.rados.0.trial023.stdout:956: oid 21 version 167 is already newer than 167 2026-02-25T08:45:56.141 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 21 v 167 (ObjNum 448 snap 120 seq_num 448) dirty exists 2026-02-25T08:45:56.141 INFO:tasks.rados.rados.0.trial023.stdout:956: left oid 21 (ObjNum 448 snap 120 seq_num 448) 2026-02-25T08:45:56.141 INFO:tasks.rados.rados.0.trial023.stdout:957: finishing write tid 4 to trial02342524-17 2026-02-25T08:45:56.141 INFO:tasks.rados.rados.0.trial023.stdout:957: finishing write tid 5 to trial02342524-17 2026-02-25T08:45:56.141 INFO:tasks.rados.rados.0.trial023.stdout:957: oid 17 updating version 0 to 22 2026-02-25T08:45:56.141 INFO:tasks.rados.rados.0.trial023.stdout:957: oid 17 version 22 is already newer than 22 2026-02-25T08:45:56.141 INFO:tasks.rados.rados.0.trial023.stdout:957: oid 17 version 22 is already newer than 22 2026-02-25T08:45:56.141 INFO:tasks.rados.rados.0.trial023.stdout:957: oid 17 version 22 is already newer than 19 2026-02-25T08:45:56.141 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 17 v 22 (ObjNum 449 snap 120 seq_num 449) dirty exists 2026-02-25T08:45:56.141 INFO:tasks.rados.rados.0.trial023.stdout:957: left oid 17 (ObjNum 449 snap 120 seq_num 449) 2026-02-25T08:45:56.699 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:56.776 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:56.772+0000 7feea5e1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:56.778 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:56.776+0000 7feea5e1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:56.799 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:45:56.827 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:56.827 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:56.928 INFO:tasks.rados.rados.0.trial023.stdout:954: done (4 left) 2026-02-25T08:45:56.928 INFO:tasks.rados.rados.0.trial023.stdout:955: done (3 left) 2026-02-25T08:45:56.928 INFO:tasks.rados.rados.0.trial023.stdout:956: done (2 left) 2026-02-25T08:45:56.928 INFO:tasks.rados.rados.0.trial023.stdout:957: done (1 left) 2026-02-25T08:45:56.928 INFO:tasks.rados.rados.0.trial023.stdout:958: done (0 left) 2026-02-25T08:45:56.929 INFO:tasks.rados.rados.0.trial023.stdout:959: write oid 32 current snap is 121 2026-02-25T08:45:56.929 INFO:tasks.rados.rados.0.trial023.stdout:959: seq_num 450 ranges {763150=726877,1982897=614818,3024582=474994} 2026-02-25T08:45:56.933 INFO:tasks.rados.rados.0.trial023.stdout:959: writing trial02342524-32 from 763150 to 1490027 tid 1 2026-02-25T08:45:56.938 INFO:tasks.rados.rados.0.trial023.stdout:959: writing trial02342524-32 from 1982897 to 2597715 tid 2 2026-02-25T08:45:56.942 INFO:tasks.rados.rados.0.trial023.stdout:959: writing trial02342524-32 from 3024582 to 3499576 tid 3 2026-02-25T08:45:56.942 INFO:tasks.rados.rados.0.trial023.stdout:960: write oid 28 current snap is 121 2026-02-25T08:45:56.942 INFO:tasks.rados.rados.0.trial023.stdout:960: seq_num 451 ranges {779957=402126,1925026=758476,3335652=244305} 2026-02-25T08:45:56.946 INFO:tasks.rados.rados.0.trial023.stdout:960: writing trial02342524-28 from 779957 to 1182083 tid 1 2026-02-25T08:45:56.952 INFO:tasks.rados.rados.0.trial023.stdout:960: writing trial02342524-28 from 1925026 to 2683502 tid 2 2026-02-25T08:45:56.955 INFO:tasks.rados.rados.0.trial023.stdout:960: writing trial02342524-28 from 3335652 to 3579957 tid 3 2026-02-25T08:45:56.956 INFO:tasks.rados.rados.0.trial023.stdout:961: read oid 26 snap 119 2026-02-25T08:45:56.956 INFO:tasks.rados.rados.0.trial023.stdout:961: expect (ObjNum 330 snap 93 seq_num 330) 2026-02-25T08:45:56.956 INFO:tasks.rados.rados.0.trial023.stdout:962: delete oid 7 current snap is 121 2026-02-25T08:45:56.962 INFO:tasks.rados.rados.0.trial023.stdout:962: done (3 left) 2026-02-25T08:45:56.962 INFO:tasks.rados.rados.0.trial023.stdout:963: write (excl) oid 15 current snap is 121 2026-02-25T08:45:56.962 INFO:tasks.rados.rados.0.trial023.stdout:963: seq_num 453 ranges {413571=552624,1641990=762145,2990498=425163} 2026-02-25T08:45:56.966 INFO:tasks.rados.rados.0.trial023.stdout:963: writing trial02342524-15 from 413571 to 966195 tid 1 2026-02-25T08:45:56.972 INFO:tasks.rados.rados.0.trial023.stdout:963: writing trial02342524-15 from 1641990 to 2404135 tid 2 2026-02-25T08:45:56.976 INFO:tasks.rados.rados.0.trial023.stdout:963: writing trial02342524-15 from 2990498 to 3415661 tid 3 2026-02-25T08:45:56.976 INFO:tasks.rados.rados.0.trial023.stdout:964: read oid 5 snap -1 2026-02-25T08:45:56.976 INFO:tasks.rados.rados.0.trial023.stdout:964: expect deleted 2026-02-25T08:45:56.976 INFO:tasks.rados.rados.0.trial023.stdout:965: write (excl) oid 39 current snap is 121 2026-02-25T08:45:56.976 INFO:tasks.rados.rados.0.trial023.stdout:965: seq_num 454 ranges {430378=627873,1700472=622156,3017921=612457} 2026-02-25T08:45:56.980 INFO:tasks.rados.rados.0.trial023.stdout:965: writing trial02342524-39 from 430378 to 1058251 tid 1 2026-02-25T08:45:56.985 INFO:tasks.rados.rados.0.trial023.stdout:965: writing trial02342524-39 from 1700472 to 2322628 tid 2 2026-02-25T08:45:56.990 INFO:tasks.rados.rados.0.trial023.stdout:965: writing trial02342524-39 from 3017921 to 3630378 tid 3 2026-02-25T08:45:56.990 INFO:tasks.rados.rados.0.trial023.stdout:966: write oid 38 current snap is 121 2026-02-25T08:45:56.990 INFO:tasks.rados.rados.0.trial023.stdout:966: seq_num 455 ranges {447185=703122,1642601=765814,2928991=447707} 2026-02-25T08:45:56.994 INFO:tasks.rados.rados.0.trial023.stdout:966: writing trial02342524-38 from 447185 to 1150307 tid 1 2026-02-25T08:45:56.999 INFO:tasks.rados.rados.0.trial023.stdout:966: writing trial02342524-38 from 1642601 to 2408415 tid 2 2026-02-25T08:45:57.003 INFO:tasks.rados.rados.0.trial023.stdout:966: writing trial02342524-38 from 2928991 to 3376698 tid 3 2026-02-25T08:45:57.003 INFO:tasks.rados.rados.0.trial023.stdout:967: read oid 43 snap -1 2026-02-25T08:45:57.003 INFO:tasks.rados.rados.0.trial023.stdout:967: expect (ObjNum 401 snap 108 seq_num 401) 2026-02-25T08:45:57.003 INFO:tasks.rados.rados.0.trial023.stdout:968: write (excl) oid 49 current snap is 121 2026-02-25T08:45:57.003 INFO:tasks.rados.rados.0.trial023.stdout:961: expect (ObjNum 330 snap 93 seq_num 330) 2026-02-25T08:45:57.003 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:57.043 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:57.085 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 691331 2026-02-25T08:45:57.119 INFO:tasks.rados.rados.0.trial023.stdout:963: finishing write tid 1 to trial02342524-15 2026-02-25T08:45:57.119 INFO:tasks.rados.rados.0.trial023.stdout:968: seq_num 456 ranges {463992=778371,1701083=625825,2956414=658979} 2026-02-25T08:45:57.123 INFO:tasks.rados.rados.0.trial023.stdout:968: writing trial02342524-49 from 463992 to 1242363 tid 1 2026-02-25T08:45:57.127 INFO:tasks.rados.rados.0.trial023.stdout:968: writing trial02342524-49 from 1701083 to 2326908 tid 2 2026-02-25T08:45:57.132 INFO:tasks.rados.rados.0.trial023.stdout:968: writing trial02342524-49 from 2956414 to 3615393 tid 3 2026-02-25T08:45:57.132 INFO:tasks.rados.rados.0.trial023.stdout:961: done (8 left) 2026-02-25T08:45:57.133 INFO:tasks.rados.rados.0.trial023.stdout:964: done (7 left) 2026-02-25T08:45:57.133 INFO:tasks.rados.rados.0.trial023.stdout:969: snap_remove snap 120 2026-02-25T08:45:57.932 INFO:tasks.rados.rados.0.trial023.stdout:969: done (7 left) 2026-02-25T08:45:57.933 INFO:tasks.rados.rados.0.trial023.stdout:970: write oid 26 current snap is 121 2026-02-25T08:45:57.933 INFO:tasks.rados.rados.0.trial023.stdout:959: finishing write tid 1 to trial02342524-32 2026-02-25T08:45:57.933 INFO:tasks.rados.rados.0.trial023.stdout:970: seq_num 457 ranges {480799=569973,1475918=769483,2700190=586604} 2026-02-25T08:45:57.936 INFO:tasks.rados.rados.0.trial023.stdout:970: writing trial02342524-26 from 480799 to 1050772 tid 1 2026-02-25T08:45:57.941 INFO:tasks.rados.rados.0.trial023.stdout:970: writing trial02342524-26 from 1475918 to 2245401 tid 2 2026-02-25T08:45:57.945 INFO:tasks.rados.rados.0.trial023.stdout:970: writing trial02342524-26 from 2700190 to 3286794 tid 3 2026-02-25T08:45:57.945 INFO:tasks.rados.rados.0.trial023.stdout:971: write (excl) oid 30 current snap is 121 2026-02-25T08:45:57.945 INFO:tasks.rados.rados.0.trial023.stdout:959: finishing write tid 2 to trial02342524-32 2026-02-25T08:45:57.945 INFO:tasks.rados.rados.0.trial023.stdout:963: finishing write tid 2 to trial02342524-15 2026-02-25T08:45:57.945 INFO:tasks.rados.rados.0.trial023.stdout:971: seq_num 458 ranges {497606=645222,1934400=629494,3243966=453640} 2026-02-25T08:45:57.949 INFO:tasks.rados.rados.0.trial023.stdout:971: writing trial02342524-30 from 497606 to 1142828 tid 1 2026-02-25T08:45:57.955 INFO:tasks.rados.rados.0.trial023.stdout:971: writing trial02342524-30 from 1934400 to 2563894 tid 2 2026-02-25T08:45:57.960 INFO:tasks.rados.rados.0.trial023.stdout:971: writing trial02342524-30 from 3243966 to 3697606 tid 3 2026-02-25T08:45:57.960 INFO:tasks.rados.rados.0.trial023.stdout:972: read oid 40 snap -1 2026-02-25T08:45:57.960 INFO:tasks.rados.rados.0.trial023.stdout:972: expect (ObjNum 433 snap 116 seq_num 433) 2026-02-25T08:45:57.960 INFO:tasks.rados.rados.0.trial023.stdout:973: read oid 34 snap -1 2026-02-25T08:45:57.961 INFO:tasks.rados.rados.0.trial023.stdout:973: expect deleted 2026-02-25T08:45:57.961 INFO:tasks.rados.rados.0.trial023.stdout:974: delete oid 4 current snap is 121 2026-02-25T08:45:57.961 INFO:tasks.rados.rados.0.trial023.stdout:959: finishing write tid 3 to trial02342524-32 2026-02-25T08:45:57.961 INFO:tasks.rados.rados.0.trial023.stdout:963: finishing write tid 3 to trial02342524-15 2026-02-25T08:45:57.961 INFO:tasks.rados.rados.0.trial023.stdout:959: finishing write tid 4 to trial02342524-32 2026-02-25T08:45:57.961 INFO:tasks.rados.rados.0.trial023.stdout:963: finishing write tid 4 to trial02342524-15 2026-02-25T08:45:57.961 INFO:tasks.rados.rados.0.trial023.stdout:959: finishing write tid 5 to trial02342524-32 2026-02-25T08:45:57.961 INFO:tasks.rados.rados.0.trial023.stdout:959: oid 32 updating version 0 to 157 2026-02-25T08:45:57.961 INFO:tasks.rados.rados.0.trial023.stdout:959: oid 32 version 157 is already newer than 157 2026-02-25T08:45:57.961 INFO:tasks.rados.rados.0.trial023.stdout:959: oid 32 version 157 is already newer than 157 2026-02-25T08:45:57.961 INFO:tasks.rados.rados.0.trial023.stdout:959: oid 32 version 157 is already newer than 157 2026-02-25T08:45:57.961 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 32 v 157 (ObjNum 450 snap 121 seq_num 450) dirty exists 2026-02-25T08:45:57.962 INFO:tasks.rados.rados.0.trial023.stdout:959: left oid 32 (ObjNum 450 snap 121 seq_num 450) 2026-02-25T08:45:57.962 INFO:tasks.rados.rados.0.trial023.stdout:963: finishing write tid 5 to trial02342524-15 2026-02-25T08:45:57.962 INFO:tasks.rados.rados.0.trial023.stdout:963: oid 15 updating version 0 to 224 2026-02-25T08:45:57.962 INFO:tasks.rados.rados.0.trial023.stdout:963: oid 15 version 224 is already newer than 224 2026-02-25T08:45:57.962 INFO:tasks.rados.rados.0.trial023.stdout:963: oid 15 version 224 is already newer than 221 2026-02-25T08:45:57.962 INFO:tasks.rados.rados.0.trial023.stdout:963: oid 15 version 224 is already newer than 224 2026-02-25T08:45:57.962 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 15 v 224 (ObjNum 453 snap 121 seq_num 453) dirty exists 2026-02-25T08:45:57.962 INFO:tasks.rados.rados.0.trial023.stdout:963: left oid 15 (ObjNum 453 snap 121 seq_num 453) 2026-02-25T08:45:57.962 INFO:tasks.rados.rados.0.trial023.stdout:967: expect (ObjNum 401 snap 108 seq_num 401) 2026-02-25T08:45:57.962 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:58.000 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:45:58.024 INFO:tasks.rados.rados.0.trial023.stdout:960: finishing write tid 1 to trial02342524-28 2026-02-25T08:45:58.025 INFO:tasks.rados.rados.0.trial023.stdout:960: finishing write tid 2 to trial02342524-28 2026-02-25T08:45:58.025 INFO:tasks.rados.rados.0.trial023.stdout:959: done (11 left) 2026-02-25T08:45:58.025 INFO:tasks.rados.rados.0.trial023.stdout:963: done (10 left) 2026-02-25T08:45:58.025 INFO:tasks.rados.rados.0.trial023.stdout:967: done (9 left) 2026-02-25T08:45:58.025 INFO:tasks.rados.rados.0.trial023.stdout:974: done (8 left) 2026-02-25T08:45:58.025 INFO:tasks.rados.rados.0.trial023.stdout:975: delete oid 16 current snap is 121 2026-02-25T08:45:58.025 INFO:tasks.rados.rados.0.trial023.stdout:966: finishing write tid 1 to trial02342524-38 2026-02-25T08:45:58.025 INFO:tasks.rados.rados.0.trial023.stdout:960: finishing write tid 3 to trial02342524-28 2026-02-25T08:45:58.025 INFO:tasks.rados.rados.0.trial023.stdout:960: finishing write tid 4 to trial02342524-28 2026-02-25T08:45:58.025 INFO:tasks.rados.rados.0.trial023.stdout:966: finishing write tid 2 to trial02342524-38 2026-02-25T08:45:58.026 INFO:tasks.rados.rados.0.trial023.stdout:960: finishing write tid 5 to trial02342524-28 2026-02-25T08:45:58.026 INFO:tasks.rados.rados.0.trial023.stdout:960: oid 28 updating version 0 to 180 2026-02-25T08:45:58.026 INFO:tasks.rados.rados.0.trial023.stdout:960: oid 28 version 180 is already newer than 180 2026-02-25T08:45:58.026 INFO:tasks.rados.rados.0.trial023.stdout:960: oid 28 version 180 is already newer than 180 2026-02-25T08:45:58.026 INFO:tasks.rados.rados.0.trial023.stdout:960: oid 28 version 180 is already newer than 180 2026-02-25T08:45:58.026 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 28 v 180 (ObjNum 451 snap 121 seq_num 451) dirty exists 2026-02-25T08:45:58.026 INFO:tasks.rados.rados.0.trial023.stdout:960: left oid 28 (ObjNum 451 snap 121 seq_num 451) 2026-02-25T08:45:58.026 INFO:tasks.rados.rados.0.trial023.stdout:966: finishing write tid 3 to trial02342524-38 2026-02-25T08:45:58.026 INFO:tasks.rados.rados.0.trial023.stdout:965: finishing write tid 1 to trial02342524-39 2026-02-25T08:45:58.026 INFO:tasks.rados.rados.0.trial023.stdout:966: finishing write tid 4 to trial02342524-38 2026-02-25T08:45:58.026 INFO:tasks.rados.rados.0.trial023.stdout:966: finishing write tid 5 to trial02342524-38 2026-02-25T08:45:58.026 INFO:tasks.rados.rados.0.trial023.stdout:966: oid 38 updating version 0 to 172 2026-02-25T08:45:58.026 INFO:tasks.rados.rados.0.trial023.stdout:966: oid 38 version 172 is already newer than 172 2026-02-25T08:45:58.027 INFO:tasks.rados.rados.0.trial023.stdout:966: oid 38 version 172 is already newer than 172 2026-02-25T08:45:58.027 INFO:tasks.rados.rados.0.trial023.stdout:966: oid 38 version 172 is already newer than 172 2026-02-25T08:45:58.027 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 38 v 172 (ObjNum 455 snap 121 seq_num 455) dirty exists 2026-02-25T08:45:58.027 INFO:tasks.rados.rados.0.trial023.stdout:966: left oid 38 (ObjNum 455 snap 121 seq_num 455) 2026-02-25T08:45:58.027 INFO:tasks.rados.rados.0.trial023.stdout:965: finishing write tid 2 to trial02342524-39 2026-02-25T08:45:58.027 INFO:tasks.rados.rados.0.trial023.stdout:965: finishing write tid 3 to trial02342524-39 2026-02-25T08:45:58.027 INFO:tasks.rados.rados.0.trial023.stdout:965: finishing write tid 4 to trial02342524-39 2026-02-25T08:45:58.027 INFO:tasks.rados.rados.0.trial023.stdout:965: finishing write tid 5 to trial02342524-39 2026-02-25T08:45:58.027 INFO:tasks.rados.rados.0.trial023.stdout:965: oid 39 updating version 0 to 185 2026-02-25T08:45:58.027 INFO:tasks.rados.rados.0.trial023.stdout:965: oid 39 version 185 is already newer than 182 2026-02-25T08:45:58.027 INFO:tasks.rados.rados.0.trial023.stdout:965: oid 39 version 185 is already newer than 185 2026-02-25T08:45:58.027 INFO:tasks.rados.rados.0.trial023.stdout:965: oid 39 version 185 is already newer than 185 2026-02-25T08:45:58.027 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 39 v 185 (ObjNum 454 snap 121 seq_num 454) dirty exists 2026-02-25T08:45:58.028 INFO:tasks.rados.rados.0.trial023.stdout:965: left oid 39 (ObjNum 454 snap 121 seq_num 454) 2026-02-25T08:45:58.028 INFO:tasks.rados.rados.0.trial023.stdout:968: finishing write tid 1 to trial02342524-49 2026-02-25T08:45:58.028 INFO:tasks.rados.rados.0.trial023.stdout:968: finishing write tid 2 to trial02342524-49 2026-02-25T08:45:58.028 INFO:tasks.rados.rados.0.trial023.stdout:968: finishing write tid 3 to trial02342524-49 2026-02-25T08:45:58.028 INFO:tasks.rados.rados.0.trial023.stdout:968: finishing write tid 4 to trial02342524-49 2026-02-25T08:45:58.028 INFO:tasks.rados.rados.0.trial023.stdout:968: finishing write tid 5 to trial02342524-49 2026-02-25T08:45:58.028 INFO:tasks.rados.rados.0.trial023.stdout:968: oid 49 updating version 0 to 164 2026-02-25T08:45:58.028 INFO:tasks.rados.rados.0.trial023.stdout:968: oid 49 version 164 is already newer than 161 2026-02-25T08:45:58.028 INFO:tasks.rados.rados.0.trial023.stdout:968: oid 49 version 164 is already newer than 164 2026-02-25T08:45:58.028 INFO:tasks.rados.rados.0.trial023.stdout:968: oid 49 version 164 is already newer than 164 2026-02-25T08:45:58.028 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 49 v 164 (ObjNum 456 snap 121 seq_num 456) dirty exists 2026-02-25T08:45:58.028 INFO:tasks.rados.rados.0.trial023.stdout:968: left oid 49 (ObjNum 456 snap 121 seq_num 456) 2026-02-25T08:45:58.029 INFO:tasks.rados.rados.0.trial023.stdout:970: finishing write tid 1 to trial02342524-26 2026-02-25T08:45:58.029 INFO:tasks.rados.rados.0.trial023.stdout:970: finishing write tid 2 to trial02342524-26 2026-02-25T08:45:58.029 INFO:tasks.rados.rados.0.trial023.stdout:972: expect (ObjNum 433 snap 116 seq_num 433) 2026-02-25T08:45:58.147 INFO:tasks.rados.rados.0.trial023.stdout:970: finishing write tid 3 to trial02342524-26 2026-02-25T08:45:58.147 INFO:tasks.rados.rados.0.trial023.stdout:960: done (8 left) 2026-02-25T08:45:58.148 INFO:tasks.rados.rados.0.trial023.stdout:965: done (7 left) 2026-02-25T08:45:58.148 INFO:tasks.rados.rados.0.trial023.stdout:966: done (6 left) 2026-02-25T08:45:58.148 INFO:tasks.rados.rados.0.trial023.stdout:968: done (5 left) 2026-02-25T08:45:58.148 INFO:tasks.rados.rados.0.trial023.stdout:972: done (4 left) 2026-02-25T08:45:58.148 INFO:tasks.rados.rados.0.trial023.stdout:973: done (3 left) 2026-02-25T08:45:58.148 INFO:tasks.rados.rados.0.trial023.stdout:975: done (2 left) 2026-02-25T08:45:58.148 INFO:tasks.rados.rados.0.trial023.stdout:976: read oid 9 snap -1 2026-02-25T08:45:58.148 INFO:tasks.rados.rados.0.trial023.stdout:976: expect deleted 2026-02-25T08:45:58.148 INFO:tasks.rados.rados.0.trial023.stdout:977: snap_remove snap 119 2026-02-25T08:45:58.828 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:58.905 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:58.902+0000 7f6fb59fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:58.908 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:45:58.905+0000 7f6fb59fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:58.922 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:45:58.935 INFO:tasks.rados.rados.0.trial023.stdout:977: done (3 left) 2026-02-25T08:45:58.935 INFO:tasks.rados.rados.0.trial023.stdout:978: write (excl) oid 48 current snap is 121 2026-02-25T08:45:58.935 INFO:tasks.rados.rados.0.trial023.stdout:971: finishing write tid 1 to trial02342524-30 2026-02-25T08:45:58.935 INFO:tasks.rados.rados.0.trial023.stdout:978: seq_num 461 ranges {548027=470969,1593493=776821,3093529=631692} 2026-02-25T08:45:58.939 INFO:tasks.rados.rados.0.trial023.stdout:978: writing trial02342524-48 from 548027 to 1018996 tid 1 2026-02-25T08:45:58.944 INFO:tasks.rados.rados.0.trial023.stdout:978: writing trial02342524-48 from 1593493 to 2370314 tid 2 2026-02-25T08:45:58.950 INFO:tasks.rados.rados.0.trial023.stdout:978: writing trial02342524-48 from 3093529 to 3725221 tid 3 2026-02-25T08:45:58.951 INFO:tasks.rados.rados.0.trial023.stdout:979: snap_create 2026-02-25T08:45:58.951 INFO:tasks.rados.rados.0.trial023.stdout:970: finishing write tid 4 to trial02342524-26 2026-02-25T08:45:58.951 INFO:tasks.rados.rados.0.trial023.stdout:970: finishing write tid 5 to trial02342524-26 2026-02-25T08:45:58.951 INFO:tasks.rados.rados.0.trial023.stdout:970: oid 26 updating version 0 to 203 2026-02-25T08:45:58.951 INFO:tasks.rados.rados.0.trial023.stdout:970: oid 26 version 203 is already newer than 203 2026-02-25T08:45:58.951 INFO:tasks.rados.rados.0.trial023.stdout:970: oid 26 version 203 is already newer than 203 2026-02-25T08:45:58.951 INFO:tasks.rados.rados.0.trial023.stdout:970: oid 26 version 203 is already newer than 203 2026-02-25T08:45:58.951 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 26 v 203 (ObjNum 457 snap 121 seq_num 457) dirty exists 2026-02-25T08:45:58.951 INFO:tasks.rados.rados.0.trial023.stdout:970: left oid 26 (ObjNum 457 snap 121 seq_num 457) 2026-02-25T08:45:58.951 INFO:tasks.rados.rados.0.trial023.stdout:971: finishing write tid 2 to trial02342524-30 2026-02-25T08:45:58.951 INFO:tasks.rados.rados.0.trial023.stdout:971: finishing write tid 3 to trial02342524-30 2026-02-25T08:45:58.951 INFO:tasks.rados.rados.0.trial023.stdout:971: finishing write tid 4 to trial02342524-30 2026-02-25T08:45:58.952 INFO:tasks.rados.rados.0.trial023.stdout:971: finishing write tid 5 to trial02342524-30 2026-02-25T08:45:58.952 INFO:tasks.rados.rados.0.trial023.stdout:971: oid 30 updating version 0 to 159 2026-02-25T08:45:58.952 INFO:tasks.rados.rados.0.trial023.stdout:971: oid 30 updating version 159 to 162 2026-02-25T08:45:58.952 INFO:tasks.rados.rados.0.trial023.stdout:971: oid 30 version 162 is already newer than 162 2026-02-25T08:45:58.952 INFO:tasks.rados.rados.0.trial023.stdout:971: oid 30 version 162 is already newer than 162 2026-02-25T08:45:58.952 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 30 v 162 (ObjNum 458 snap 121 seq_num 458) dirty exists 2026-02-25T08:45:58.952 INFO:tasks.rados.rados.0.trial023.stdout:971: left oid 30 (ObjNum 458 snap 121 seq_num 458) 2026-02-25T08:45:58.960 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:58.961 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:58.969 INFO:tasks.rados.rados.0.trial023.stdout:978: finishing write tid 1 to trial02342524-48 2026-02-25T08:45:58.987 INFO:tasks.rados.rados.0.trial023.stdout:978: finishing write tid 2 to trial02342524-48 2026-02-25T08:45:58.994 INFO:tasks.rados.rados.0.trial023.stdout:978: finishing write tid 3 to trial02342524-48 2026-02-25T08:45:58.999 INFO:tasks.rados.rados.0.trial023.stdout:978: finishing write tid 4 to trial02342524-48 2026-02-25T08:45:59.001 INFO:tasks.rados.rados.0.trial023.stdout:978: finishing write tid 5 to trial02342524-48 2026-02-25T08:45:59.002 INFO:tasks.rados.rados.0.trial023.stdout:978: oid 48 updating version 0 to 190 2026-02-25T08:45:59.002 INFO:tasks.rados.rados.0.trial023.stdout:978: oid 48 updating version 190 to 193 2026-02-25T08:45:59.002 INFO:tasks.rados.rados.0.trial023.stdout:978: oid 48 version 193 is already newer than 193 2026-02-25T08:45:59.002 INFO:tasks.rados.rados.0.trial023.stdout:978: oid 48 version 193 is already newer than 193 2026-02-25T08:45:59.002 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 48 v 193 (ObjNum 461 snap 121 seq_num 461) dirty exists 2026-02-25T08:45:59.002 INFO:tasks.rados.rados.0.trial023.stdout:978: left oid 48 (ObjNum 461 snap 121 seq_num 461) 2026-02-25T08:45:59.096 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~259s 2026-02-25T08:45:59.838 INFO:tasks.rados.rados.0.trial023.stdout:970: done (4 left) 2026-02-25T08:45:59.839 INFO:tasks.rados.rados.0.trial023.stdout:971: done (3 left) 2026-02-25T08:45:59.839 INFO:tasks.rados.rados.0.trial023.stdout:976: done (2 left) 2026-02-25T08:45:59.839 INFO:tasks.rados.rados.0.trial023.stdout:978: done (1 left) 2026-02-25T08:45:59.839 INFO:tasks.rados.rados.0.trial023.stdout:979: done (0 left) 2026-02-25T08:45:59.839 INFO:tasks.rados.rados.0.trial023.stdout:980: delete oid 40 current snap is 122 2026-02-25T08:45:59.878 INFO:tasks.rados.rados.0.trial023.stdout:980: done (0 left) 2026-02-25T08:45:59.878 INFO:tasks.rados.rados.0.trial023.stdout:981: write (excl) oid 27 current snap is 122 2026-02-25T08:45:59.878 INFO:tasks.rados.rados.0.trial023.stdout:981: seq_num 463 ranges {581641=621467,1994104=780490,3432022=349619} 2026-02-25T08:45:59.882 INFO:tasks.rados.rados.0.trial023.stdout:981: writing trial02342524-27 from 581641 to 1203108 tid 1 2026-02-25T08:45:59.888 INFO:tasks.rados.rados.0.trial023.stdout:981: writing trial02342524-27 from 1994104 to 2774594 tid 2 2026-02-25T08:45:59.892 INFO:tasks.rados.rados.0.trial023.stdout:981: writing trial02342524-27 from 3432022 to 3781641 tid 3 2026-02-25T08:45:59.892 INFO:tasks.rados.rados.0.trial023.stdout:982: delete oid 49 current snap is 122 2026-02-25T08:45:59.910 INFO:tasks.rados.rados.0.trial023.stdout:981: finishing write tid 1 to trial02342524-27 2026-02-25T08:45:59.920 INFO:tasks.rados.rados.0.trial023.stdout:982: done (1 left) 2026-02-25T08:45:59.920 INFO:tasks.rados.rados.0.trial023.stdout:983: delete oid 21 current snap is 122 2026-02-25T08:45:59.927 INFO:tasks.rados.rados.0.trial023.stdout:981: finishing write tid 2 to trial02342524-27 2026-02-25T08:45:59.937 INFO:tasks.rados.rados.0.trial023.stdout:981: finishing write tid 3 to trial02342524-27 2026-02-25T08:45:59.944 INFO:tasks.rados.rados.0.trial023.stdout:981: finishing write tid 4 to trial02342524-27 2026-02-25T08:45:59.946 INFO:tasks.rados.rados.0.trial023.stdout:981: finishing write tid 5 to trial02342524-27 2026-02-25T08:45:59.946 INFO:tasks.rados.rados.0.trial023.stdout:981: oid 27 updating version 0 to 61 2026-02-25T08:45:59.947 INFO:tasks.rados.rados.0.trial023.stdout:981: oid 27 version 61 is already newer than 61 2026-02-25T08:45:59.947 INFO:tasks.rados.rados.0.trial023.stdout:981: oid 27 version 61 is already newer than 61 2026-02-25T08:45:59.947 INFO:tasks.rados.rados.0.trial023.stdout:981: oid 27 version 61 is already newer than 58 2026-02-25T08:45:59.947 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 27 v 61 (ObjNum 463 snap 122 seq_num 463) dirty exists 2026-02-25T08:45:59.947 INFO:tasks.rados.rados.0.trial023.stdout:981: left oid 27 (ObjNum 463 snap 122 seq_num 463) 2026-02-25T08:45:59.960 INFO:tasks.rados.rados.0.trial023.stdout:981: done (1 left) 2026-02-25T08:45:59.960 INFO:tasks.rados.rados.0.trial023.stdout:983: done (0 left) 2026-02-25T08:45:59.960 INFO:tasks.rados.rados.0.trial023.stdout:984: snap_create 2026-02-25T08:46:00.518 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:46:00.588 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:46:00.591 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:46:00.591 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:46:00.938 INFO:tasks.rados.rados.0.trial023.stdout:984: done (0 left) 2026-02-25T08:46:00.939 INFO:tasks.rados.rados.0.trial023.stdout:985: snap_create 2026-02-25T08:46:00.962 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:46:01.038 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:01.034+0000 7fc2ca1dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:01.040 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:01.038+0000 7fc2ca1dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:01.055 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:46:01.082 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:01.083 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:01.942 INFO:tasks.rados.rados.0.trial023.stdout:985: done (0 left) 2026-02-25T08:46:01.942 INFO:tasks.rados.rados.0.trial023.stdout:986: read oid 20 snap -1 2026-02-25T08:46:01.942 INFO:tasks.rados.rados.0.trial023.stdout:986: expect (ObjNum 410 snap 113 seq_num 410) 2026-02-25T08:46:01.942 INFO:tasks.rados.rados.0.trial023.stdout:987: read oid 29 snap -1 2026-02-25T08:46:01.942 INFO:tasks.rados.rados.0.trial023.stdout:987: expect deleted 2026-02-25T08:46:01.942 INFO:tasks.rados.rados.0.trial023.stdout:988: delete oid 24 current snap is 124 2026-02-25T08:46:01.958 INFO:tasks.rados.rados.0.trial023.stdout:987: done (2 left) 2026-02-25T08:46:01.958 INFO:tasks.rados.rados.0.trial023.stdout:988: done (1 left) 2026-02-25T08:46:01.958 INFO:tasks.rados.rados.0.trial023.stdout:989: snap_create 2026-02-25T08:46:01.977 INFO:tasks.rados.rados.0.trial023.stdout:986: expect (ObjNum 410 snap 113 seq_num 410) 2026-02-25T08:46:02.049 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:46:02.947 INFO:tasks.rados.rados.0.trial023.stdout:986: done (1 left) 2026-02-25T08:46:02.947 INFO:tasks.rados.rados.0.trial023.stdout:989: done (0 left) 2026-02-25T08:46:02.947 INFO:tasks.rados.rados.0.trial023.stdout:990: delete oid 46 current snap is 125 2026-02-25T08:46:02.987 INFO:tasks.rados.rados.0.trial023.stdout:990: done (0 left) 2026-02-25T08:46:02.987 INFO:tasks.rados.rados.0.trial023.stdout:991: delete oid 16 current snap is 125 2026-02-25T08:46:02.990 INFO:tasks.rados.rados.0.trial023.stdout:991: done (0 left) 2026-02-25T08:46:02.990 INFO:tasks.rados.rados.0.trial023.stdout:992: write (excl) oid 24 current snap is 125 2026-02-25T08:46:02.990 INFO:tasks.rados.rados.0.trial023.stdout:992: seq_num 469 ranges {682483=789314,1944996=791497,3196560=554574} 2026-02-25T08:46:02.996 INFO:tasks.rados.rados.0.trial023.stdout:992: writing trial02342524-24 from 682483 to 1471797 tid 1 2026-02-25T08:46:03.000 INFO:tasks.rados.rados.0.trial023.stdout:992: writing trial02342524-24 from 1944996 to 2736493 tid 2 2026-02-25T08:46:03.005 INFO:tasks.rados.rados.0.trial023.stdout:992: writing trial02342524-24 from 3196560 to 3751134 tid 3 2026-02-25T08:46:03.005 INFO:tasks.rados.rados.0.trial023.stdout:993: read oid 40 snap -1 2026-02-25T08:46:03.005 INFO:tasks.rados.rados.0.trial023.stdout:993: expect deleted 2026-02-25T08:46:03.005 INFO:tasks.rados.rados.0.trial023.stdout:994: write oid 6 current snap is 125 2026-02-25T08:46:03.005 INFO:tasks.rados.rados.0.trial023.stdout:994: seq_num 470 ranges {699290=464563,1603478=535155,2823983=765846} 2026-02-25T08:46:03.009 INFO:tasks.rados.rados.0.trial023.stdout:994: writing trial02342524-6 from 699290 to 1163853 tid 1 2026-02-25T08:46:03.012 INFO:tasks.rados.rados.0.trial023.stdout:994: writing trial02342524-6 from 1603478 to 2138633 tid 2 2026-02-25T08:46:03.018 INFO:tasks.rados.rados.0.trial023.stdout:994: writing trial02342524-6 from 2823983 to 3589829 tid 3 2026-02-25T08:46:03.018 INFO:tasks.rados.rados.0.trial023.stdout:995: read oid 8 snap 123 2026-02-25T08:46:03.019 INFO:tasks.rados.rados.0.trial023.stdout:995: expect (ObjNum 438 snap 117 seq_num 438) 2026-02-25T08:46:03.019 INFO:tasks.rados.rados.0.trial023.stdout:996: delete oid 7 current snap is 125 2026-02-25T08:46:03.021 INFO:tasks.rados.rados.0.trial023.stdout:992: finishing write tid 1 to trial02342524-24 2026-02-25T08:46:03.032 INFO:tasks.rados.rados.0.trial023.stdout:993: done (4 left) 2026-02-25T08:46:03.032 INFO:tasks.rados.rados.0.trial023.stdout:996: done (3 left) 2026-02-25T08:46:03.032 INFO:tasks.rados.rados.0.trial023.stdout:997: write (excl) oid 40 current snap is 125 2026-02-25T08:46:03.032 INFO:tasks.rados.rados.0.trial023.stdout:997: seq_num 472 ranges {732904=731414,2120442=538824,3278829=504743} 2026-02-25T08:46:03.037 INFO:tasks.rados.rados.0.trial023.stdout:997: writing trial02342524-40 from 732904 to 1464318 tid 1 2026-02-25T08:46:03.042 INFO:tasks.rados.rados.0.trial023.stdout:997: writing trial02342524-40 from 2120442 to 2659266 tid 2 2026-02-25T08:46:03.047 INFO:tasks.rados.rados.0.trial023.stdout:997: writing trial02342524-40 from 3278829 to 3783572 tid 3 2026-02-25T08:46:03.047 INFO:tasks.rados.rados.0.trial023.stdout:998: read oid 14 snap -1 2026-02-25T08:46:03.047 INFO:tasks.rados.rados.0.trial023.stdout:998: expect (ObjNum 447 snap 120 seq_num 447) 2026-02-25T08:46:03.047 INFO:tasks.rados.rados.0.trial023.stdout:999: read oid 32 snap -1 2026-02-25T08:46:03.047 INFO:tasks.rados.rados.0.trial023.stdout:999: expect (ObjNum 450 snap 121 seq_num 450) 2026-02-25T08:46:03.048 INFO:tasks.rados.rados.0.trial023.stdout:1000: delete oid 48 current snap is 125 2026-02-25T08:46:03.048 INFO:tasks.rados.rados.0.trial023.stdout:994: finishing write tid 1 to trial02342524-6 2026-02-25T08:46:03.048 INFO:tasks.rados.rados.0.trial023.stdout:992: finishing write tid 2 to trial02342524-24 2026-02-25T08:46:03.054 INFO:tasks.rados.rados.0.trial023.stdout:995: expect (ObjNum 438 snap 117 seq_num 438) 2026-02-25T08:46:03.083 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:46:03.141 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:46:03.159 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:03.156+0000 7f3fec50b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:03.162 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:03.160+0000 7f3fec50b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:03.179 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:46:03.185 INFO:tasks.rados.rados.0.trial023.stdout:995: done (6 left) 2026-02-25T08:46:03.185 INFO:tasks.rados.rados.0.trial023.stdout:1000: done (5 left) 2026-02-25T08:46:03.185 INFO:tasks.rados.rados.0.trial023.stdout:1001: write (excl) oid 5 current snap is 125 2026-02-25T08:46:03.185 INFO:tasks.rados.rados.0.trial023.stdout:997: finishing write tid 1 to trial02342524-40 2026-02-25T08:46:03.185 INFO:tasks.rados.rados.0.trial023.stdout:997: finishing write tid 2 to trial02342524-40 2026-02-25T08:46:03.186 INFO:tasks.rados.rados.0.trial023.stdout:992: finishing write tid 3 to trial02342524-24 2026-02-25T08:46:03.186 INFO:tasks.rados.rados.0.trial023.stdout:997: finishing write tid 3 to trial02342524-40 2026-02-25T08:46:03.186 INFO:tasks.rados.rados.0.trial023.stdout:994: finishing write tid 2 to trial02342524-6 2026-02-25T08:46:03.186 INFO:tasks.rados.rados.0.trial023.stdout:997: finishing write tid 4 to trial02342524-40 2026-02-25T08:46:03.186 INFO:tasks.rados.rados.0.trial023.stdout:997: finishing write tid 5 to trial02342524-40 2026-02-25T08:46:03.186 INFO:tasks.rados.rados.0.trial023.stdout:997: oid 40 updating version 0 to 230 2026-02-25T08:46:03.186 INFO:tasks.rados.rados.0.trial023.stdout:997: oid 40 version 230 is already newer than 230 2026-02-25T08:46:03.186 INFO:tasks.rados.rados.0.trial023.stdout:997: oid 40 version 230 is already newer than 227 2026-02-25T08:46:03.186 INFO:tasks.rados.rados.0.trial023.stdout:997: oid 40 version 230 is already newer than 230 2026-02-25T08:46:03.186 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 40 v 230 (ObjNum 472 snap 125 seq_num 472) dirty exists 2026-02-25T08:46:03.186 INFO:tasks.rados.rados.0.trial023.stdout:997: left oid 40 (ObjNum 472 snap 125 seq_num 472) 2026-02-25T08:46:03.186 INFO:tasks.rados.rados.0.trial023.stdout:992: finishing write tid 4 to trial02342524-24 2026-02-25T08:46:03.187 INFO:tasks.rados.rados.0.trial023.stdout:994: finishing write tid 3 to trial02342524-6 2026-02-25T08:46:03.187 INFO:tasks.rados.rados.0.trial023.stdout:992: finishing write tid 5 to trial02342524-24 2026-02-25T08:46:03.187 INFO:tasks.rados.rados.0.trial023.stdout:992: oid 24 updating version 0 to 163 2026-02-25T08:46:03.187 INFO:tasks.rados.rados.0.trial023.stdout:992: oid 24 updating version 163 to 166 2026-02-25T08:46:03.187 INFO:tasks.rados.rados.0.trial023.stdout:992: oid 24 version 166 is already newer than 166 2026-02-25T08:46:03.187 INFO:tasks.rados.rados.0.trial023.stdout:992: oid 24 version 166 is already newer than 166 2026-02-25T08:46:03.187 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 24 v 166 (ObjNum 469 snap 125 seq_num 469) dirty exists 2026-02-25T08:46:03.187 INFO:tasks.rados.rados.0.trial023.stdout:992: left oid 24 (ObjNum 469 snap 125 seq_num 469) 2026-02-25T08:46:03.187 INFO:tasks.rados.rados.0.trial023.stdout:994: finishing write tid 4 to trial02342524-6 2026-02-25T08:46:03.187 INFO:tasks.rados.rados.0.trial023.stdout:994: finishing write tid 5 to trial02342524-6 2026-02-25T08:46:03.187 INFO:tasks.rados.rados.0.trial023.stdout:994: oid 6 updating version 0 to 66 2026-02-25T08:46:03.187 INFO:tasks.rados.rados.0.trial023.stdout:994: oid 6 version 66 is already newer than 66 2026-02-25T08:46:03.187 INFO:tasks.rados.rados.0.trial023.stdout:994: oid 6 version 66 is already newer than 66 2026-02-25T08:46:03.187 INFO:tasks.rados.rados.0.trial023.stdout:994: oid 6 version 66 is already newer than 66 2026-02-25T08:46:03.188 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 6 v 66 (ObjNum 470 snap 125 seq_num 470) dirty exists 2026-02-25T08:46:03.188 INFO:tasks.rados.rados.0.trial023.stdout:994: left oid 6 (ObjNum 470 snap 125 seq_num 470) 2026-02-25T08:46:03.188 INFO:tasks.rados.rados.0.trial023.stdout:999: expect (ObjNum 450 snap 121 seq_num 450) 2026-02-25T08:46:03.206 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:03.207 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:03.226 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:46:03.332 INFO:tasks.rados.rados.0.trial023.stdout:998: expect (ObjNum 447 snap 120 seq_num 447) 2026-02-25T08:46:03.446 INFO:tasks.rados.rados.0.trial023.stdout:1001: seq_num 474 ranges {766518=481912,1837406=542493,2933675=527287,3876584=89934} 2026-02-25T08:46:03.451 INFO:tasks.rados.rados.0.trial023.stdout:1001: writing trial02342524-5 from 766518 to 1248430 tid 1 2026-02-25T08:46:03.455 INFO:tasks.rados.rados.0.trial023.stdout:1001: writing trial02342524-5 from 1837406 to 2379899 tid 2 2026-02-25T08:46:03.459 INFO:tasks.rados.rados.0.trial023.stdout:1001: writing trial02342524-5 from 2933675 to 3460962 tid 3 2026-02-25T08:46:03.462 INFO:tasks.rados.rados.0.trial023.stdout:1001: writing trial02342524-5 from 3876584 to 3966518 tid 4 2026-02-25T08:46:03.462 INFO:tasks.rados.rados.0.trial023.stdout:992: done (5 left) 2026-02-25T08:46:03.462 INFO:tasks.rados.rados.0.trial023.stdout:994: done (4 left) 2026-02-25T08:46:03.462 INFO:tasks.rados.rados.0.trial023.stdout:997: done (3 left) 2026-02-25T08:46:03.462 INFO:tasks.rados.rados.0.trial023.stdout:998: done (2 left) 2026-02-25T08:46:03.462 INFO:tasks.rados.rados.0.trial023.stdout:999: done (1 left) 2026-02-25T08:46:03.462 INFO:tasks.rados.rados.0.trial023.stdout:1002: read oid 50 snap 124 2026-02-25T08:46:03.462 INFO:tasks.rados.rados.0.trial023.stdout:1002: expect (ObjNum 439 snap 117 seq_num 439) 2026-02-25T08:46:03.463 INFO:tasks.rados.rados.0.trial023.stdout:1003: read oid 35 snap -1 2026-02-25T08:46:03.463 INFO:tasks.rados.rados.0.trial023.stdout:1003: expect (ObjNum 394 snap 106 seq_num 394) 2026-02-25T08:46:03.463 INFO:tasks.rados.rados.0.trial023.stdout:1004: snap_create 2026-02-25T08:46:03.473 INFO:tasks.rados.rados.0.trial023.stdout:1001: finishing write tid 1 to trial02342524-5 2026-02-25T08:46:03.481 INFO:tasks.rados.rados.0.trial023.stdout:1002: expect (ObjNum 439 snap 117 seq_num 439) 2026-02-25T08:46:03.482 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:46:03.524 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:46:03.580 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2383980 2026-02-25T08:46:03.595 INFO:tasks.rados.rados.0.trial023.stdout:1003: expect (ObjNum 394 snap 106 seq_num 394) 2026-02-25T08:46:03.645 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1535656 2026-02-25T08:46:03.669 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:46:03.710 INFO:tasks.rados.rados.0.trial023.stdout:1001: finishing write tid 2 to trial02342524-5 2026-02-25T08:46:03.710 INFO:tasks.rados.rados.0.trial023.stdout:1001: finishing write tid 3 to trial02342524-5 2026-02-25T08:46:03.710 INFO:tasks.rados.rados.0.trial023.stdout:1001: finishing write tid 4 to trial02342524-5 2026-02-25T08:46:03.711 INFO:tasks.rados.rados.0.trial023.stdout:1001: finishing write tid 5 to trial02342524-5 2026-02-25T08:46:03.711 INFO:tasks.rados.rados.0.trial023.stdout:1001: finishing write tid 6 to trial02342524-5 2026-02-25T08:46:03.711 INFO:tasks.rados.rados.0.trial023.stdout:1001: oid 5 updating version 0 to 33 2026-02-25T08:46:03.711 INFO:tasks.rados.rados.0.trial023.stdout:1001: oid 5 version 33 is already newer than 33 2026-02-25T08:46:03.711 INFO:tasks.rados.rados.0.trial023.stdout:1001: oid 5 version 33 is already newer than 33 2026-02-25T08:46:03.711 INFO:tasks.rados.rados.0.trial023.stdout:1001: oid 5 version 33 is already newer than 33 2026-02-25T08:46:03.711 INFO:tasks.rados.rados.0.trial023.stdout:1001: oid 5 version 33 is already newer than 29 2026-02-25T08:46:03.711 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 5 v 33 (ObjNum 474 snap 125 seq_num 474) dirty exists 2026-02-25T08:46:03.711 INFO:tasks.rados.rados.0.trial023.stdout:1001: left oid 5 (ObjNum 474 snap 125 seq_num 474) 2026-02-25T08:46:03.951 INFO:tasks.rados.rados.0.trial023.stdout:1001: done (3 left) 2026-02-25T08:46:03.951 INFO:tasks.rados.rados.0.trial023.stdout:1002: done (2 left) 2026-02-25T08:46:03.951 INFO:tasks.rados.rados.0.trial023.stdout:1003: done (1 left) 2026-02-25T08:46:03.951 INFO:tasks.rados.rados.0.trial023.stdout:1004: done (0 left) 2026-02-25T08:46:03.951 INFO:tasks.rados.rados.0.trial023.stdout:1005: write oid 26 current snap is 126 2026-02-25T08:46:03.951 INFO:tasks.rados.rados.0.trial023.stdout:1005: seq_num 475 ranges {783325=557161,1779535=686151,3244745=454912} 2026-02-25T08:46:03.955 INFO:tasks.rados.rados.0.trial023.stdout:1005: writing trial02342524-26 from 783325 to 1340486 tid 1 2026-02-25T08:46:03.959 INFO:tasks.rados.rados.0.trial023.stdout:1005: writing trial02342524-26 from 1779535 to 2465686 tid 2 2026-02-25T08:46:03.965 INFO:tasks.rados.rados.0.trial023.stdout:1005: writing trial02342524-26 from 3244745 to 3699657 tid 3 2026-02-25T08:46:03.965 INFO:tasks.rados.rados.0.trial023.stdout:1006: delete oid 15 current snap is 126 2026-02-25T08:46:03.991 INFO:tasks.rados.rados.0.trial023.stdout:1006: done (1 left) 2026-02-25T08:46:03.991 INFO:tasks.rados.rados.0.trial023.stdout:1007: delete oid 41 current snap is 126 2026-02-25T08:46:03.997 INFO:tasks.rados.rados.0.trial023.stdout:1005: finishing write tid 1 to trial02342524-26 2026-02-25T08:46:04.006 INFO:tasks.rados.rados.0.trial023.stdout:1005: finishing write tid 2 to trial02342524-26 2026-02-25T08:46:04.014 INFO:tasks.rados.rados.0.trial023.stdout:1005: finishing write tid 3 to trial02342524-26 2026-02-25T08:46:04.018 INFO:tasks.rados.rados.0.trial023.stdout:1005: finishing write tid 4 to trial02342524-26 2026-02-25T08:46:04.020 INFO:tasks.rados.rados.0.trial023.stdout:1005: finishing write tid 5 to trial02342524-26 2026-02-25T08:46:04.020 INFO:tasks.rados.rados.0.trial023.stdout:1005: oid 26 updating version 0 to 208 2026-02-25T08:46:04.020 INFO:tasks.rados.rados.0.trial023.stdout:1005: oid 26 version 208 is already newer than 208 2026-02-25T08:46:04.020 INFO:tasks.rados.rados.0.trial023.stdout:1005: oid 26 version 208 is already newer than 208 2026-02-25T08:46:04.021 INFO:tasks.rados.rados.0.trial023.stdout:1005: oid 26 version 208 is already newer than 208 2026-02-25T08:46:04.021 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 26 v 208 (ObjNum 475 snap 126 seq_num 475) dirty exists 2026-02-25T08:46:04.021 INFO:tasks.rados.rados.0.trial023.stdout:1005: left oid 26 (ObjNum 475 snap 126 seq_num 475) 2026-02-25T08:46:04.026 INFO:tasks.rados.rados.0.trial023.stdout:1005: done (1 left) 2026-02-25T08:46:04.026 INFO:tasks.rados.rados.0.trial023.stdout:1007: done (0 left) 2026-02-25T08:46:04.026 INFO:tasks.rados.rados.0.trial023.stdout:1008: read oid 36 snap -1 2026-02-25T08:46:04.026 INFO:tasks.rados.rados.0.trial023.stdout:1008: expect (ObjNum 428 snap 116 seq_num 428) 2026-02-25T08:46:04.026 INFO:tasks.rados.rados.0.trial023.stdout:1009: write oid 25 current snap is 126 2026-02-25T08:46:04.026 INFO:tasks.rados.rados.0.trial023.stdout:1009: seq_num 478 ranges {433746=782908,1954981=78765} 2026-02-25T08:46:04.030 INFO:tasks.rados.rados.0.trial023.stdout:1009: writing trial02342524-25 from 433746 to 1216654 tid 1 2026-02-25T08:46:04.033 INFO:tasks.rados.rados.0.trial023.stdout:1009: writing trial02342524-25 from 1954981 to 2033746 tid 2 2026-02-25T08:46:04.034 INFO:tasks.rados.rados.0.trial023.stdout:1010: read oid 44 snap 123 2026-02-25T08:46:04.034 INFO:tasks.rados.rados.0.trial023.stdout:1010: expect (ObjNum 416 snap 114 seq_num 416) 2026-02-25T08:46:04.034 INFO:tasks.rados.rados.0.trial023.stdout:1011: delete oid 49 current snap is 126 2026-02-25T08:46:04.036 INFO:tasks.rados.rados.0.trial023.stdout:1011: done (3 left) 2026-02-25T08:46:04.036 INFO:tasks.rados.rados.0.trial023.stdout:1012: delete oid 16 current snap is 126 2026-02-25T08:46:04.038 INFO:tasks.rados.rados.0.trial023.stdout:1012: done (3 left) 2026-02-25T08:46:04.038 INFO:tasks.rados.rados.0.trial023.stdout:1013: read oid 15 snap -1 2026-02-25T08:46:04.038 INFO:tasks.rados.rados.0.trial023.stdout:1013: expect deleted 2026-02-25T08:46:04.038 INFO:tasks.rados.rados.0.trial023.stdout:1014: write oid 47 current snap is 126 2026-02-25T08:46:04.038 INFO:tasks.rados.rados.0.trial023.stdout:1014: seq_num 481 ranges {484167=725008,1730427=353740} 2026-02-25T08:46:04.042 INFO:tasks.rados.rados.0.trial023.stdout:1014: writing trial02342524-47 from 484167 to 1209175 tid 1 2026-02-25T08:46:04.046 INFO:tasks.rados.rados.0.trial023.stdout:1014: writing trial02342524-47 from 1730427 to 2084167 tid 2 2026-02-25T08:46:04.046 INFO:tasks.rados.rados.0.trial023.stdout:1015: snap_remove snap 124 2026-02-25T08:46:04.599 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~264s 2026-02-25T08:46:04.953 INFO:tasks.rados.rados.0.trial023.stdout:1015: done (5 left) 2026-02-25T08:46:04.953 INFO:tasks.rados.rados.0.trial023.stdout:1016: delete oid 37 current snap is 126 2026-02-25T08:46:04.953 INFO:tasks.rados.rados.0.trial023.stdout:1010: expect (ObjNum 416 snap 114 seq_num 416) 2026-02-25T08:46:04.960 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2763136 2026-02-25T08:46:04.966 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 276903 2026-02-25T08:46:04.990 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 241501 2026-02-25T08:46:05.015 INFO:tasks.rados.rados.0.trial023.stdout:1009: finishing write tid 1 to trial02342524-25 2026-02-25T08:46:05.015 INFO:tasks.rados.rados.0.trial023.stdout:1008: expect (ObjNum 428 snap 116 seq_num 428) 2026-02-25T08:46:05.043 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2190644 2026-02-25T08:46:05.098 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:46:05.147 INFO:tasks.rados.rados.0.trial023.stdout:1014: finishing write tid 1 to trial02342524-47 2026-02-25T08:46:05.147 INFO:tasks.rados.rados.0.trial023.stdout:1009: finishing write tid 2 to trial02342524-25 2026-02-25T08:46:05.147 INFO:tasks.rados.rados.0.trial023.stdout:1008: done (5 left) 2026-02-25T08:46:05.147 INFO:tasks.rados.rados.0.trial023.stdout:1010: done (4 left) 2026-02-25T08:46:05.148 INFO:tasks.rados.rados.0.trial023.stdout:1013: done (3 left) 2026-02-25T08:46:05.148 INFO:tasks.rados.rados.0.trial023.stdout:1016: done (2 left) 2026-02-25T08:46:05.148 INFO:tasks.rados.rados.0.trial023.stdout:1017: write oid 29 current snap is 126 2026-02-25T08:46:05.148 INFO:tasks.rados.rados.0.trial023.stdout:1014: finishing write tid 2 to trial02342524-47 2026-02-25T08:46:05.148 INFO:tasks.rados.rados.0.trial023.stdout:1009: finishing write tid 3 to trial02342524-25 2026-02-25T08:46:05.148 INFO:tasks.rados.rados.0.trial023.stdout:1017: seq_num 483 ranges {517781=475506,1731038=386743} 2026-02-25T08:46:05.151 INFO:tasks.rados.rados.0.trial023.stdout:1017: writing trial02342524-29 from 517781 to 993287 tid 1 2026-02-25T08:46:05.156 INFO:tasks.rados.rados.0.trial023.stdout:1017: writing trial02342524-29 from 1731038 to 2117781 tid 2 2026-02-25T08:46:05.156 INFO:tasks.rados.rados.0.trial023.stdout:1018: write oid 7 current snap is 126 2026-02-25T08:46:05.156 INFO:tasks.rados.rados.0.trial023.stdout:1009: finishing write tid 4 to trial02342524-25 2026-02-25T08:46:05.156 INFO:tasks.rados.rados.0.trial023.stdout:1009: oid 25 updating version 0 to 232 2026-02-25T08:46:05.156 INFO:tasks.rados.rados.0.trial023.stdout:1009: oid 25 version 232 is already newer than 232 2026-02-25T08:46:05.157 INFO:tasks.rados.rados.0.trial023.stdout:1009: oid 25 version 232 is already newer than 232 2026-02-25T08:46:05.157 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 25 v 232 (ObjNum 478 snap 126 seq_num 478) dirty exists 2026-02-25T08:46:05.157 INFO:tasks.rados.rados.0.trial023.stdout:1009: left oid 25 (ObjNum 478 snap 126 seq_num 478) 2026-02-25T08:46:05.157 INFO:tasks.rados.rados.0.trial023.stdout:1014: finishing write tid 3 to trial02342524-47 2026-02-25T08:46:05.157 INFO:tasks.rados.rados.0.trial023.stdout:1018: seq_num 484 ranges {534588=550755,1789520=345068} 2026-02-25T08:46:05.160 INFO:tasks.rados.rados.0.trial023.stdout:1018: writing trial02342524-7 from 534588 to 1085343 tid 1 2026-02-25T08:46:05.164 INFO:tasks.rados.rados.0.trial023.stdout:1018: writing trial02342524-7 from 1789520 to 2134588 tid 2 2026-02-25T08:46:05.164 INFO:tasks.rados.rados.0.trial023.stdout:1009: done (3 left) 2026-02-25T08:46:05.164 INFO:tasks.rados.rados.0.trial023.stdout:1019: write (excl) oid 14 current snap is 126 2026-02-25T08:46:05.164 INFO:tasks.rados.rados.0.trial023.stdout:1019: seq_num 485 ranges {551395=626004,1848002=303393} 2026-02-25T08:46:05.168 INFO:tasks.rados.rados.0.trial023.stdout:1019: writing trial02342524-14 from 551395 to 1177399 tid 1 2026-02-25T08:46:05.172 INFO:tasks.rados.rados.0.trial023.stdout:1019: writing trial02342524-14 from 1848002 to 2151395 tid 2 2026-02-25T08:46:05.172 INFO:tasks.rados.rados.0.trial023.stdout:1020: snap_remove snap 121 2026-02-25T08:46:05.208 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:46:05.286 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:05.283+0000 7f85c610b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:05.289 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:05.287+0000 7f85c610b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:05.305 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:46:05.331 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:05.331 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:05.593 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:46:05.664 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:46:05.667 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:46:05.667 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:46:05.957 INFO:tasks.rados.rados.0.trial023.stdout:1020: done (4 left) 2026-02-25T08:46:05.957 INFO:tasks.rados.rados.0.trial023.stdout:1021: snap_remove snap 125 2026-02-25T08:46:06.959 INFO:tasks.rados.rados.0.trial023.stdout:1021: done (4 left) 2026-02-25T08:46:06.959 INFO:tasks.rados.rados.0.trial023.stdout:1022: snap_remove snap 122 2026-02-25T08:46:07.332 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:46:07.410 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:07.407+0000 7f499425a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:07.412 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:07.410+0000 7f499425a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:07.427 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:46:07.453 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:07.454 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:07.965 INFO:tasks.rados.rados.0.trial023.stdout:1022: done (4 left) 2026-02-25T08:46:07.966 INFO:tasks.rados.rados.0.trial023.stdout:1023: delete oid 6 current snap is 126 2026-02-25T08:46:07.966 INFO:tasks.rados.rados.0.trial023.stdout:1014: finishing write tid 4 to trial02342524-47 2026-02-25T08:46:07.966 INFO:tasks.rados.rados.0.trial023.stdout:1014: oid 47 updating version 0 to 70 2026-02-25T08:46:07.966 INFO:tasks.rados.rados.0.trial023.stdout:1014: oid 47 version 70 is already newer than 70 2026-02-25T08:46:07.966 INFO:tasks.rados.rados.0.trial023.stdout:1014: oid 47 version 70 is already newer than 70 2026-02-25T08:46:07.966 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 47 v 70 (ObjNum 481 snap 126 seq_num 481) dirty exists 2026-02-25T08:46:07.966 INFO:tasks.rados.rados.0.trial023.stdout:1014: left oid 47 (ObjNum 481 snap 126 seq_num 481) 2026-02-25T08:46:07.966 INFO:tasks.rados.rados.0.trial023.stdout:1017: finishing write tid 1 to trial02342524-29 2026-02-25T08:46:07.966 INFO:tasks.rados.rados.0.trial023.stdout:1017: finishing write tid 2 to trial02342524-29 2026-02-25T08:46:07.966 INFO:tasks.rados.rados.0.trial023.stdout:1017: finishing write tid 3 to trial02342524-29 2026-02-25T08:46:07.967 INFO:tasks.rados.rados.0.trial023.stdout:1017: finishing write tid 4 to trial02342524-29 2026-02-25T08:46:07.967 INFO:tasks.rados.rados.0.trial023.stdout:1017: oid 29 updating version 0 to 211 2026-02-25T08:46:07.967 INFO:tasks.rados.rados.0.trial023.stdout:1017: oid 29 version 211 is already newer than 211 2026-02-25T08:46:07.967 INFO:tasks.rados.rados.0.trial023.stdout:1017: oid 29 version 211 is already newer than 211 2026-02-25T08:46:07.967 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 29 v 211 (ObjNum 483 snap 126 seq_num 483) dirty exists 2026-02-25T08:46:07.967 INFO:tasks.rados.rados.0.trial023.stdout:1017: left oid 29 (ObjNum 483 snap 126 seq_num 483) 2026-02-25T08:46:07.967 INFO:tasks.rados.rados.0.trial023.stdout:1018: finishing write tid 1 to trial02342524-7 2026-02-25T08:46:07.967 INFO:tasks.rados.rados.0.trial023.stdout:1018: finishing write tid 2 to trial02342524-7 2026-02-25T08:46:07.967 INFO:tasks.rados.rados.0.trial023.stdout:1018: finishing write tid 3 to trial02342524-7 2026-02-25T08:46:07.967 INFO:tasks.rados.rados.0.trial023.stdout:1018: finishing write tid 4 to trial02342524-7 2026-02-25T08:46:07.967 INFO:tasks.rados.rados.0.trial023.stdout:1018: oid 7 updating version 0 to 171 2026-02-25T08:46:07.967 INFO:tasks.rados.rados.0.trial023.stdout:1018: oid 7 version 171 is already newer than 171 2026-02-25T08:46:07.968 INFO:tasks.rados.rados.0.trial023.stdout:1018: oid 7 version 171 is already newer than 171 2026-02-25T08:46:07.968 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 7 v 171 (ObjNum 484 snap 126 seq_num 484) dirty exists 2026-02-25T08:46:07.968 INFO:tasks.rados.rados.0.trial023.stdout:1018: left oid 7 (ObjNum 484 snap 126 seq_num 484) 2026-02-25T08:46:07.968 INFO:tasks.rados.rados.0.trial023.stdout:1019: finishing write tid 1 to trial02342524-14 2026-02-25T08:46:07.968 INFO:tasks.rados.rados.0.trial023.stdout:1019: finishing write tid 2 to trial02342524-14 2026-02-25T08:46:07.968 INFO:tasks.rados.rados.0.trial023.stdout:1019: finishing write tid 3 to trial02342524-14 2026-02-25T08:46:07.968 INFO:tasks.rados.rados.0.trial023.stdout:1019: finishing write tid 4 to trial02342524-14 2026-02-25T08:46:07.968 INFO:tasks.rados.rados.0.trial023.stdout:1019: oid 14 updating version 0 to 84 2026-02-25T08:46:07.968 INFO:tasks.rados.rados.0.trial023.stdout:1019: oid 14 updating version 84 to 86 2026-02-25T08:46:07.968 INFO:tasks.rados.rados.0.trial023.stdout:1019: oid 14 version 86 is already newer than 86 2026-02-25T08:46:07.968 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 14 v 86 (ObjNum 485 snap 126 seq_num 485) dirty exists 2026-02-25T08:46:07.968 INFO:tasks.rados.rados.0.trial023.stdout:1019: left oid 14 (ObjNum 485 snap 126 seq_num 485) 2026-02-25T08:46:07.998 INFO:tasks.rados.rados.0.trial023.stdout:1014: done (4 left) 2026-02-25T08:46:07.998 INFO:tasks.rados.rados.0.trial023.stdout:1017: done (3 left) 2026-02-25T08:46:07.998 INFO:tasks.rados.rados.0.trial023.stdout:1018: done (2 left) 2026-02-25T08:46:07.998 INFO:tasks.rados.rados.0.trial023.stdout:1019: done (1 left) 2026-02-25T08:46:07.998 INFO:tasks.rados.rados.0.trial023.stdout:1023: done (0 left) 2026-02-25T08:46:07.999 INFO:tasks.rados.rados.0.trial023.stdout:1024: read oid 29 snap 123 2026-02-25T08:46:07.999 INFO:tasks.rados.rados.0.trial023.stdout:1024: expect deleted 2026-02-25T08:46:07.999 INFO:tasks.rados.rados.0.trial023.stdout:1025: read oid 47 snap -1 2026-02-25T08:46:07.999 INFO:tasks.rados.rados.0.trial023.stdout:1025: expect (ObjNum 481 snap 126 seq_num 481) 2026-02-25T08:46:07.999 INFO:tasks.rados.rados.0.trial023.stdout:1026: read oid 42 snap -1 2026-02-25T08:46:07.999 INFO:tasks.rados.rados.0.trial023.stdout:1026: expect deleted 2026-02-25T08:46:07.999 INFO:tasks.rados.rados.0.trial023.stdout:1027: write oid 48 current snap is 126 2026-02-25T08:46:07.999 INFO:tasks.rados.rados.0.trial023.stdout:1027: seq_num 487 ranges {585009=492855,1564966=620043} 2026-02-25T08:46:08.002 INFO:tasks.rados.rados.0.trial023.stdout:1027: writing trial02342524-48 from 585009 to 1077864 tid 1 2026-02-25T08:46:08.007 INFO:tasks.rados.rados.0.trial023.stdout:1027: writing trial02342524-48 from 1564966 to 2185009 tid 2 2026-02-25T08:46:08.007 INFO:tasks.rados.rados.0.trial023.stdout:1028: snap_create 2026-02-25T08:46:08.021 INFO:tasks.rados.rados.0.trial023.stdout:1025: expect (ObjNum 481 snap 126 seq_num 481) 2026-02-25T08:46:08.026 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1158038 2026-02-25T08:46:08.045 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:46:08.100 INFO:tasks.rados.rados.0.trial023.stdout:1027: finishing write tid 1 to trial02342524-48 2026-02-25T08:46:08.100 INFO:tasks.rados.rados.0.trial023.stdout:1027: finishing write tid 2 to trial02342524-48 2026-02-25T08:46:08.100 INFO:tasks.rados.rados.0.trial023.stdout:1027: finishing write tid 3 to trial02342524-48 2026-02-25T08:46:08.100 INFO:tasks.rados.rados.0.trial023.stdout:1027: finishing write tid 4 to trial02342524-48 2026-02-25T08:46:08.100 INFO:tasks.rados.rados.0.trial023.stdout:1027: oid 48 updating version 0 to 206 2026-02-25T08:46:08.100 INFO:tasks.rados.rados.0.trial023.stdout:1027: oid 48 version 206 is already newer than 206 2026-02-25T08:46:08.101 INFO:tasks.rados.rados.0.trial023.stdout:1027: oid 48 version 206 is already newer than 206 2026-02-25T08:46:08.101 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 48 v 206 (ObjNum 487 snap 126 seq_num 487) dirty exists 2026-02-25T08:46:08.101 INFO:tasks.rados.rados.0.trial023.stdout:1027: left oid 48 (ObjNum 487 snap 126 seq_num 487) 2026-02-25T08:46:08.969 INFO:tasks.rados.rados.0.trial023.stdout:1024: done (4 left) 2026-02-25T08:46:08.969 INFO:tasks.rados.rados.0.trial023.stdout:1025: done (3 left) 2026-02-25T08:46:08.969 INFO:tasks.rados.rados.0.trial023.stdout:1026: done (2 left) 2026-02-25T08:46:08.969 INFO:tasks.rados.rados.0.trial023.stdout:1027: done (1 left) 2026-02-25T08:46:08.969 INFO:tasks.rados.rados.0.trial023.stdout:1028: done (0 left) 2026-02-25T08:46:08.969 INFO:tasks.rados.rados.0.trial023.stdout:1029: delete oid 16 current snap is 127 2026-02-25T08:46:08.991 INFO:tasks.rados.rados.0.trial023.stdout:1029: done (0 left) 2026-02-25T08:46:08.991 INFO:tasks.rados.rados.0.trial023.stdout:1030: snap_create 2026-02-25T08:46:09.455 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:46:09.531 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:09.529+0000 7fdf5fc5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:09.533 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:09.532+0000 7fdf5fc5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:09.548 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:46:09.575 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:09.576 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:09.840 INFO:tasks.rados.rados.0.trial023.stdout:1030: done (0 left) 2026-02-25T08:46:09.841 INFO:tasks.rados.rados.0.trial023.stdout:1031: write (excl) oid 50 current snap is 128 2026-02-25T08:46:09.841 INFO:tasks.rados.rados.0.trial023.stdout:1031: seq_num 489 ranges {618623=643353,1681930=536693} 2026-02-25T08:46:09.845 INFO:tasks.rados.rados.0.trial023.stdout:1031: writing trial02342524-50 from 618623 to 1261976 tid 1 2026-02-25T08:46:09.850 INFO:tasks.rados.rados.0.trial023.stdout:1031: writing trial02342524-50 from 1681930 to 2218623 tid 2 2026-02-25T08:46:09.850 INFO:tasks.rados.rados.0.trial023.stdout:1032: read oid 37 snap 126 2026-02-25T08:46:09.850 INFO:tasks.rados.rados.0.trial023.stdout:1032: expect deleted 2026-02-25T08:46:09.850 INFO:tasks.rados.rados.0.trial023.stdout:1033: read oid 42 snap -1 2026-02-25T08:46:09.850 INFO:tasks.rados.rados.0.trial023.stdout:1033: expect deleted 2026-02-25T08:46:09.851 INFO:tasks.rados.rados.0.trial023.stdout:1034: write oid 38 current snap is 128 2026-02-25T08:46:09.851 INFO:tasks.rados.rados.0.trial023.stdout:1034: seq_num 490 ranges {635430=718602,2140412=95018} 2026-02-25T08:46:09.856 INFO:tasks.rados.rados.0.trial023.stdout:1034: writing trial02342524-38 from 635430 to 1354032 tid 1 2026-02-25T08:46:09.860 INFO:tasks.rados.rados.0.trial023.stdout:1034: writing trial02342524-38 from 2140412 to 2235430 tid 2 2026-02-25T08:46:09.860 INFO:tasks.rados.rados.0.trial023.stdout:1035: read oid 15 snap -1 2026-02-25T08:46:09.860 INFO:tasks.rados.rados.0.trial023.stdout:1035: expect deleted 2026-02-25T08:46:09.860 INFO:tasks.rados.rados.0.trial023.stdout:1036: write oid 27 current snap is 128 2026-02-25T08:46:09.860 INFO:tasks.rados.rados.0.trial023.stdout:1036: seq_num 491 ranges {652237=793851,2198894=53343} 2026-02-25T08:46:09.865 INFO:tasks.rados.rados.0.trial023.stdout:1036: writing trial02342524-27 from 652237 to 1446088 tid 1 2026-02-25T08:46:09.869 INFO:tasks.rados.rados.0.trial023.stdout:1036: writing trial02342524-27 from 2198894 to 2252237 tid 2 2026-02-25T08:46:09.869 INFO:tasks.rados.rados.0.trial023.stdout:1037: write oid 43 current snap is 128 2026-02-25T08:46:09.869 INFO:tasks.rados.rados.0.trial023.stdout:1037: seq_num 492 ranges {669044=469100,1741023=459161} 2026-02-25T08:46:09.873 INFO:tasks.rados.rados.0.trial023.stdout:1037: writing trial02342524-43 from 669044 to 1138144 tid 1 2026-02-25T08:46:09.878 INFO:tasks.rados.rados.0.trial023.stdout:1037: writing trial02342524-43 from 1741023 to 2200184 tid 2 2026-02-25T08:46:09.878 INFO:tasks.rados.rados.0.trial023.stdout:1038: snap_remove snap 127 2026-02-25T08:46:10.102 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~270s 2026-02-25T08:46:10.668 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:46:10.741 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:46:10.743 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:46:10.744 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:46:10.843 INFO:tasks.rados.rados.0.trial023.stdout:1038: done (7 left) 2026-02-25T08:46:10.843 INFO:tasks.rados.rados.0.trial023.stdout:1039: read oid 14 snap -1 2026-02-25T08:46:10.843 INFO:tasks.rados.rados.0.trial023.stdout:1039: expect (ObjNum 485 snap 126 seq_num 485) 2026-02-25T08:46:10.843 INFO:tasks.rados.rados.0.trial023.stdout:1040: snap_create 2026-02-25T08:46:10.843 INFO:tasks.rados.rados.0.trial023.stdout:1031: finishing write tid 1 to trial02342524-50 2026-02-25T08:46:10.843 INFO:tasks.rados.rados.0.trial023.stdout:1036: finishing write tid 1 to trial02342524-27 2026-02-25T08:46:10.843 INFO:tasks.rados.rados.0.trial023.stdout:1036: finishing write tid 2 to trial02342524-27 2026-02-25T08:46:10.843 INFO:tasks.rados.rados.0.trial023.stdout:1031: finishing write tid 2 to trial02342524-50 2026-02-25T08:46:10.844 INFO:tasks.rados.rados.0.trial023.stdout:1037: finishing write tid 1 to trial02342524-43 2026-02-25T08:46:10.844 INFO:tasks.rados.rados.0.trial023.stdout:1036: finishing write tid 3 to trial02342524-27 2026-02-25T08:46:10.844 INFO:tasks.rados.rados.0.trial023.stdout:1036: finishing write tid 4 to trial02342524-27 2026-02-25T08:46:10.844 INFO:tasks.rados.rados.0.trial023.stdout:1036: oid 27 updating version 0 to 65 2026-02-25T08:46:10.844 INFO:tasks.rados.rados.0.trial023.stdout:1036: oid 27 version 65 is already newer than 65 2026-02-25T08:46:10.844 INFO:tasks.rados.rados.0.trial023.stdout:1036: oid 27 version 65 is already newer than 65 2026-02-25T08:46:10.844 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 27 v 65 (ObjNum 491 snap 128 seq_num 491) dirty exists 2026-02-25T08:46:10.844 INFO:tasks.rados.rados.0.trial023.stdout:1036: left oid 27 (ObjNum 491 snap 128 seq_num 491) 2026-02-25T08:46:10.844 INFO:tasks.rados.rados.0.trial023.stdout:1037: finishing write tid 2 to trial02342524-43 2026-02-25T08:46:10.844 INFO:tasks.rados.rados.0.trial023.stdout:1031: finishing write tid 3 to trial02342524-50 2026-02-25T08:46:10.844 INFO:tasks.rados.rados.0.trial023.stdout:1031: finishing write tid 4 to trial02342524-50 2026-02-25T08:46:10.844 INFO:tasks.rados.rados.0.trial023.stdout:1031: oid 50 updating version 0 to 234 2026-02-25T08:46:10.844 INFO:tasks.rados.rados.0.trial023.stdout:1031: oid 50 updating version 234 to 236 2026-02-25T08:46:10.845 INFO:tasks.rados.rados.0.trial023.stdout:1031: oid 50 version 236 is already newer than 236 2026-02-25T08:46:10.845 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 50 v 236 (ObjNum 489 snap 128 seq_num 489) dirty exists 2026-02-25T08:46:10.845 INFO:tasks.rados.rados.0.trial023.stdout:1031: left oid 50 (ObjNum 489 snap 128 seq_num 489) 2026-02-25T08:46:10.845 INFO:tasks.rados.rados.0.trial023.stdout:1034: finishing write tid 1 to trial02342524-38 2026-02-25T08:46:10.845 INFO:tasks.rados.rados.0.trial023.stdout:1037: finishing write tid 3 to trial02342524-43 2026-02-25T08:46:10.845 INFO:tasks.rados.rados.0.trial023.stdout:1037: finishing write tid 4 to trial02342524-43 2026-02-25T08:46:10.845 INFO:tasks.rados.rados.0.trial023.stdout:1037: oid 43 updating version 0 to 31 2026-02-25T08:46:10.845 INFO:tasks.rados.rados.0.trial023.stdout:1037: oid 43 version 31 is already newer than 31 2026-02-25T08:46:10.845 INFO:tasks.rados.rados.0.trial023.stdout:1037: oid 43 version 31 is already newer than 31 2026-02-25T08:46:10.845 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 43 v 31 (ObjNum 492 snap 128 seq_num 492) dirty exists 2026-02-25T08:46:10.845 INFO:tasks.rados.rados.0.trial023.stdout:1037: left oid 43 (ObjNum 492 snap 128 seq_num 492) 2026-02-25T08:46:10.845 INFO:tasks.rados.rados.0.trial023.stdout:1034: finishing write tid 2 to trial02342524-38 2026-02-25T08:46:10.846 INFO:tasks.rados.rados.0.trial023.stdout:1034: finishing write tid 3 to trial02342524-38 2026-02-25T08:46:10.846 INFO:tasks.rados.rados.0.trial023.stdout:1034: finishing write tid 4 to trial02342524-38 2026-02-25T08:46:10.846 INFO:tasks.rados.rados.0.trial023.stdout:1034: oid 38 updating version 0 to 180 2026-02-25T08:46:10.846 INFO:tasks.rados.rados.0.trial023.stdout:1034: oid 38 version 180 is already newer than 180 2026-02-25T08:46:10.846 INFO:tasks.rados.rados.0.trial023.stdout:1034: oid 38 version 180 is already newer than 180 2026-02-25T08:46:10.846 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 38 v 180 (ObjNum 490 snap 128 seq_num 490) dirty exists 2026-02-25T08:46:10.846 INFO:tasks.rados.rados.0.trial023.stdout:1034: left oid 38 (ObjNum 490 snap 128 seq_num 490) 2026-02-25T08:46:10.883 INFO:tasks.rados.rados.0.trial023.stdout:1039: expect (ObjNum 485 snap 126 seq_num 485) 2026-02-25T08:46:10.913 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:46:10.943 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:46:11.578 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:46:11.655 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:11.652+0000 7f29bcb64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:11.657 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:11.656+0000 7f29bcb64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:11.673 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:46:11.701 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:11.701 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:11.844 INFO:tasks.rados.rados.0.trial023.stdout:1031: done (8 left) 2026-02-25T08:46:11.844 INFO:tasks.rados.rados.0.trial023.stdout:1032: done (7 left) 2026-02-25T08:46:11.844 INFO:tasks.rados.rados.0.trial023.stdout:1033: done (6 left) 2026-02-25T08:46:11.844 INFO:tasks.rados.rados.0.trial023.stdout:1034: done (5 left) 2026-02-25T08:46:11.845 INFO:tasks.rados.rados.0.trial023.stdout:1035: done (4 left) 2026-02-25T08:46:11.845 INFO:tasks.rados.rados.0.trial023.stdout:1036: done (3 left) 2026-02-25T08:46:11.845 INFO:tasks.rados.rados.0.trial023.stdout:1037: done (2 left) 2026-02-25T08:46:11.845 INFO:tasks.rados.rados.0.trial023.stdout:1039: done (1 left) 2026-02-25T08:46:11.845 INFO:tasks.rados.rados.0.trial023.stdout:1040: done (0 left) 2026-02-25T08:46:11.845 INFO:tasks.rados.rados.0.trial023.stdout:1041: write (excl) oid 17 current snap is 129 2026-02-25T08:46:11.845 INFO:tasks.rados.rados.0.trial023.stdout:1041: seq_num 493 ranges {685851=544349,1799505=486346} 2026-02-25T08:46:11.849 INFO:tasks.rados.rados.0.trial023.stdout:1041: writing trial02342524-17 from 685851 to 1230200 tid 1 2026-02-25T08:46:11.854 INFO:tasks.rados.rados.0.trial023.stdout:1041: writing trial02342524-17 from 1799505 to 2285851 tid 2 2026-02-25T08:46:11.854 INFO:tasks.rados.rados.0.trial023.stdout:1042: snap_create 2026-02-25T08:46:11.882 INFO:tasks.rados.rados.0.trial023.stdout:1041: finishing write tid 1 to trial02342524-17 2026-02-25T08:46:11.896 INFO:tasks.rados.rados.0.trial023.stdout:1041: finishing write tid 2 to trial02342524-17 2026-02-25T08:46:11.902 INFO:tasks.rados.rados.0.trial023.stdout:1041: finishing write tid 3 to trial02342524-17 2026-02-25T08:46:11.904 INFO:tasks.rados.rados.0.trial023.stdout:1041: finishing write tid 4 to trial02342524-17 2026-02-25T08:46:11.904 INFO:tasks.rados.rados.0.trial023.stdout:1041: oid 17 updating version 0 to 26 2026-02-25T08:46:11.904 INFO:tasks.rados.rados.0.trial023.stdout:1041: oid 17 version 26 is already newer than 24 2026-02-25T08:46:11.904 INFO:tasks.rados.rados.0.trial023.stdout:1041: oid 17 version 26 is already newer than 26 2026-02-25T08:46:11.905 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 17 v 26 (ObjNum 493 snap 129 seq_num 493) dirty exists 2026-02-25T08:46:11.905 INFO:tasks.rados.rados.0.trial023.stdout:1041: left oid 17 (ObjNum 493 snap 129 seq_num 493) 2026-02-25T08:46:12.847 INFO:tasks.rados.rados.0.trial023.stdout:1041: done (1 left) 2026-02-25T08:46:12.848 INFO:tasks.rados.rados.0.trial023.stdout:1042: done (0 left) 2026-02-25T08:46:12.848 INFO:tasks.rados.rados.0.trial023.stdout:1043: delete oid 13 current snap is 130 2026-02-25T08:46:12.897 INFO:tasks.rados.rados.0.trial023.stdout:1043: done (0 left) 2026-02-25T08:46:12.898 INFO:tasks.rados.rados.0.trial023.stdout:1044: snap_create 2026-02-25T08:46:13.702 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:46:13.776 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:13.774+0000 7fbcd635b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:13.778 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:13.777+0000 7fbcd635b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:13.795 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:46:13.822 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:13.822 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:13.849 INFO:tasks.rados.rados.0.trial023.stdout:1044: done (0 left) 2026-02-25T08:46:13.849 INFO:tasks.rados.rados.0.trial023.stdout:1045: snap_remove snap 129 2026-02-25T08:46:14.852 INFO:tasks.rados.rados.0.trial023.stdout:1045: done (0 left) 2026-02-25T08:46:14.852 INFO:tasks.rados.rados.0.trial023.stdout:1046: delete oid 1 current snap is 131 2026-02-25T08:46:14.895 INFO:tasks.rados.rados.0.trial023.stdout:1046: done (0 left) 2026-02-25T08:46:14.895 INFO:tasks.rados.rados.0.trial023.stdout:1047: delete oid 38 current snap is 131 2026-02-25T08:46:14.925 INFO:tasks.rados.rados.0.trial023.stdout:1047: done (0 left) 2026-02-25T08:46:14.925 INFO:tasks.rados.rados.0.trial023.stdout:1048: write (excl) oid 33 current snap is 131 2026-02-25T08:46:14.925 INFO:tasks.rados.rados.0.trial023.stdout:1048: seq_num 497 ranges {753079=561698,2033433=319646} 2026-02-25T08:46:14.930 INFO:tasks.rados.rados.0.trial023.stdout:1048: writing trial02342524-33 from 753079 to 1314777 tid 1 2026-02-25T08:46:14.934 INFO:tasks.rados.rados.0.trial023.stdout:1048: writing trial02342524-33 from 2033433 to 2353079 tid 2 2026-02-25T08:46:14.934 INFO:tasks.rados.rados.0.trial023.stdout:1049: snap_create 2026-02-25T08:46:14.962 INFO:tasks.rados.rados.0.trial023.stdout:1048: finishing write tid 1 to trial02342524-33 2026-02-25T08:46:14.974 INFO:tasks.rados.rados.0.trial023.stdout:1048: finishing write tid 2 to trial02342524-33 2026-02-25T08:46:14.982 INFO:tasks.rados.rados.0.trial023.stdout:1048: finishing write tid 3 to trial02342524-33 2026-02-25T08:46:14.984 INFO:tasks.rados.rados.0.trial023.stdout:1048: finishing write tid 4 to trial02342524-33 2026-02-25T08:46:14.984 INFO:tasks.rados.rados.0.trial023.stdout:1048: oid 33 updating version 0 to 48 2026-02-25T08:46:14.984 INFO:tasks.rados.rados.0.trial023.stdout:1048: oid 33 version 48 is already newer than 46 2026-02-25T08:46:14.984 INFO:tasks.rados.rados.0.trial023.stdout:1048: oid 33 version 48 is already newer than 48 2026-02-25T08:46:14.984 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 33 v 48 (ObjNum 497 snap 131 seq_num 497) dirty exists 2026-02-25T08:46:14.984 INFO:tasks.rados.rados.0.trial023.stdout:1048: left oid 33 (ObjNum 497 snap 131 seq_num 497) 2026-02-25T08:46:15.606 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~275s 2026-02-25T08:46:15.746 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:46:15.818 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:46:15.821 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:46:15.821 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:46:15.824 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:46:15.857 INFO:tasks.rados.rados.0.trial023.stdout:1048: done (1 left) 2026-02-25T08:46:15.858 INFO:tasks.rados.rados.0.trial023.stdout:1049: done (0 left) 2026-02-25T08:46:15.858 INFO:tasks.rados.rados.0.trial023.stdout:1050: read oid 17 snap -1 2026-02-25T08:46:15.858 INFO:tasks.rados.rados.0.trial023.stdout:1050: expect (ObjNum 493 snap 129 seq_num 493) 2026-02-25T08:46:15.858 INFO:tasks.rados.rados.0.trial023.stdout:1051: read oid 9 snap -1 2026-02-25T08:46:15.858 INFO:tasks.rados.rados.0.trial023.stdout:1051: expect deleted 2026-02-25T08:46:15.858 INFO:tasks.rados.rados.0.trial023.stdout:1052: read oid 30 snap 130 2026-02-25T08:46:15.858 INFO:tasks.rados.rados.0.trial023.stdout:1052: expect (ObjNum 458 snap 121 seq_num 458) 2026-02-25T08:46:15.858 INFO:tasks.rados.rados.0.trial023.stdout:1053: snap_remove snap 123 2026-02-25T08:46:15.900 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:15.898+0000 7f1137d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:15.903 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:15.902+0000 7f1137d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:15.917 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:46:15.943 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:15.944 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:16.862 INFO:tasks.rados.rados.0.trial023.stdout:1053: done (3 left) 2026-02-25T08:46:16.862 INFO:tasks.rados.rados.0.trial023.stdout:1054: snap_create 2026-02-25T08:46:16.862 INFO:tasks.rados.rados.0.trial023.stdout:1050: expect (ObjNum 493 snap 129 seq_num 493) 2026-02-25T08:46:16.862 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:46:16.893 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1625490 2026-02-25T08:46:16.901 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:46:16.925 INFO:tasks.rados.rados.0.trial023.stdout:1052: expect (ObjNum 458 snap 121 seq_num 458) 2026-02-25T08:46:16.977 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:46:17.862 INFO:tasks.rados.rados.0.trial023.stdout:1050: done (3 left) 2026-02-25T08:46:17.862 INFO:tasks.rados.rados.0.trial023.stdout:1051: done (2 left) 2026-02-25T08:46:17.862 INFO:tasks.rados.rados.0.trial023.stdout:1052: done (1 left) 2026-02-25T08:46:17.862 INFO:tasks.rados.rados.0.trial023.stdout:1054: done (0 left) 2026-02-25T08:46:17.863 INFO:tasks.rados.rados.0.trial023.stdout:1055: read oid 43 snap -1 2026-02-25T08:46:17.863 INFO:tasks.rados.rados.0.trial023.stdout:1055: expect (ObjNum 492 snap 128 seq_num 492) 2026-02-25T08:46:17.863 INFO:tasks.rados.rados.0.trial023.stdout:1056: snap_remove snap 132 2026-02-25T08:46:17.944 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:46:18.020 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:18.017+0000 7fe66e16f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:18.022 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:18.021+0000 7fe66e16f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:18.037 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:46:18.065 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:18.065 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:18.868 INFO:tasks.rados.rados.0.trial023.stdout:1056: done (1 left) 2026-02-25T08:46:18.868 INFO:tasks.rados.rados.0.trial023.stdout:1057: read oid 45 snap -1 2026-02-25T08:46:18.868 INFO:tasks.rados.rados.0.trial023.stdout:1057: expect (ObjNum 341 snap 97 seq_num 341) 2026-02-25T08:46:18.868 INFO:tasks.rados.rados.0.trial023.stdout:1058: read oid 7 snap -1 2026-02-25T08:46:18.868 INFO:tasks.rados.rados.0.trial023.stdout:1058: expect (ObjNum 484 snap 126 seq_num 484) 2026-02-25T08:46:18.869 INFO:tasks.rados.rados.0.trial023.stdout:1059: write oid 41 current snap is 133 2026-02-25T08:46:18.869 INFO:tasks.rados.rados.0.trial023.stdout:1059: seq_num 498 ranges {769886=636947,2091915=277971} 2026-02-25T08:46:18.873 INFO:tasks.rados.rados.0.trial023.stdout:1059: writing trial02342524-41 from 769886 to 1406833 tid 1 2026-02-25T08:46:18.876 INFO:tasks.rados.rados.0.trial023.stdout:1059: writing trial02342524-41 from 2091915 to 2369886 tid 2 2026-02-25T08:46:18.877 INFO:tasks.rados.rados.0.trial023.stdout:1060: write (excl) oid 33 current snap is 133 2026-02-25T08:46:18.877 INFO:tasks.rados.rados.0.trial023.stdout:1060: seq_num 499 ranges {786693=712196,2150397=236296} 2026-02-25T08:46:18.882 INFO:tasks.rados.rados.0.trial023.stdout:1060: writing trial02342524-33 from 786693 to 1498889 tid 1 2026-02-25T08:46:18.885 INFO:tasks.rados.rados.0.trial023.stdout:1060: writing trial02342524-33 from 2150397 to 2386693 tid 2 2026-02-25T08:46:18.885 INFO:tasks.rados.rados.0.trial023.stdout:1061: write (excl) oid 18 current snap is 133 2026-02-25T08:46:18.885 INFO:tasks.rados.rados.0.trial023.stdout:1061: seq_num 500 ranges {403500=787445,1692526=473837} 2026-02-25T08:46:18.890 INFO:tasks.rados.rados.0.trial023.stdout:1061: writing trial02342524-18 from 403500 to 1190945 tid 1 2026-02-25T08:46:18.894 INFO:tasks.rados.rados.0.trial023.stdout:1061: writing trial02342524-18 from 1692526 to 2166363 tid 2 2026-02-25T08:46:18.894 INFO:tasks.rados.rados.0.trial023.stdout:1062: read oid 2 snap 126 2026-02-25T08:46:18.894 INFO:tasks.rados.rados.0.trial023.stdout:1062: expect (ObjNum 425 snap 116 seq_num 425) 2026-02-25T08:46:18.894 INFO:tasks.rados.rados.0.trial023.stdout:1063: read oid 15 snap -1 2026-02-25T08:46:18.894 INFO:tasks.rados.rados.0.trial023.stdout:1063: expect deleted 2026-02-25T08:46:18.895 INFO:tasks.rados.rados.0.trial023.stdout:1064: write (excl) oid 14 current snap is 133 2026-02-25T08:46:18.895 INFO:tasks.rados.rados.0.trial023.stdout:1064: seq_num 501 ranges {420307=462694,1351008=617495} 2026-02-25T08:46:18.897 INFO:tasks.rados.rados.0.trial023.stdout:1064: writing trial02342524-14 from 420307 to 883001 tid 1 2026-02-25T08:46:18.901 INFO:tasks.rados.rados.0.trial023.stdout:1064: writing trial02342524-14 from 1351008 to 1968503 tid 2 2026-02-25T08:46:18.901 INFO:tasks.rados.rados.0.trial023.stdout:1065: write oid 25 current snap is 133 2026-02-25T08:46:18.901 INFO:tasks.rados.rados.0.trial023.stdout:1055: expect (ObjNum 492 snap 128 seq_num 492) 2026-02-25T08:46:18.902 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:46:18.962 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1386117 2026-02-25T08:46:18.976 INFO:tasks.rados.rados.0.trial023.stdout:1065: seq_num 502 ranges {437114=654296,1525843=477506} 2026-02-25T08:46:18.980 INFO:tasks.rados.rados.0.trial023.stdout:1065: writing trial02342524-25 from 437114 to 1091410 tid 1 2026-02-25T08:46:18.984 INFO:tasks.rados.rados.0.trial023.stdout:1065: writing trial02342524-25 from 1525843 to 2003349 tid 2 2026-02-25T08:46:18.984 INFO:tasks.rados.rados.0.trial023.stdout:1055: done (9 left) 2026-02-25T08:46:18.984 INFO:tasks.rados.rados.0.trial023.stdout:1066: write (excl) oid 11 current snap is 133 2026-02-25T08:46:18.984 INFO:tasks.rados.rados.0.trial023.stdout:1066: seq_num 503 ranges {453921=729545,1584325=621164} 2026-02-25T08:46:18.988 INFO:tasks.rados.rados.0.trial023.stdout:1066: writing trial02342524-11 from 453921 to 1183466 tid 1 2026-02-25T08:46:18.992 INFO:tasks.rados.rados.0.trial023.stdout:1066: writing trial02342524-11 from 1584325 to 2205489 tid 2 2026-02-25T08:46:18.993 INFO:tasks.rados.rados.0.trial023.stdout:1067: write oid 12 current snap is 133 2026-02-25T08:46:18.993 INFO:tasks.rados.rados.0.trial023.stdout:1067: seq_num 504 ranges {470728=404794,1526454=481175} 2026-02-25T08:46:18.995 INFO:tasks.rados.rados.0.trial023.stdout:1067: writing trial02342524-12 from 470728 to 875522 tid 1 2026-02-25T08:46:19.000 INFO:tasks.rados.rados.0.trial023.stdout:1067: writing trial02342524-12 from 1526454 to 2007629 tid 2 2026-02-25T08:46:19.000 INFO:tasks.rados.rados.0.trial023.stdout:1063: done (10 left) 2026-02-25T08:46:19.001 INFO:tasks.rados.rados.0.trial023.stdout:1068: read oid 39 snap -1 2026-02-25T08:46:19.001 INFO:tasks.rados.rados.0.trial023.stdout:1068: expect (ObjNum 454 snap 121 seq_num 454) 2026-02-25T08:46:19.001 INFO:tasks.rados.rados.0.trial023.stdout:1069: snap_remove snap 128 2026-02-25T08:46:19.842 INFO:tasks.rados.rados.0.trial023.stdout:1069: done (11 left) 2026-02-25T08:46:19.842 INFO:tasks.rados.rados.0.trial023.stdout:1070: snap_create 2026-02-25T08:46:19.843 INFO:tasks.rados.rados.0.trial023.stdout:1057: expect (ObjNum 341 snap 97 seq_num 341) 2026-02-25T08:46:19.844 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2006716 2026-02-25T08:46:19.866 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 567875 2026-02-25T08:46:19.923 INFO:tasks.rados.rados.0.trial023.stdout:1061: finishing write tid 1 to trial02342524-18 2026-02-25T08:46:19.923 INFO:tasks.rados.rados.0.trial023.stdout:1058: expect (ObjNum 484 snap 126 seq_num 484) 2026-02-25T08:46:19.925 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1765055 2026-02-25T08:46:19.932 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:46:19.950 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:46:19.969 INFO:tasks.rados.rados.0.trial023.stdout:1061: finishing write tid 2 to trial02342524-18 2026-02-25T08:46:19.969 INFO:tasks.rados.rados.0.trial023.stdout:1059: finishing write tid 1 to trial02342524-41 2026-02-25T08:46:19.969 INFO:tasks.rados.rados.0.trial023.stdout:1061: finishing write tid 3 to trial02342524-18 2026-02-25T08:46:19.970 INFO:tasks.rados.rados.0.trial023.stdout:1061: finishing write tid 4 to trial02342524-18 2026-02-25T08:46:19.970 INFO:tasks.rados.rados.0.trial023.stdout:1061: oid 18 updating version 0 to 185 2026-02-25T08:46:19.970 INFO:tasks.rados.rados.0.trial023.stdout:1061: oid 18 version 185 is already newer than 185 2026-02-25T08:46:19.970 INFO:tasks.rados.rados.0.trial023.stdout:1061: oid 18 version 185 is already newer than 183 2026-02-25T08:46:19.970 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 18 v 185 (ObjNum 500 snap 133 seq_num 500) dirty exists 2026-02-25T08:46:19.970 INFO:tasks.rados.rados.0.trial023.stdout:1061: left oid 18 (ObjNum 500 snap 133 seq_num 500) 2026-02-25T08:46:19.970 INFO:tasks.rados.rados.0.trial023.stdout:1059: finishing write tid 2 to trial02342524-41 2026-02-25T08:46:19.970 INFO:tasks.rados.rados.0.trial023.stdout:1060: finishing write tid 1 to trial02342524-33 2026-02-25T08:46:19.970 INFO:tasks.rados.rados.0.trial023.stdout:1062: expect (ObjNum 425 snap 116 seq_num 425) 2026-02-25T08:46:19.990 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2146245 2026-02-25T08:46:20.052 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1291946 2026-02-25T08:46:20.067 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:46:20.082 INFO:tasks.rados.rados.0.trial023.stdout:1059: finishing write tid 3 to trial02342524-41 2026-02-25T08:46:20.083 INFO:tasks.rados.rados.0.trial023.stdout:1059: finishing write tid 4 to trial02342524-41 2026-02-25T08:46:20.083 INFO:tasks.rados.rados.0.trial023.stdout:1059: oid 41 updating version 0 to 174 2026-02-25T08:46:20.083 INFO:tasks.rados.rados.0.trial023.stdout:1059: oid 41 version 174 is already newer than 174 2026-02-25T08:46:20.083 INFO:tasks.rados.rados.0.trial023.stdout:1059: oid 41 version 174 is already newer than 174 2026-02-25T08:46:20.083 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 41 v 174 (ObjNum 498 snap 133 seq_num 498) dirty exists 2026-02-25T08:46:20.083 INFO:tasks.rados.rados.0.trial023.stdout:1059: left oid 41 (ObjNum 498 snap 133 seq_num 498) 2026-02-25T08:46:20.083 INFO:tasks.rados.rados.0.trial023.stdout:1060: finishing write tid 2 to trial02342524-33 2026-02-25T08:46:20.083 INFO:tasks.rados.rados.0.trial023.stdout:1060: finishing write tid 3 to trial02342524-33 2026-02-25T08:46:20.083 INFO:tasks.rados.rados.0.trial023.stdout:1060: finishing write tid 4 to trial02342524-33 2026-02-25T08:46:20.083 INFO:tasks.rados.rados.0.trial023.stdout:1060: oid 33 updating version 0 to 52 2026-02-25T08:46:20.084 INFO:tasks.rados.rados.0.trial023.stdout:1060: oid 33 version 52 is already newer than 52 2026-02-25T08:46:20.084 INFO:tasks.rados.rados.0.trial023.stdout:1060: oid 33 version 52 is already newer than 50 2026-02-25T08:46:20.084 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 33 v 52 (ObjNum 499 snap 133 seq_num 499) dirty exists 2026-02-25T08:46:20.084 INFO:tasks.rados.rados.0.trial023.stdout:1060: left oid 33 (ObjNum 499 snap 133 seq_num 499) 2026-02-25T08:46:20.084 INFO:tasks.rados.rados.0.trial023.stdout:1065: finishing write tid 1 to trial02342524-25 2026-02-25T08:46:20.084 INFO:tasks.rados.rados.0.trial023.stdout:1064: finishing write tid 1 to trial02342524-14 2026-02-25T08:46:20.084 INFO:tasks.rados.rados.0.trial023.stdout:1065: finishing write tid 2 to trial02342524-25 2026-02-25T08:46:20.084 INFO:tasks.rados.rados.0.trial023.stdout:1068: expect (ObjNum 454 snap 121 seq_num 454) 2026-02-25T08:46:20.084 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:46:20.141 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:20.139+0000 7f81f948b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:20.144 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:20.143+0000 7f81f948b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:20.160 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:46:20.174 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:46:20.188 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:20.189 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:20.224 INFO:tasks.rados.rados.0.trial023.stdout:1065: finishing write tid 3 to trial02342524-25 2026-02-25T08:46:20.224 INFO:tasks.rados.rados.0.trial023.stdout:1065: finishing write tid 4 to trial02342524-25 2026-02-25T08:46:20.224 INFO:tasks.rados.rados.0.trial023.stdout:1065: oid 25 updating version 0 to 240 2026-02-25T08:46:20.224 INFO:tasks.rados.rados.0.trial023.stdout:1065: oid 25 version 240 is already newer than 240 2026-02-25T08:46:20.224 INFO:tasks.rados.rados.0.trial023.stdout:1065: oid 25 version 240 is already newer than 240 2026-02-25T08:46:20.224 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 25 v 240 (ObjNum 502 snap 133 seq_num 502) dirty exists 2026-02-25T08:46:20.224 INFO:tasks.rados.rados.0.trial023.stdout:1065: left oid 25 (ObjNum 502 snap 133 seq_num 502) 2026-02-25T08:46:20.224 INFO:tasks.rados.rados.0.trial023.stdout:1066: finishing write tid 1 to trial02342524-11 2026-02-25T08:46:20.224 INFO:tasks.rados.rados.0.trial023.stdout:1067: finishing write tid 1 to trial02342524-12 2026-02-25T08:46:20.224 INFO:tasks.rados.rados.0.trial023.stdout:1066: finishing write tid 2 to trial02342524-11 2026-02-25T08:46:20.225 INFO:tasks.rados.rados.0.trial023.stdout:1066: finishing write tid 3 to trial02342524-11 2026-02-25T08:46:20.225 INFO:tasks.rados.rados.0.trial023.stdout:1066: finishing write tid 4 to trial02342524-11 2026-02-25T08:46:20.225 INFO:tasks.rados.rados.0.trial023.stdout:1066: oid 11 updating version 0 to 243 2026-02-25T08:46:20.225 INFO:tasks.rados.rados.0.trial023.stdout:1066: oid 11 version 243 is already newer than 243 2026-02-25T08:46:20.225 INFO:tasks.rados.rados.0.trial023.stdout:1066: oid 11 version 243 is already newer than 241 2026-02-25T08:46:20.225 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 11 v 243 (ObjNum 503 snap 133 seq_num 503) dirty exists 2026-02-25T08:46:20.225 INFO:tasks.rados.rados.0.trial023.stdout:1066: left oid 11 (ObjNum 503 snap 133 seq_num 503) 2026-02-25T08:46:20.225 INFO:tasks.rados.rados.0.trial023.stdout:1067: finishing write tid 2 to trial02342524-12 2026-02-25T08:46:20.225 INFO:tasks.rados.rados.0.trial023.stdout:1067: finishing write tid 3 to trial02342524-12 2026-02-25T08:46:20.225 INFO:tasks.rados.rados.0.trial023.stdout:1067: finishing write tid 4 to trial02342524-12 2026-02-25T08:46:20.225 INFO:tasks.rados.rados.0.trial023.stdout:1067: oid 12 updating version 0 to 180 2026-02-25T08:46:20.225 INFO:tasks.rados.rados.0.trial023.stdout:1067: oid 12 version 180 is already newer than 180 2026-02-25T08:46:20.225 INFO:tasks.rados.rados.0.trial023.stdout:1067: oid 12 version 180 is already newer than 180 2026-02-25T08:46:20.226 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 12 v 180 (ObjNum 504 snap 133 seq_num 504) dirty exists 2026-02-25T08:46:20.226 INFO:tasks.rados.rados.0.trial023.stdout:1067: left oid 12 (ObjNum 504 snap 133 seq_num 504) 2026-02-25T08:46:20.226 INFO:tasks.rados.rados.0.trial023.stdout:1064: finishing write tid 2 to trial02342524-14 2026-02-25T08:46:20.226 INFO:tasks.rados.rados.0.trial023.stdout:1064: finishing write tid 3 to trial02342524-14 2026-02-25T08:46:20.226 INFO:tasks.rados.rados.0.trial023.stdout:1064: finishing write tid 4 to trial02342524-14 2026-02-25T08:46:20.226 INFO:tasks.rados.rados.0.trial023.stdout:1064: oid 14 updating version 0 to 90 2026-02-25T08:46:20.226 INFO:tasks.rados.rados.0.trial023.stdout:1064: oid 14 version 90 is already newer than 90 2026-02-25T08:46:20.226 INFO:tasks.rados.rados.0.trial023.stdout:1064: oid 14 version 90 is already newer than 88 2026-02-25T08:46:20.226 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 14 v 90 (ObjNum 501 snap 133 seq_num 501) dirty exists 2026-02-25T08:46:20.226 INFO:tasks.rados.rados.0.trial023.stdout:1064: left oid 14 (ObjNum 501 snap 133 seq_num 501) 2026-02-25T08:46:20.823 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:46:20.845 INFO:tasks.rados.rados.0.trial023.stdout:1057: done (11 left) 2026-02-25T08:46:20.845 INFO:tasks.rados.rados.0.trial023.stdout:1058: done (10 left) 2026-02-25T08:46:20.845 INFO:tasks.rados.rados.0.trial023.stdout:1059: done (9 left) 2026-02-25T08:46:20.845 INFO:tasks.rados.rados.0.trial023.stdout:1060: done (8 left) 2026-02-25T08:46:20.845 INFO:tasks.rados.rados.0.trial023.stdout:1061: done (7 left) 2026-02-25T08:46:20.845 INFO:tasks.rados.rados.0.trial023.stdout:1062: done (6 left) 2026-02-25T08:46:20.845 INFO:tasks.rados.rados.0.trial023.stdout:1064: done (5 left) 2026-02-25T08:46:20.845 INFO:tasks.rados.rados.0.trial023.stdout:1065: done (4 left) 2026-02-25T08:46:20.845 INFO:tasks.rados.rados.0.trial023.stdout:1066: done (3 left) 2026-02-25T08:46:20.845 INFO:tasks.rados.rados.0.trial023.stdout:1067: done (2 left) 2026-02-25T08:46:20.846 INFO:tasks.rados.rados.0.trial023.stdout:1068: done (1 left) 2026-02-25T08:46:20.846 INFO:tasks.rados.rados.0.trial023.stdout:1070: done (0 left) 2026-02-25T08:46:20.846 INFO:tasks.rados.rados.0.trial023.stdout:1071: snap_create 2026-02-25T08:46:20.897 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:46:20.899 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:46:20.900 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:46:21.110 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~281s 2026-02-25T08:46:21.846 INFO:tasks.rados.rados.0.trial023.stdout:1071: done (0 left) 2026-02-25T08:46:21.847 INFO:tasks.rados.rados.0.trial023.stdout:1072: delete oid 25 current snap is 135 2026-02-25T08:46:21.893 INFO:tasks.rados.rados.0.trial023.stdout:1072: done (0 left) 2026-02-25T08:46:21.893 INFO:tasks.rados.rados.0.trial023.stdout:1073: write oid 35 current snap is 135 2026-02-25T08:46:21.893 INFO:tasks.rados.rados.0.trial023.stdout:1073: seq_num 506 ranges {504342=555292,1643418=484844} 2026-02-25T08:46:21.896 INFO:tasks.rados.rados.0.trial023.stdout:1073: writing trial02342524-35 from 504342 to 1059634 tid 1 2026-02-25T08:46:21.901 INFO:tasks.rados.rados.0.trial023.stdout:1073: writing trial02342524-35 from 1643418 to 2128262 tid 2 2026-02-25T08:46:21.901 INFO:tasks.rados.rados.0.trial023.stdout:1074: read oid 12 snap -1 2026-02-25T08:46:21.901 INFO:tasks.rados.rados.0.trial023.stdout:1074: expect (ObjNum 504 snap 133 seq_num 504) 2026-02-25T08:46:21.901 INFO:tasks.rados.rados.0.trial023.stdout:1075: read oid 42 snap -1 2026-02-25T08:46:21.901 INFO:tasks.rados.rados.0.trial023.stdout:1075: expect deleted 2026-02-25T08:46:21.901 INFO:tasks.rados.rados.0.trial023.stdout:1076: write (excl) oid 43 current snap is 135 2026-02-25T08:46:21.901 INFO:tasks.rados.rados.0.trial023.stdout:1076: seq_num 507 ranges {521149=630541,1701900=628502} 2026-02-25T08:46:21.905 INFO:tasks.rados.rados.0.trial023.stdout:1076: writing trial02342524-43 from 521149 to 1151690 tid 1 2026-02-25T08:46:21.910 INFO:tasks.rados.rados.0.trial023.stdout:1076: writing trial02342524-43 from 1701900 to 2330402 tid 2 2026-02-25T08:46:21.910 INFO:tasks.rados.rados.0.trial023.stdout:1077: read oid 9 snap 130 2026-02-25T08:46:21.910 INFO:tasks.rados.rados.0.trial023.stdout:1077: expect deleted 2026-02-25T08:46:21.911 INFO:tasks.rados.rados.0.trial023.stdout:1078: snap_create 2026-02-25T08:46:21.926 INFO:tasks.rados.rados.0.trial023.stdout:1074: expect (ObjNum 504 snap 133 seq_num 504) 2026-02-25T08:46:21.926 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:46:21.970 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2045636 2026-02-25T08:46:22.006 INFO:tasks.rados.rados.0.trial023.stdout:1076: finishing write tid 1 to trial02342524-43 2026-02-25T08:46:22.006 INFO:tasks.rados.rados.0.trial023.stdout:1076: finishing write tid 2 to trial02342524-43 2026-02-25T08:46:22.006 INFO:tasks.rados.rados.0.trial023.stdout:1076: finishing write tid 3 to trial02342524-43 2026-02-25T08:46:22.006 INFO:tasks.rados.rados.0.trial023.stdout:1076: finishing write tid 4 to trial02342524-43 2026-02-25T08:46:22.006 INFO:tasks.rados.rados.0.trial023.stdout:1076: oid 43 updating version 0 to 33 2026-02-25T08:46:22.006 INFO:tasks.rados.rados.0.trial023.stdout:1076: oid 43 updating version 33 to 35 2026-02-25T08:46:22.007 INFO:tasks.rados.rados.0.trial023.stdout:1076: oid 43 version 35 is already newer than 35 2026-02-25T08:46:22.007 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 43 v 35 (ObjNum 507 snap 135 seq_num 507) dirty exists 2026-02-25T08:46:22.007 INFO:tasks.rados.rados.0.trial023.stdout:1076: left oid 43 (ObjNum 507 snap 135 seq_num 507) 2026-02-25T08:46:22.007 INFO:tasks.rados.rados.0.trial023.stdout:1073: finishing write tid 1 to trial02342524-35 2026-02-25T08:46:22.007 INFO:tasks.rados.rados.0.trial023.stdout:1073: finishing write tid 2 to trial02342524-35 2026-02-25T08:46:22.007 INFO:tasks.rados.rados.0.trial023.stdout:1073: finishing write tid 3 to trial02342524-35 2026-02-25T08:46:22.007 INFO:tasks.rados.rados.0.trial023.stdout:1073: finishing write tid 4 to trial02342524-35 2026-02-25T08:46:22.007 INFO:tasks.rados.rados.0.trial023.stdout:1073: oid 35 updating version 0 to 282 2026-02-25T08:46:22.007 INFO:tasks.rados.rados.0.trial023.stdout:1073: oid 35 version 282 is already newer than 282 2026-02-25T08:46:22.007 INFO:tasks.rados.rados.0.trial023.stdout:1073: oid 35 version 282 is already newer than 282 2026-02-25T08:46:22.007 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 35 v 282 (ObjNum 506 snap 135 seq_num 506) dirty exists 2026-02-25T08:46:22.007 INFO:tasks.rados.rados.0.trial023.stdout:1073: left oid 35 (ObjNum 506 snap 135 seq_num 506) 2026-02-25T08:46:22.190 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:46:22.266 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:22.264+0000 7f16acb0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:22.269 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:22.268+0000 7f16acb0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:22.284 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:46:22.311 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:22.311 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:22.849 INFO:tasks.rados.rados.0.trial023.stdout:1073: done (5 left) 2026-02-25T08:46:22.849 INFO:tasks.rados.rados.0.trial023.stdout:1074: done (4 left) 2026-02-25T08:46:22.849 INFO:tasks.rados.rados.0.trial023.stdout:1075: done (3 left) 2026-02-25T08:46:22.849 INFO:tasks.rados.rados.0.trial023.stdout:1076: done (2 left) 2026-02-25T08:46:22.849 INFO:tasks.rados.rados.0.trial023.stdout:1077: done (1 left) 2026-02-25T08:46:22.849 INFO:tasks.rados.rados.0.trial023.stdout:1078: done (0 left) 2026-02-25T08:46:22.850 INFO:tasks.rados.rados.0.trial023.stdout:1079: delete oid 50 current snap is 136 2026-02-25T08:46:22.893 INFO:tasks.rados.rados.0.trial023.stdout:1079: done (0 left) 2026-02-25T08:46:22.893 INFO:tasks.rados.rados.0.trial023.stdout:1080: delete oid 28 current snap is 136 2026-02-25T08:46:22.924 INFO:tasks.rados.rados.0.trial023.stdout:1080: done (0 left) 2026-02-25T08:46:22.924 INFO:tasks.rados.rados.0.trial023.stdout:1081: write (excl) oid 42 current snap is 136 2026-02-25T08:46:22.924 INFO:tasks.rados.rados.0.trial023.stdout:1081: seq_num 510 ranges {571570=572641,1877346=694224} 2026-02-25T08:46:22.928 INFO:tasks.rados.rados.0.trial023.stdout:1081: writing trial02342524-42 from 571570 to 1144211 tid 1 2026-02-25T08:46:22.934 INFO:tasks.rados.rados.0.trial023.stdout:1081: writing trial02342524-42 from 1877346 to 2571570 tid 2 2026-02-25T08:46:22.934 INFO:tasks.rados.rados.0.trial023.stdout:1082: snap_remove snap 134 2026-02-25T08:46:23.851 INFO:tasks.rados.rados.0.trial023.stdout:1082: done (1 left) 2026-02-25T08:46:23.851 INFO:tasks.rados.rados.0.trial023.stdout:1083: write oid 48 current snap is 136 2026-02-25T08:46:23.851 INFO:tasks.rados.rados.0.trial023.stdout:1081: finishing write tid 1 to trial02342524-42 2026-02-25T08:46:23.851 INFO:tasks.rados.rados.0.trial023.stdout:1083: seq_num 511 ranges {588377=647890,1935828=635840} 2026-02-25T08:46:23.855 INFO:tasks.rados.rados.0.trial023.stdout:1083: writing trial02342524-48 from 588377 to 1236267 tid 1 2026-02-25T08:46:23.861 INFO:tasks.rados.rados.0.trial023.stdout:1083: writing trial02342524-48 from 1935828 to 2571668 tid 2 2026-02-25T08:46:23.861 INFO:tasks.rados.rados.0.trial023.stdout:1084: read oid 20 snap -1 2026-02-25T08:46:23.861 INFO:tasks.rados.rados.0.trial023.stdout:1084: expect (ObjNum 410 snap 113 seq_num 410) 2026-02-25T08:46:23.861 INFO:tasks.rados.rados.0.trial023.stdout:1085: write (excl) oid 24 current snap is 136 2026-02-25T08:46:23.861 INFO:tasks.rados.rados.0.trial023.stdout:1085: seq_num 512 ranges {605184=723139,1877957=727227} 2026-02-25T08:46:23.865 INFO:tasks.rados.rados.0.trial023.stdout:1085: writing trial02342524-24 from 605184 to 1328323 tid 1 2026-02-25T08:46:23.871 INFO:tasks.rados.rados.0.trial023.stdout:1085: writing trial02342524-24 from 1877957 to 2605184 tid 2 2026-02-25T08:46:23.871 INFO:tasks.rados.rados.0.trial023.stdout:1086: delete oid 1 current snap is 136 2026-02-25T08:46:23.871 INFO:tasks.rados.rados.0.trial023.stdout:1081: finishing write tid 2 to trial02342524-42 2026-02-25T08:46:23.871 INFO:tasks.rados.rados.0.trial023.stdout:1081: finishing write tid 3 to trial02342524-42 2026-02-25T08:46:23.871 INFO:tasks.rados.rados.0.trial023.stdout:1081: finishing write tid 4 to trial02342524-42 2026-02-25T08:46:23.871 INFO:tasks.rados.rados.0.trial023.stdout:1081: oid 42 updating version 0 to 285 2026-02-25T08:46:23.872 INFO:tasks.rados.rados.0.trial023.stdout:1081: oid 42 version 285 is already newer than 285 2026-02-25T08:46:23.872 INFO:tasks.rados.rados.0.trial023.stdout:1081: oid 42 version 285 is already newer than 283 2026-02-25T08:46:23.872 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 42 v 285 (ObjNum 510 snap 136 seq_num 510) dirty exists 2026-02-25T08:46:23.872 INFO:tasks.rados.rados.0.trial023.stdout:1081: left oid 42 (ObjNum 510 snap 136 seq_num 510) 2026-02-25T08:46:23.884 INFO:tasks.rados.rados.0.trial023.stdout:1081: done (4 left) 2026-02-25T08:46:23.884 INFO:tasks.rados.rados.0.trial023.stdout:1086: done (3 left) 2026-02-25T08:46:23.884 INFO:tasks.rados.rados.0.trial023.stdout:1087: write (excl) oid 34 current snap is 136 2026-02-25T08:46:23.884 INFO:tasks.rados.rados.0.trial023.stdout:1087: seq_num 514 ranges {638798=473637,1594921=783167} 2026-02-25T08:46:23.888 INFO:tasks.rados.rados.0.trial023.stdout:1087: writing trial02342524-34 from 638798 to 1112435 tid 1 2026-02-25T08:46:23.893 INFO:tasks.rados.rados.0.trial023.stdout:1087: writing trial02342524-34 from 1594921 to 2378088 tid 2 2026-02-25T08:46:23.894 INFO:tasks.rados.rados.0.trial023.stdout:1088: write (excl) oid 37 current snap is 136 2026-02-25T08:46:23.894 INFO:tasks.rados.rados.0.trial023.stdout:1088: seq_num 515 ranges {655605=548886,1653403=643178} 2026-02-25T08:46:23.898 INFO:tasks.rados.rados.0.trial023.stdout:1088: writing trial02342524-37 from 655605 to 1204491 tid 1 2026-02-25T08:46:23.902 INFO:tasks.rados.rados.0.trial023.stdout:1088: writing trial02342524-37 from 1653403 to 2296581 tid 2 2026-02-25T08:46:23.902 INFO:tasks.rados.rados.0.trial023.stdout:1089: write oid 46 current snap is 136 2026-02-25T08:46:23.902 INFO:tasks.rados.rados.0.trial023.stdout:1089: seq_num 516 ranges {672412=624135,1995532=676880} 2026-02-25T08:46:23.907 INFO:tasks.rados.rados.0.trial023.stdout:1089: writing trial02342524-46 from 672412 to 1296547 tid 1 2026-02-25T08:46:23.912 INFO:tasks.rados.rados.0.trial023.stdout:1089: writing trial02342524-46 from 1995532 to 2672412 tid 2 2026-02-25T08:46:23.912 INFO:tasks.rados.rados.0.trial023.stdout:1090: write (excl) oid 1 current snap is 136 2026-02-25T08:46:23.913 INFO:tasks.rados.rados.0.trial023.stdout:1090: seq_num 517 ranges {689219=415737,1770367=646847} 2026-02-25T08:46:23.916 INFO:tasks.rados.rados.0.trial023.stdout:1090: writing trial02342524-1 from 689219 to 1104956 tid 1 2026-02-25T08:46:23.922 INFO:tasks.rados.rados.0.trial023.stdout:1090: writing trial02342524-1 from 1770367 to 2417214 tid 2 2026-02-25T08:46:23.922 INFO:tasks.rados.rados.0.trial023.stdout:1091: snap_create 2026-02-25T08:46:23.922 INFO:tasks.rados.rados.0.trial023.stdout:1084: expect (ObjNum 410 snap 113 seq_num 410) 2026-02-25T08:46:23.922 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:46:24.029 INFO:tasks.rados.rados.0.trial023.stdout:1087: finishing write tid 1 to trial02342524-34 2026-02-25T08:46:24.029 INFO:tasks.rados.rados.0.trial023.stdout:1083: finishing write tid 1 to trial02342524-48 2026-02-25T08:46:24.029 INFO:tasks.rados.rados.0.trial023.stdout:1088: finishing write tid 1 to trial02342524-37 2026-02-25T08:46:24.030 INFO:tasks.rados.rados.0.trial023.stdout:1089: finishing write tid 1 to trial02342524-46 2026-02-25T08:46:24.030 INFO:tasks.rados.rados.0.trial023.stdout:1090: finishing write tid 1 to trial02342524-1 2026-02-25T08:46:24.030 INFO:tasks.rados.rados.0.trial023.stdout:1088: finishing write tid 2 to trial02342524-37 2026-02-25T08:46:24.030 INFO:tasks.rados.rados.0.trial023.stdout:1083: finishing write tid 2 to trial02342524-48 2026-02-25T08:46:24.030 INFO:tasks.rados.rados.0.trial023.stdout:1088: finishing write tid 3 to trial02342524-37 2026-02-25T08:46:24.030 INFO:tasks.rados.rados.0.trial023.stdout:1089: finishing write tid 2 to trial02342524-46 2026-02-25T08:46:24.030 INFO:tasks.rados.rados.0.trial023.stdout:1088: finishing write tid 4 to trial02342524-37 2026-02-25T08:46:24.030 INFO:tasks.rados.rados.0.trial023.stdout:1088: oid 37 updating version 0 to 216 2026-02-25T08:46:24.030 INFO:tasks.rados.rados.0.trial023.stdout:1088: oid 37 version 216 is already newer than 214 2026-02-25T08:46:24.030 INFO:tasks.rados.rados.0.trial023.stdout:1088: oid 37 version 216 is already newer than 216 2026-02-25T08:46:24.030 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 37 v 216 (ObjNum 515 snap 136 seq_num 515) dirty exists 2026-02-25T08:46:24.030 INFO:tasks.rados.rados.0.trial023.stdout:1088: left oid 37 (ObjNum 515 snap 136 seq_num 515) 2026-02-25T08:46:24.031 INFO:tasks.rados.rados.0.trial023.stdout:1083: finishing write tid 3 to trial02342524-48 2026-02-25T08:46:24.031 INFO:tasks.rados.rados.0.trial023.stdout:1083: finishing write tid 4 to trial02342524-48 2026-02-25T08:46:24.031 INFO:tasks.rados.rados.0.trial023.stdout:1083: oid 48 updating version 0 to 218 2026-02-25T08:46:24.031 INFO:tasks.rados.rados.0.trial023.stdout:1083: oid 48 version 218 is already newer than 218 2026-02-25T08:46:24.031 INFO:tasks.rados.rados.0.trial023.stdout:1083: oid 48 version 218 is already newer than 218 2026-02-25T08:46:24.031 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 48 v 218 (ObjNum 511 snap 136 seq_num 511) dirty exists 2026-02-25T08:46:24.031 INFO:tasks.rados.rados.0.trial023.stdout:1083: left oid 48 (ObjNum 511 snap 136 seq_num 511) 2026-02-25T08:46:24.031 INFO:tasks.rados.rados.0.trial023.stdout:1089: finishing write tid 3 to trial02342524-46 2026-02-25T08:46:24.031 INFO:tasks.rados.rados.0.trial023.stdout:1089: finishing write tid 4 to trial02342524-46 2026-02-25T08:46:24.031 INFO:tasks.rados.rados.0.trial023.stdout:1089: oid 46 updating version 0 to 290 2026-02-25T08:46:24.031 INFO:tasks.rados.rados.0.trial023.stdout:1089: oid 46 version 290 is already newer than 290 2026-02-25T08:46:24.031 INFO:tasks.rados.rados.0.trial023.stdout:1089: oid 46 version 290 is already newer than 290 2026-02-25T08:46:24.032 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 46 v 290 (ObjNum 516 snap 136 seq_num 516) dirty exists 2026-02-25T08:46:24.032 INFO:tasks.rados.rados.0.trial023.stdout:1089: left oid 46 (ObjNum 516 snap 136 seq_num 516) 2026-02-25T08:46:24.032 INFO:tasks.rados.rados.0.trial023.stdout:1090: finishing write tid 2 to trial02342524-1 2026-02-25T08:46:24.032 INFO:tasks.rados.rados.0.trial023.stdout:1087: finishing write tid 2 to trial02342524-34 2026-02-25T08:46:24.032 INFO:tasks.rados.rados.0.trial023.stdout:1090: finishing write tid 3 to trial02342524-1 2026-02-25T08:46:24.032 INFO:tasks.rados.rados.0.trial023.stdout:1087: finishing write tid 3 to trial02342524-34 2026-02-25T08:46:24.032 INFO:tasks.rados.rados.0.trial023.stdout:1090: finishing write tid 4 to trial02342524-1 2026-02-25T08:46:24.032 INFO:tasks.rados.rados.0.trial023.stdout:1090: oid 1 updating version 0 to 187 2026-02-25T08:46:24.032 INFO:tasks.rados.rados.0.trial023.stdout:1090: oid 1 version 187 is already newer than 187 2026-02-25T08:46:24.032 INFO:tasks.rados.rados.0.trial023.stdout:1090: oid 1 version 187 is already newer than 185 2026-02-25T08:46:24.032 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 1 v 187 (ObjNum 517 snap 136 seq_num 517) dirty exists 2026-02-25T08:46:24.032 INFO:tasks.rados.rados.0.trial023.stdout:1090: left oid 1 (ObjNum 517 snap 136 seq_num 517) 2026-02-25T08:46:24.032 INFO:tasks.rados.rados.0.trial023.stdout:1087: finishing write tid 4 to trial02342524-34 2026-02-25T08:46:24.032 INFO:tasks.rados.rados.0.trial023.stdout:1087: oid 34 updating version 0 to 64 2026-02-25T08:46:24.033 INFO:tasks.rados.rados.0.trial023.stdout:1087: oid 34 updating version 64 to 66 2026-02-25T08:46:24.033 INFO:tasks.rados.rados.0.trial023.stdout:1087: oid 34 version 66 is already newer than 66 2026-02-25T08:46:24.033 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 34 v 66 (ObjNum 514 snap 136 seq_num 514) dirty exists 2026-02-25T08:46:24.033 INFO:tasks.rados.rados.0.trial023.stdout:1087: left oid 34 (ObjNum 514 snap 136 seq_num 514) 2026-02-25T08:46:24.033 INFO:tasks.rados.rados.0.trial023.stdout:1085: finishing write tid 1 to trial02342524-24 2026-02-25T08:46:24.033 INFO:tasks.rados.rados.0.trial023.stdout:1085: finishing write tid 2 to trial02342524-24 2026-02-25T08:46:24.033 INFO:tasks.rados.rados.0.trial023.stdout:1085: finishing write tid 3 to trial02342524-24 2026-02-25T08:46:24.033 INFO:tasks.rados.rados.0.trial023.stdout:1085: finishing write tid 4 to trial02342524-24 2026-02-25T08:46:24.033 INFO:tasks.rados.rados.0.trial023.stdout:1085: oid 24 updating version 0 to 178 2026-02-25T08:46:24.033 INFO:tasks.rados.rados.0.trial023.stdout:1085: oid 24 version 178 is already newer than 178 2026-02-25T08:46:24.033 INFO:tasks.rados.rados.0.trial023.stdout:1085: oid 24 version 178 is already newer than 176 2026-02-25T08:46:24.033 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 24 v 178 (ObjNum 512 snap 136 seq_num 512) dirty exists 2026-02-25T08:46:24.034 INFO:tasks.rados.rados.0.trial023.stdout:1085: left oid 24 (ObjNum 512 snap 136 seq_num 512) 2026-02-25T08:46:24.312 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:46:24.391 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:24.388+0000 7fa64c646640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:24.393 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:24.392+0000 7fa64c646640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:24.409 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:46:24.442 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:24.442 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:24.854 INFO:tasks.rados.rados.0.trial023.stdout:1083: done (7 left) 2026-02-25T08:46:24.854 INFO:tasks.rados.rados.0.trial023.stdout:1084: done (6 left) 2026-02-25T08:46:24.854 INFO:tasks.rados.rados.0.trial023.stdout:1085: done (5 left) 2026-02-25T08:46:24.855 INFO:tasks.rados.rados.0.trial023.stdout:1087: done (4 left) 2026-02-25T08:46:24.855 INFO:tasks.rados.rados.0.trial023.stdout:1088: done (3 left) 2026-02-25T08:46:24.855 INFO:tasks.rados.rados.0.trial023.stdout:1089: done (2 left) 2026-02-25T08:46:24.855 INFO:tasks.rados.rados.0.trial023.stdout:1090: done (1 left) 2026-02-25T08:46:24.855 INFO:tasks.rados.rados.0.trial023.stdout:1091: done (0 left) 2026-02-25T08:46:24.855 INFO:tasks.rados.rados.0.trial023.stdout:1092: write oid 43 current snap is 137 2026-02-25T08:46:24.855 INFO:tasks.rados.rados.0.trial023.stdout:1092: seq_num 518 ranges {706026=490986,1828849=790505} 2026-02-25T08:46:24.858 INFO:tasks.rados.rados.0.trial023.stdout:1092: writing trial02342524-43 from 706026 to 1197012 tid 1 2026-02-25T08:46:24.864 INFO:tasks.rados.rados.0.trial023.stdout:1092: writing trial02342524-43 from 1828849 to 2619354 tid 2 2026-02-25T08:46:24.864 INFO:tasks.rados.rados.0.trial023.stdout:1093: snap_create 2026-02-25T08:46:24.906 INFO:tasks.rados.rados.0.trial023.stdout:1092: finishing write tid 1 to trial02342524-43 2026-02-25T08:46:24.914 INFO:tasks.rados.rados.0.trial023.stdout:1092: finishing write tid 2 to trial02342524-43 2026-02-25T08:46:24.919 INFO:tasks.rados.rados.0.trial023.stdout:1092: finishing write tid 3 to trial02342524-43 2026-02-25T08:46:24.921 INFO:tasks.rados.rados.0.trial023.stdout:1092: finishing write tid 4 to trial02342524-43 2026-02-25T08:46:24.921 INFO:tasks.rados.rados.0.trial023.stdout:1092: oid 43 updating version 0 to 39 2026-02-25T08:46:24.921 INFO:tasks.rados.rados.0.trial023.stdout:1092: oid 43 version 39 is already newer than 39 2026-02-25T08:46:24.921 INFO:tasks.rados.rados.0.trial023.stdout:1092: oid 43 version 39 is already newer than 39 2026-02-25T08:46:24.921 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 43 v 39 (ObjNum 518 snap 137 seq_num 518) dirty exists 2026-02-25T08:46:24.921 INFO:tasks.rados.rados.0.trial023.stdout:1092: left oid 43 (ObjNum 518 snap 137 seq_num 518) 2026-02-25T08:46:25.857 INFO:tasks.rados.rados.0.trial023.stdout:1092: done (1 left) 2026-02-25T08:46:25.857 INFO:tasks.rados.rados.0.trial023.stdout:1093: done (0 left) 2026-02-25T08:46:25.857 INFO:tasks.rados.rados.0.trial023.stdout:1094: read oid 22 snap -1 2026-02-25T08:46:25.857 INFO:tasks.rados.rados.0.trial023.stdout:1094: expect deleted 2026-02-25T08:46:25.857 INFO:tasks.rados.rados.0.trial023.stdout:1095: read oid 12 snap -1 2026-02-25T08:46:25.858 INFO:tasks.rados.rados.0.trial023.stdout:1095: expect (ObjNum 504 snap 133 seq_num 504) 2026-02-25T08:46:25.858 INFO:tasks.rados.rados.0.trial023.stdout:1096: snap_create 2026-02-25T08:46:25.901 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:46:25.904 INFO:tasks.rados.rados.0.trial023.stdout:1095: expect (ObjNum 504 snap 133 seq_num 504) 2026-02-25T08:46:25.959 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:46:25.971 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:46:25.974 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:46:25.974 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:46:26.444 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:46:26.519 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:26.517+0000 7faf768cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:26.522 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:26.521+0000 7faf768cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:26.536 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:46:26.564 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:26.565 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:26.613 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~286s 2026-02-25T08:46:26.859 INFO:tasks.rados.rados.0.trial023.stdout:1094: done (2 left) 2026-02-25T08:46:26.860 INFO:tasks.rados.rados.0.trial023.stdout:1095: done (1 left) 2026-02-25T08:46:26.860 INFO:tasks.rados.rados.0.trial023.stdout:1096: done (0 left) 2026-02-25T08:46:26.860 INFO:tasks.rados.rados.0.trial023.stdout:1097: delete oid 47 current snap is 139 2026-02-25T08:46:26.904 INFO:tasks.rados.rados.0.trial023.stdout:1097: done (0 left) 2026-02-25T08:46:26.904 INFO:tasks.rados.rados.0.trial023.stdout:1098: write oid 27 current snap is 139 2026-02-25T08:46:26.904 INFO:tasks.rados.rados.0.trial023.stdout:1098: seq_num 520 ranges {739640=641484,1829460=794174} 2026-02-25T08:46:26.909 INFO:tasks.rados.rados.0.trial023.stdout:1098: writing trial02342524-27 from 739640 to 1381124 tid 1 2026-02-25T08:46:26.914 INFO:tasks.rados.rados.0.trial023.stdout:1098: writing trial02342524-27 from 1829460 to 2623634 tid 2 2026-02-25T08:46:26.914 INFO:tasks.rados.rados.0.trial023.stdout:1099: delete oid 38 current snap is 139 2026-02-25T08:46:26.917 INFO:tasks.rados.rados.0.trial023.stdout:1099: done (1 left) 2026-02-25T08:46:26.917 INFO:tasks.rados.rados.0.trial023.stdout:1100: snap_remove snap 135 2026-02-25T08:46:27.864 INFO:tasks.rados.rados.0.trial023.stdout:1100: done (1 left) 2026-02-25T08:46:27.865 INFO:tasks.rados.rados.0.trial023.stdout:1101: write (excl) oid 42 current snap is 139 2026-02-25T08:46:27.865 INFO:tasks.rados.rados.0.trial023.stdout:1098: finishing write tid 1 to trial02342524-27 2026-02-25T08:46:27.865 INFO:tasks.rados.rados.0.trial023.stdout:1101: seq_num 522 ranges {773254=791982,2346424=426830} 2026-02-25T08:46:27.870 INFO:tasks.rados.rados.0.trial023.stdout:1101: writing trial02342524-42 from 773254 to 1565236 tid 1 2026-02-25T08:46:27.875 INFO:tasks.rados.rados.0.trial023.stdout:1101: writing trial02342524-42 from 2346424 to 2773254 tid 2 2026-02-25T08:46:27.875 INFO:tasks.rados.rados.0.trial023.stdout:1102: snap_remove snap 138 2026-02-25T08:46:28.565 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:46:28.640 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:28.637+0000 7f64e9783640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:28.643 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:28.641+0000 7f64e9783640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:28.658 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:46:28.685 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:28.685 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:28.869 INFO:tasks.rados.rados.0.trial023.stdout:1102: done (2 left) 2026-02-25T08:46:28.869 INFO:tasks.rados.rados.0.trial023.stdout:1103: write (excl) oid 44 current snap is 139 2026-02-25T08:46:28.869 INFO:tasks.rados.rados.0.trial023.stdout:1098: finishing write tid 2 to trial02342524-27 2026-02-25T08:46:28.869 INFO:tasks.rados.rados.0.trial023.stdout:1103: seq_num 523 ranges {790061=467231,2004906=541501} 2026-02-25T08:46:28.873 INFO:tasks.rados.rados.0.trial023.stdout:1103: writing trial02342524-44 from 790061 to 1257292 tid 1 2026-02-25T08:46:28.879 INFO:tasks.rados.rados.0.trial023.stdout:1103: writing trial02342524-44 from 2004906 to 2546407 tid 2 2026-02-25T08:46:28.879 INFO:tasks.rados.rados.0.trial023.stdout:1104: snap_create 2026-02-25T08:46:28.879 INFO:tasks.rados.rados.0.trial023.stdout:1098: finishing write tid 3 to trial02342524-27 2026-02-25T08:46:28.879 INFO:tasks.rados.rados.0.trial023.stdout:1098: finishing write tid 4 to trial02342524-27 2026-02-25T08:46:28.879 INFO:tasks.rados.rados.0.trial023.stdout:1098: oid 27 updating version 0 to 69 2026-02-25T08:46:28.879 INFO:tasks.rados.rados.0.trial023.stdout:1098: oid 27 version 69 is already newer than 69 2026-02-25T08:46:28.879 INFO:tasks.rados.rados.0.trial023.stdout:1098: oid 27 version 69 is already newer than 69 2026-02-25T08:46:28.880 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 27 v 69 (ObjNum 520 snap 139 seq_num 520) dirty exists 2026-02-25T08:46:28.880 INFO:tasks.rados.rados.0.trial023.stdout:1098: left oid 27 (ObjNum 520 snap 139 seq_num 520) 2026-02-25T08:46:28.880 INFO:tasks.rados.rados.0.trial023.stdout:1101: finishing write tid 1 to trial02342524-42 2026-02-25T08:46:28.880 INFO:tasks.rados.rados.0.trial023.stdout:1101: finishing write tid 2 to trial02342524-42 2026-02-25T08:46:28.880 INFO:tasks.rados.rados.0.trial023.stdout:1101: finishing write tid 3 to trial02342524-42 2026-02-25T08:46:28.880 INFO:tasks.rados.rados.0.trial023.stdout:1101: finishing write tid 4 to trial02342524-42 2026-02-25T08:46:28.880 INFO:tasks.rados.rados.0.trial023.stdout:1101: oid 42 updating version 0 to 294 2026-02-25T08:46:28.880 INFO:tasks.rados.rados.0.trial023.stdout:1101: oid 42 version 294 is already newer than 294 2026-02-25T08:46:28.880 INFO:tasks.rados.rados.0.trial023.stdout:1101: oid 42 version 294 is already newer than 292 2026-02-25T08:46:28.880 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 42 v 294 (ObjNum 522 snap 139 seq_num 522) dirty exists 2026-02-25T08:46:28.880 INFO:tasks.rados.rados.0.trial023.stdout:1101: left oid 42 (ObjNum 522 snap 139 seq_num 522) 2026-02-25T08:46:28.919 INFO:tasks.rados.rados.0.trial023.stdout:1103: finishing write tid 1 to trial02342524-44 2026-02-25T08:46:28.932 INFO:tasks.rados.rados.0.trial023.stdout:1103: finishing write tid 2 to trial02342524-44 2026-02-25T08:46:28.936 INFO:tasks.rados.rados.0.trial023.stdout:1103: finishing write tid 3 to trial02342524-44 2026-02-25T08:46:28.938 INFO:tasks.rados.rados.0.trial023.stdout:1103: finishing write tid 4 to trial02342524-44 2026-02-25T08:46:28.939 INFO:tasks.rados.rados.0.trial023.stdout:1103: oid 44 updating version 0 to 35 2026-02-25T08:46:28.939 INFO:tasks.rados.rados.0.trial023.stdout:1103: oid 44 updating version 35 to 37 2026-02-25T08:46:28.939 INFO:tasks.rados.rados.0.trial023.stdout:1103: oid 44 version 37 is already newer than 37 2026-02-25T08:46:28.939 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 44 v 37 (ObjNum 523 snap 139 seq_num 523) dirty exists 2026-02-25T08:46:28.939 INFO:tasks.rados.rados.0.trial023.stdout:1103: left oid 44 (ObjNum 523 snap 139 seq_num 523) 2026-02-25T08:46:29.844 INFO:tasks.rados.rados.0.trial023.stdout:1098: done (3 left) 2026-02-25T08:46:29.845 INFO:tasks.rados.rados.0.trial023.stdout:1101: done (2 left) 2026-02-25T08:46:29.845 INFO:tasks.rados.rados.0.trial023.stdout:1103: done (1 left) 2026-02-25T08:46:29.845 INFO:tasks.rados.rados.0.trial023.stdout:1104: done (0 left) 2026-02-25T08:46:29.845 INFO:tasks.rados.rados.0.trial023.stdout:1105: read oid 26 snap 126 2026-02-25T08:46:29.845 INFO:tasks.rados.rados.0.trial023.stdout:1105: expect (ObjNum 475 snap 126 seq_num 475) 2026-02-25T08:46:29.845 INFO:tasks.rados.rados.0.trial023.stdout:1106: delete oid 32 current snap is 140 2026-02-25T08:46:29.901 INFO:tasks.rados.rados.0.trial023.stdout:1105: expect (ObjNum 475 snap 126 seq_num 475) 2026-02-25T08:46:30.091 INFO:tasks.rados.rados.0.trial023.stdout:1105: done (1 left) 2026-02-25T08:46:30.092 INFO:tasks.rados.rados.0.trial023.stdout:1106: done (0 left) 2026-02-25T08:46:30.092 INFO:tasks.rados.rados.0.trial023.stdout:1107: snap_create 2026-02-25T08:46:30.685 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:46:30.763 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:30.760+0000 7f948759c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:30.765 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:30.763+0000 7f948759c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:30.781 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:46:30.808 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:30.808 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:30.871 INFO:tasks.rados.rados.0.trial023.stdout:1107: done (0 left) 2026-02-25T08:46:30.871 INFO:tasks.rados.rados.0.trial023.stdout:1108: snap_remove snap 139 2026-02-25T08:46:30.975 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:46:31.047 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:46:31.050 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:46:31.050 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:46:31.877 INFO:tasks.rados.rados.0.trial023.stdout:1108: done (0 left) 2026-02-25T08:46:31.877 INFO:tasks.rados.rados.0.trial023.stdout:1109: read oid 36 snap -1 2026-02-25T08:46:31.877 INFO:tasks.rados.rados.0.trial023.stdout:1109: expect (ObjNum 428 snap 116 seq_num 428) 2026-02-25T08:46:31.877 INFO:tasks.rados.rados.0.trial023.stdout:1110: snap_remove snap 137 2026-02-25T08:46:32.117 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~292s 2026-02-25T08:46:32.810 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:46:32.878 INFO:tasks.rados.rados.0.trial023.stdout:1110: done (1 left) 2026-02-25T08:46:32.878 INFO:tasks.rados.rados.0.trial023.stdout:1111: write oid 37 current snap is 141 2026-02-25T08:46:32.878 INFO:tasks.rados.rados.0.trial023.stdout:1111: seq_num 525 ranges {423675=734082,1721870=545170,2717777=105898} 2026-02-25T08:46:32.882 INFO:tasks.rados.rados.0.trial023.stdout:1111: writing trial02342524-37 from 423675 to 1157757 tid 1 2026-02-25T08:46:32.887 INFO:tasks.rados.rados.0.trial023.stdout:1111: writing trial02342524-37 from 1721870 to 2267040 tid 2 2026-02-25T08:46:32.888 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:32.885+0000 7f6b4aa06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:32.889 INFO:tasks.rados.rados.0.trial023.stdout:1111: writing trial02342524-37 from 2717777 to 2823675 tid 3 2026-02-25T08:46:32.889 INFO:tasks.rados.rados.0.trial023.stdout:1112: read oid 17 snap -1 2026-02-25T08:46:32.889 INFO:tasks.rados.rados.0.trial023.stdout:1112: expect (ObjNum 493 snap 129 seq_num 493) 2026-02-25T08:46:32.890 INFO:tasks.rados.rados.0.trial023.stdout:1113: snap_create 2026-02-25T08:46:32.890 INFO:tasks.rados.rados.0.trial023.stdout:1109: expect (ObjNum 428 snap 116 seq_num 428) 2026-02-25T08:46:32.890 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:32.888+0000 7f6b4aa06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:32.907 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:46:32.943 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:32.944 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:33.033 INFO:tasks.rados.rados.0.trial023.stdout:1112: expect (ObjNum 493 snap 129 seq_num 493) 2026-02-25T08:46:33.033 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 584944 2026-02-25T08:46:33.073 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1145148 2026-02-25T08:46:33.087 INFO:tasks.rados.rados.0.trial023.stdout:1111: finishing write tid 1 to trial02342524-37 2026-02-25T08:46:33.088 INFO:tasks.rados.rados.0.trial023.stdout:1111: finishing write tid 2 to trial02342524-37 2026-02-25T08:46:33.088 INFO:tasks.rados.rados.0.trial023.stdout:1111: finishing write tid 3 to trial02342524-37 2026-02-25T08:46:33.088 INFO:tasks.rados.rados.0.trial023.stdout:1111: finishing write tid 4 to trial02342524-37 2026-02-25T08:46:33.088 INFO:tasks.rados.rados.0.trial023.stdout:1111: finishing write tid 5 to trial02342524-37 2026-02-25T08:46:33.088 INFO:tasks.rados.rados.0.trial023.stdout:1111: oid 37 updating version 0 to 221 2026-02-25T08:46:33.088 INFO:tasks.rados.rados.0.trial023.stdout:1111: oid 37 version 221 is already newer than 221 2026-02-25T08:46:33.088 INFO:tasks.rados.rados.0.trial023.stdout:1111: oid 37 version 221 is already newer than 221 2026-02-25T08:46:33.088 INFO:tasks.rados.rados.0.trial023.stdout:1111: oid 37 version 221 is already newer than 221 2026-02-25T08:46:33.088 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 37 v 221 (ObjNum 525 snap 141 seq_num 525) dirty exists 2026-02-25T08:46:33.088 INFO:tasks.rados.rados.0.trial023.stdout:1111: left oid 37 (ObjNum 525 snap 141 seq_num 525) 2026-02-25T08:46:33.881 INFO:tasks.rados.rados.0.trial023.stdout:1109: done (3 left) 2026-02-25T08:46:33.881 INFO:tasks.rados.rados.0.trial023.stdout:1111: done (2 left) 2026-02-25T08:46:33.881 INFO:tasks.rados.rados.0.trial023.stdout:1112: done (1 left) 2026-02-25T08:46:33.882 INFO:tasks.rados.rados.0.trial023.stdout:1113: done (0 left) 2026-02-25T08:46:33.882 INFO:tasks.rados.rados.0.trial023.stdout:1114: read oid 43 snap 136 2026-02-25T08:46:33.882 INFO:tasks.rados.rados.0.trial023.stdout:1114: expect (ObjNum 507 snap 135 seq_num 507) 2026-02-25T08:46:33.882 INFO:tasks.rados.rados.0.trial023.stdout:1115: read oid 5 snap -1 2026-02-25T08:46:33.882 INFO:tasks.rados.rados.0.trial023.stdout:1115: expect (ObjNum 474 snap 125 seq_num 474) 2026-02-25T08:46:33.882 INFO:tasks.rados.rados.0.trial023.stdout:1116: write (excl) oid 32 current snap is 142 2026-02-25T08:46:33.882 INFO:tasks.rados.rados.0.trial023.stdout:1116: seq_num 526 ranges {440482=409331,1380352=405181,2461553=378929} 2026-02-25T08:46:33.884 INFO:tasks.rados.rados.0.trial023.stdout:1116: writing trial02342524-32 from 440482 to 849813 tid 1 2026-02-25T08:46:33.888 INFO:tasks.rados.rados.0.trial023.stdout:1116: writing trial02342524-32 from 1380352 to 1785533 tid 2 2026-02-25T08:46:33.892 INFO:tasks.rados.rados.0.trial023.stdout:1116: writing trial02342524-32 from 2461553 to 2840482 tid 3 2026-02-25T08:46:33.892 INFO:tasks.rados.rados.0.trial023.stdout:1117: read oid 1 snap 136 2026-02-25T08:46:33.892 INFO:tasks.rados.rados.0.trial023.stdout:1117: expect (ObjNum 517 snap 136 seq_num 517) 2026-02-25T08:46:33.892 INFO:tasks.rados.rados.0.trial023.stdout:1118: snap_create 2026-02-25T08:46:33.911 INFO:tasks.rados.rados.0.trial023.stdout:1115: expect (ObjNum 474 snap 125 seq_num 474) 2026-02-25T08:46:33.922 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 3642094 2026-02-25T08:46:33.927 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:46:33.961 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:46:33.995 INFO:tasks.rados.rados.0.trial023.stdout:1114: expect (ObjNum 507 snap 135 seq_num 507) 2026-02-25T08:46:33.995 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:46:34.088 INFO:tasks.rados.rados.0.trial023.stdout:1116: finishing write tid 1 to trial02342524-32 2026-02-25T08:46:34.088 INFO:tasks.rados.rados.0.trial023.stdout:1117: expect (ObjNum 517 snap 136 seq_num 517) 2026-02-25T08:46:34.108 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 1570577 2026-02-25T08:46:34.121 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:46:34.144 INFO:tasks.rados.rados.0.trial023.stdout:1116: finishing write tid 2 to trial02342524-32 2026-02-25T08:46:34.144 INFO:tasks.rados.rados.0.trial023.stdout:1116: finishing write tid 3 to trial02342524-32 2026-02-25T08:46:34.145 INFO:tasks.rados.rados.0.trial023.stdout:1116: finishing write tid 4 to trial02342524-32 2026-02-25T08:46:34.145 INFO:tasks.rados.rados.0.trial023.stdout:1116: finishing write tid 5 to trial02342524-32 2026-02-25T08:46:34.145 INFO:tasks.rados.rados.0.trial023.stdout:1116: oid 32 updating version 0 to 184 2026-02-25T08:46:34.145 INFO:tasks.rados.rados.0.trial023.stdout:1116: oid 32 version 184 is already newer than 181 2026-02-25T08:46:34.145 INFO:tasks.rados.rados.0.trial023.stdout:1116: oid 32 version 184 is already newer than 184 2026-02-25T08:46:34.145 INFO:tasks.rados.rados.0.trial023.stdout:1116: oid 32 version 184 is already newer than 184 2026-02-25T08:46:34.145 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 32 v 184 (ObjNum 526 snap 142 seq_num 526) dirty exists 2026-02-25T08:46:34.145 INFO:tasks.rados.rados.0.trial023.stdout:1116: left oid 32 (ObjNum 526 snap 142 seq_num 526) 2026-02-25T08:46:34.886 INFO:tasks.rados.rados.0.trial023.stdout:1114: done (4 left) 2026-02-25T08:46:34.886 INFO:tasks.rados.rados.0.trial023.stdout:1115: done (3 left) 2026-02-25T08:46:34.887 INFO:tasks.rados.rados.0.trial023.stdout:1116: done (2 left) 2026-02-25T08:46:34.887 INFO:tasks.rados.rados.0.trial023.stdout:1117: done (1 left) 2026-02-25T08:46:34.887 INFO:tasks.rados.rados.0.trial023.stdout:1118: done (0 left) 2026-02-25T08:46:34.887 INFO:tasks.rados.rados.0.trial023.stdout:1119: read oid 33 snap 130 2026-02-25T08:46:34.887 INFO:tasks.rados.rados.0.trial023.stdout:1119: expect (ObjNum 445 snap 119 seq_num 445) 2026-02-25T08:46:34.887 INFO:tasks.rados.rados.0.trial023.stdout:1120: write (excl) oid 12 current snap is 143 2026-02-25T08:46:34.887 INFO:tasks.rados.rados.0.trial023.stdout:1120: seq_num 527 ranges {457289=484580,1438834=548839,2772623=84666} 2026-02-25T08:46:34.889 INFO:tasks.rados.rados.0.trial023.stdout:1120: writing trial02342524-12 from 457289 to 941869 tid 1 2026-02-25T08:46:34.893 INFO:tasks.rados.rados.0.trial023.stdout:1120: writing trial02342524-12 from 1438834 to 1987673 tid 2 2026-02-25T08:46:34.897 INFO:tasks.rados.rados.0.trial023.stdout:1120: writing trial02342524-12 from 2772623 to 2857289 tid 3 2026-02-25T08:46:34.897 INFO:tasks.rados.rados.0.trial023.stdout:1121: write (excl) oid 46 current snap is 143 2026-02-25T08:46:34.897 INFO:tasks.rados.rados.0.trial023.stdout:1121: seq_num 528 ranges {474096=559829,1780963=408850,2800046=74050} 2026-02-25T08:46:34.901 INFO:tasks.rados.rados.0.trial023.stdout:1121: writing trial02342524-46 from 474096 to 1033925 tid 1 2026-02-25T08:46:34.905 INFO:tasks.rados.rados.0.trial023.stdout:1121: writing trial02342524-46 from 1780963 to 2189813 tid 2 2026-02-25T08:46:34.908 INFO:tasks.rados.rados.0.trial023.stdout:1121: writing trial02342524-46 from 2800046 to 2874096 tid 3 2026-02-25T08:46:34.908 INFO:tasks.rados.rados.0.trial023.stdout:1122: write (excl) oid 19 current snap is 143 2026-02-25T08:46:34.908 INFO:tasks.rados.rados.0.trial023.stdout:1122: seq_num 529 ranges {490903=635078,1839445=552508} 2026-02-25T08:46:34.912 INFO:tasks.rados.rados.0.trial023.stdout:1122: writing trial02342524-19 from 490903 to 1125981 tid 1 2026-02-25T08:46:34.917 INFO:tasks.rados.rados.0.trial023.stdout:1122: writing trial02342524-19 from 1839445 to 2391953 tid 2 2026-02-25T08:46:34.917 INFO:tasks.rados.rados.0.trial023.stdout:1123: write oid 5 current snap is 143 2026-02-25T08:46:34.917 INFO:tasks.rados.rados.0.trial023.stdout:1123: seq_num 530 ranges {507710=710327,1897927=412519,2854892=52818} 2026-02-25T08:46:34.921 INFO:tasks.rados.rados.0.trial023.stdout:1123: writing trial02342524-5 from 507710 to 1218037 tid 1 2026-02-25T08:46:34.925 INFO:tasks.rados.rados.0.trial023.stdout:1123: writing trial02342524-5 from 1897927 to 2310446 tid 2 2026-02-25T08:46:34.928 INFO:tasks.rados.rados.0.trial023.stdout:1123: writing trial02342524-5 from 2854892 to 2907710 tid 3 2026-02-25T08:46:34.928 INFO:tasks.rados.rados.0.trial023.stdout:1124: write oid 28 current snap is 143 2026-02-25T08:46:34.928 INFO:tasks.rados.rados.0.trial023.stdout:1124: seq_num 531 ranges {524517=785576,1956409=556177} 2026-02-25T08:46:34.933 INFO:tasks.rados.rados.0.trial023.stdout:1124: writing trial02342524-28 from 524517 to 1310093 tid 1 2026-02-25T08:46:34.937 INFO:tasks.rados.rados.0.trial023.stdout:1124: writing trial02342524-28 from 1956409 to 2512586 tid 2 2026-02-25T08:46:34.937 INFO:tasks.rados.rados.0.trial023.stdout:1125: delete oid 50 current snap is 143 2026-02-25T08:46:34.941 INFO:tasks.rados.rados.0.trial023.stdout:1121: finishing write tid 1 to trial02342524-46 2026-02-25T08:46:34.941 INFO:tasks.rados.rados.0.trial023.stdout:1125: done (6 left) 2026-02-25T08:46:34.941 INFO:tasks.rados.rados.0.trial023.stdout:1126: snap_create 2026-02-25T08:46:34.942 INFO:tasks.rados.rados.0.trial023.stdout:1119: expect (ObjNum 445 snap 119 seq_num 445) 2026-02-25T08:46:34.942 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:46:34.944 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:46:35.022 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:35.018+0000 7f8963b5b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:35.024 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:35.022+0000 7f8963b5b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:35.037 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 2396507 2026-02-25T08:46:35.060 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:46:35.087 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:35.087 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:35.115 INFO:tasks.rados.rados.0.trial023.stdout:1120: finishing write tid 1 to trial02342524-12 2026-02-25T08:46:35.115 INFO:tasks.rados.rados.0.trial023.stdout:1121: finishing write tid 2 to trial02342524-46 2026-02-25T08:46:35.116 INFO:tasks.rados.rados.0.trial023.stdout:1124: finishing write tid 1 to trial02342524-28 2026-02-25T08:46:35.116 INFO:tasks.rados.rados.0.trial023.stdout:1124: finishing write tid 2 to trial02342524-28 2026-02-25T08:46:35.116 INFO:tasks.rados.rados.0.trial023.stdout:1121: finishing write tid 3 to trial02342524-46 2026-02-25T08:46:35.116 INFO:tasks.rados.rados.0.trial023.stdout:1120: finishing write tid 2 to trial02342524-12 2026-02-25T08:46:35.116 INFO:tasks.rados.rados.0.trial023.stdout:1124: finishing write tid 3 to trial02342524-28 2026-02-25T08:46:35.116 INFO:tasks.rados.rados.0.trial023.stdout:1124: finishing write tid 4 to trial02342524-28 2026-02-25T08:46:35.116 INFO:tasks.rados.rados.0.trial023.stdout:1124: oid 28 updating version 0 to 225 2026-02-25T08:46:35.116 INFO:tasks.rados.rados.0.trial023.stdout:1124: oid 28 version 225 is already newer than 225 2026-02-25T08:46:35.116 INFO:tasks.rados.rados.0.trial023.stdout:1124: oid 28 version 225 is already newer than 225 2026-02-25T08:46:35.116 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 28 v 225 (ObjNum 531 snap 143 seq_num 531) dirty exists 2026-02-25T08:46:35.116 INFO:tasks.rados.rados.0.trial023.stdout:1124: left oid 28 (ObjNum 531 snap 143 seq_num 531) 2026-02-25T08:46:35.116 INFO:tasks.rados.rados.0.trial023.stdout:1121: finishing write tid 4 to trial02342524-46 2026-02-25T08:46:35.117 INFO:tasks.rados.rados.0.trial023.stdout:1121: finishing write tid 5 to trial02342524-46 2026-02-25T08:46:35.117 INFO:tasks.rados.rados.0.trial023.stdout:1121: oid 46 updating version 0 to 303 2026-02-25T08:46:35.117 INFO:tasks.rados.rados.0.trial023.stdout:1121: oid 46 version 303 is already newer than 303 2026-02-25T08:46:35.117 INFO:tasks.rados.rados.0.trial023.stdout:1121: oid 46 version 303 is already newer than 300 2026-02-25T08:46:35.117 INFO:tasks.rados.rados.0.trial023.stdout:1121: oid 46 version 303 is already newer than 303 2026-02-25T08:46:35.117 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 46 v 303 (ObjNum 528 snap 143 seq_num 528) dirty exists 2026-02-25T08:46:35.117 INFO:tasks.rados.rados.0.trial023.stdout:1121: left oid 46 (ObjNum 528 snap 143 seq_num 528) 2026-02-25T08:46:35.117 INFO:tasks.rados.rados.0.trial023.stdout:1120: finishing write tid 3 to trial02342524-12 2026-02-25T08:46:35.117 INFO:tasks.rados.rados.0.trial023.stdout:1120: finishing write tid 4 to trial02342524-12 2026-02-25T08:46:35.117 INFO:tasks.rados.rados.0.trial023.stdout:1120: finishing write tid 5 to trial02342524-12 2026-02-25T08:46:35.117 INFO:tasks.rados.rados.0.trial023.stdout:1120: oid 12 updating version 0 to 192 2026-02-25T08:46:35.117 INFO:tasks.rados.rados.0.trial023.stdout:1120: oid 12 version 192 is already newer than 189 2026-02-25T08:46:35.118 INFO:tasks.rados.rados.0.trial023.stdout:1120: oid 12 version 192 is already newer than 192 2026-02-25T08:46:35.118 INFO:tasks.rados.rados.0.trial023.stdout:1120: oid 12 version 192 is already newer than 192 2026-02-25T08:46:35.118 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 12 v 192 (ObjNum 527 snap 143 seq_num 527) dirty exists 2026-02-25T08:46:35.118 INFO:tasks.rados.rados.0.trial023.stdout:1120: left oid 12 (ObjNum 527 snap 143 seq_num 527) 2026-02-25T08:46:35.118 INFO:tasks.rados.rados.0.trial023.stdout:1122: finishing write tid 1 to trial02342524-19 2026-02-25T08:46:35.118 INFO:tasks.rados.rados.0.trial023.stdout:1123: finishing write tid 1 to trial02342524-5 2026-02-25T08:46:35.118 INFO:tasks.rados.rados.0.trial023.stdout:1123: finishing write tid 2 to trial02342524-5 2026-02-25T08:46:35.118 INFO:tasks.rados.rados.0.trial023.stdout:1123: finishing write tid 3 to trial02342524-5 2026-02-25T08:46:35.118 INFO:tasks.rados.rados.0.trial023.stdout:1123: finishing write tid 4 to trial02342524-5 2026-02-25T08:46:35.118 INFO:tasks.rados.rados.0.trial023.stdout:1123: finishing write tid 5 to trial02342524-5 2026-02-25T08:46:35.118 INFO:tasks.rados.rados.0.trial023.stdout:1123: oid 5 updating version 0 to 42 2026-02-25T08:46:35.118 INFO:tasks.rados.rados.0.trial023.stdout:1123: oid 5 version 42 is already newer than 42 2026-02-25T08:46:35.119 INFO:tasks.rados.rados.0.trial023.stdout:1123: oid 5 version 42 is already newer than 42 2026-02-25T08:46:35.119 INFO:tasks.rados.rados.0.trial023.stdout:1123: oid 5 version 42 is already newer than 42 2026-02-25T08:46:35.119 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 5 v 42 (ObjNum 530 snap 143 seq_num 530) dirty exists 2026-02-25T08:46:35.119 INFO:tasks.rados.rados.0.trial023.stdout:1123: left oid 5 (ObjNum 530 snap 143 seq_num 530) 2026-02-25T08:46:35.119 INFO:tasks.rados.rados.0.trial023.stdout:1122: finishing write tid 2 to trial02342524-19 2026-02-25T08:46:35.119 INFO:tasks.rados.rados.0.trial023.stdout:1122: finishing write tid 3 to trial02342524-19 2026-02-25T08:46:35.119 INFO:tasks.rados.rados.0.trial023.stdout:1122: finishing write tid 4 to trial02342524-19 2026-02-25T08:46:35.119 INFO:tasks.rados.rados.0.trial023.stdout:1122: oid 19 updating version 0 to 189 2026-02-25T08:46:35.119 INFO:tasks.rados.rados.0.trial023.stdout:1122: oid 19 version 189 is already newer than 189 2026-02-25T08:46:35.119 INFO:tasks.rados.rados.0.trial023.stdout:1122: oid 19 version 189 is already newer than 187 2026-02-25T08:46:35.119 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 19 v 189 (ObjNum 529 snap 143 seq_num 529) dirty exists 2026-02-25T08:46:35.120 INFO:tasks.rados.rados.0.trial023.stdout:1122: left oid 19 (ObjNum 529 snap 143 seq_num 529) 2026-02-25T08:46:35.890 INFO:tasks.rados.rados.0.trial023.stdout:1119: done (6 left) 2026-02-25T08:46:35.890 INFO:tasks.rados.rados.0.trial023.stdout:1120: done (5 left) 2026-02-25T08:46:35.890 INFO:tasks.rados.rados.0.trial023.stdout:1121: done (4 left) 2026-02-25T08:46:35.891 INFO:tasks.rados.rados.0.trial023.stdout:1122: done (3 left) 2026-02-25T08:46:35.891 INFO:tasks.rados.rados.0.trial023.stdout:1123: done (2 left) 2026-02-25T08:46:35.891 INFO:tasks.rados.rados.0.trial023.stdout:1124: done (1 left) 2026-02-25T08:46:35.891 INFO:tasks.rados.rados.0.trial023.stdout:1126: done (0 left) 2026-02-25T08:46:35.891 INFO:tasks.rados.rados.0.trial023.stdout:1127: write (excl) oid 45 current snap is 144 2026-02-25T08:46:35.891 INFO:tasks.rados.rados.0.trial023.stdout:1127: seq_num 533 ranges {558131=652427,1673373=559846,2820808=137323} 2026-02-25T08:46:35.894 INFO:tasks.rados.rados.0.trial023.stdout:1127: writing trial02342524-45 from 558131 to 1210558 tid 1 2026-02-25T08:46:35.898 INFO:tasks.rados.rados.0.trial023.stdout:1127: writing trial02342524-45 from 1673373 to 2233219 tid 2 2026-02-25T08:46:35.901 INFO:tasks.rados.rados.0.trial023.stdout:1127: writing trial02342524-45 from 2820808 to 2958131 tid 3 2026-02-25T08:46:35.901 INFO:tasks.rados.rados.0.trial023.stdout:1128: snap_remove snap 141 2026-02-25T08:46:36.051 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:46:36.122 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:46:36.124 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:46:36.125 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:46:36.893 INFO:tasks.rados.rados.0.trial023.stdout:1128: done (1 left) 2026-02-25T08:46:36.894 INFO:tasks.rados.rados.0.trial023.stdout:1129: write oid 31 current snap is 144 2026-02-25T08:46:36.894 INFO:tasks.rados.rados.0.trial023.stdout:1127: finishing write tid 1 to trial02342524-45 2026-02-25T08:46:36.894 INFO:tasks.rados.rados.0.trial023.stdout:1129: seq_num 534 ranges {574938=727676,1731855=703504,2848231=126707} 2026-02-25T08:46:36.898 INFO:tasks.rados.rados.0.trial023.stdout:1129: writing trial02342524-31 from 574938 to 1302614 tid 1 2026-02-25T08:46:36.902 INFO:tasks.rados.rados.0.trial023.stdout:1129: writing trial02342524-31 from 1731855 to 2435359 tid 2 2026-02-25T08:46:36.905 INFO:tasks.rados.rados.0.trial023.stdout:1129: writing trial02342524-31 from 2848231 to 2974938 tid 3 2026-02-25T08:46:36.905 INFO:tasks.rados.rados.0.trial023.stdout:1130: read oid 40 snap -1 2026-02-25T08:46:36.905 INFO:tasks.rados.rados.0.trial023.stdout:1130: expect (ObjNum 472 snap 125 seq_num 472) 2026-02-25T08:46:36.905 INFO:tasks.rados.rados.0.trial023.stdout:1131: snap_remove snap 142 2026-02-25T08:46:37.088 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:46:37.165 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:37.162+0000 7f61a3c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:37.168 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:37.165+0000 7f61a3c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:37.183 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:46:37.207 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:37.208 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:37.621 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~297s 2026-02-25T08:46:37.897 INFO:tasks.rados.rados.0.trial023.stdout:1131: done (3 left) 2026-02-25T08:46:37.897 INFO:tasks.rados.rados.0.trial023.stdout:1132: snap_remove snap 126 2026-02-25T08:46:38.902 INFO:tasks.rados.rados.0.trial023.stdout:1132: done (3 left) 2026-02-25T08:46:38.902 INFO:tasks.rados.rados.0.trial023.stdout:1133: delete oid 30 current snap is 144 2026-02-25T08:46:38.902 INFO:tasks.rados.rados.0.trial023.stdout:1127: finishing write tid 2 to trial02342524-45 2026-02-25T08:46:38.902 INFO:tasks.rados.rados.0.trial023.stdout:1127: finishing write tid 3 to trial02342524-45 2026-02-25T08:46:38.902 INFO:tasks.rados.rados.0.trial023.stdout:1127: finishing write tid 4 to trial02342524-45 2026-02-25T08:46:38.903 INFO:tasks.rados.rados.0.trial023.stdout:1127: finishing write tid 5 to trial02342524-45 2026-02-25T08:46:38.903 INFO:tasks.rados.rados.0.trial023.stdout:1127: oid 45 updating version 0 to 252 2026-02-25T08:46:38.903 INFO:tasks.rados.rados.0.trial023.stdout:1127: oid 45 version 252 is already newer than 252 2026-02-25T08:46:38.903 INFO:tasks.rados.rados.0.trial023.stdout:1127: oid 45 version 252 is already newer than 249 2026-02-25T08:46:38.903 INFO:tasks.rados.rados.0.trial023.stdout:1127: oid 45 version 252 is already newer than 252 2026-02-25T08:46:38.903 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 45 v 252 (ObjNum 533 snap 144 seq_num 533) dirty exists 2026-02-25T08:46:38.903 INFO:tasks.rados.rados.0.trial023.stdout:1127: left oid 45 (ObjNum 533 snap 144 seq_num 533) 2026-02-25T08:46:38.903 INFO:tasks.rados.rados.0.trial023.stdout:1129: finishing write tid 1 to trial02342524-31 2026-02-25T08:46:38.903 INFO:tasks.rados.rados.0.trial023.stdout:1130: expect (ObjNum 472 snap 125 seq_num 472) 2026-02-25T08:46:38.988 INFO:tasks.rados.rados.0.trial023.stdout:1127: done (3 left) 2026-02-25T08:46:38.988 INFO:tasks.rados.rados.0.trial023.stdout:1130: done (2 left) 2026-02-25T08:46:38.988 INFO:tasks.rados.rados.0.trial023.stdout:1133: done (1 left) 2026-02-25T08:46:38.988 INFO:tasks.rados.rados.0.trial023.stdout:1134: snap_remove snap 130 2026-02-25T08:46:39.209 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:46:39.284 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:39.281+0000 7f6e027ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:39.286 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:39.284+0000 7f6e027ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:39.302 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:46:39.329 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:39.329 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:39.848 INFO:tasks.rados.rados.0.trial023.stdout:1134: done (1 left) 2026-02-25T08:46:39.848 INFO:tasks.rados.rados.0.trial023.stdout:1135: snap_create 2026-02-25T08:46:39.848 INFO:tasks.rados.rados.0.trial023.stdout:1129: finishing write tid 2 to trial02342524-31 2026-02-25T08:46:39.849 INFO:tasks.rados.rados.0.trial023.stdout:1129: finishing write tid 3 to trial02342524-31 2026-02-25T08:46:39.849 INFO:tasks.rados.rados.0.trial023.stdout:1129: finishing write tid 4 to trial02342524-31 2026-02-25T08:46:39.849 INFO:tasks.rados.rados.0.trial023.stdout:1129: finishing write tid 5 to trial02342524-31 2026-02-25T08:46:39.849 INFO:tasks.rados.rados.0.trial023.stdout:1129: oid 31 updating version 0 to 77 2026-02-25T08:46:39.849 INFO:tasks.rados.rados.0.trial023.stdout:1129: oid 31 version 77 is already newer than 77 2026-02-25T08:46:39.849 INFO:tasks.rados.rados.0.trial023.stdout:1129: oid 31 version 77 is already newer than 77 2026-02-25T08:46:39.849 INFO:tasks.rados.rados.0.trial023.stdout:1129: oid 31 version 77 is already newer than 77 2026-02-25T08:46:39.849 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 31 v 77 (ObjNum 534 snap 144 seq_num 534) dirty exists 2026-02-25T08:46:39.849 INFO:tasks.rados.rados.0.trial023.stdout:1129: left oid 31 (ObjNum 534 snap 144 seq_num 534) 2026-02-25T08:46:40.850 INFO:tasks.rados.rados.0.trial023.stdout:1129: done (1 left) 2026-02-25T08:46:40.850 INFO:tasks.rados.rados.0.trial023.stdout:1135: done (0 left) 2026-02-25T08:46:40.850 INFO:tasks.rados.rados.0.trial023.stdout:1136: write (excl) oid 23 current snap is 145 2026-02-25T08:46:40.850 INFO:tasks.rados.rados.0.trial023.stdout:1136: seq_num 536 ranges {608552=478174,1848819=707173} 2026-02-25T08:46:40.853 INFO:tasks.rados.rados.0.trial023.stdout:1136: writing trial02342524-23 from 608552 to 1086726 tid 1 2026-02-25T08:46:40.859 INFO:tasks.rados.rados.0.trial023.stdout:1136: writing trial02342524-23 from 1848819 to 2555992 tid 2 2026-02-25T08:46:40.860 INFO:tasks.rados.rados.0.trial023.stdout:1137: read oid 33 snap 133 2026-02-25T08:46:40.860 INFO:tasks.rados.rados.0.trial023.stdout:1137: expect (ObjNum 499 snap 133 seq_num 499) 2026-02-25T08:46:40.860 INFO:tasks.rados.rados.0.trial023.stdout:1138: snap_create 2026-02-25T08:46:40.882 INFO:tasks.rados.rados.0.trial023.stdout:1136: finishing write tid 1 to trial02342524-23 2026-02-25T08:46:40.902 INFO:tasks.rados.rados.0.trial023.stdout:1136: finishing write tid 2 to trial02342524-23 2026-02-25T08:46:40.911 INFO:tasks.rados.rados.0.trial023.stdout:1136: finishing write tid 3 to trial02342524-23 2026-02-25T08:46:40.913 INFO:tasks.rados.rados.0.trial023.stdout:1136: finishing write tid 4 to trial02342524-23 2026-02-25T08:46:40.914 INFO:tasks.rados.rados.0.trial023.stdout:1136: oid 23 updating version 0 to 224 2026-02-25T08:46:40.914 INFO:tasks.rados.rados.0.trial023.stdout:1136: oid 23 version 224 is already newer than 224 2026-02-25T08:46:40.914 INFO:tasks.rados.rados.0.trial023.stdout:1136: oid 23 version 224 is already newer than 222 2026-02-25T08:46:40.914 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 23 v 224 (ObjNum 536 snap 145 seq_num 536) dirty exists 2026-02-25T08:46:40.914 INFO:tasks.rados.rados.0.trial023.stdout:1136: left oid 23 (ObjNum 536 snap 145 seq_num 536) 2026-02-25T08:46:40.924 INFO:tasks.rados.rados.0.trial023.stdout:1137: expect (ObjNum 499 snap 133 seq_num 499) 2026-02-25T08:46:41.034 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:46:41.127 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:46:41.198 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:46:41.201 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:46:41.202 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:46:41.330 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:46:41.406 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:41.403+0000 7fd4e7dd3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:41.408 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:41.406+0000 7fd4e7dd3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:41.423 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:46:41.451 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:41.451 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:41.851 INFO:tasks.rados.rados.0.trial023.stdout:1136: done (2 left) 2026-02-25T08:46:41.852 INFO:tasks.rados.rados.0.trial023.stdout:1137: done (1 left) 2026-02-25T08:46:41.852 INFO:tasks.rados.rados.0.trial023.stdout:1138: done (0 left) 2026-02-25T08:46:41.852 INFO:tasks.rados.rados.0.trial023.stdout:1139: write oid 49 current snap is 146 2026-02-25T08:46:41.852 INFO:tasks.rados.rados.0.trial023.stdout:1139: seq_num 537 ranges {625359=553423,1790948=567184,2930500=94859} 2026-02-25T08:46:41.856 INFO:tasks.rados.rados.0.trial023.stdout:1139: writing trial02342524-49 from 625359 to 1178782 tid 1 2026-02-25T08:46:41.860 INFO:tasks.rados.rados.0.trial023.stdout:1139: writing trial02342524-49 from 1790948 to 2358132 tid 2 2026-02-25T08:46:41.863 INFO:tasks.rados.rados.0.trial023.stdout:1139: writing trial02342524-49 from 2930500 to 3025359 tid 3 2026-02-25T08:46:41.863 INFO:tasks.rados.rados.0.trial023.stdout:1140: write oid 48 current snap is 146 2026-02-25T08:46:41.863 INFO:tasks.rados.rados.0.trial023.stdout:1140: seq_num 538 ranges {642166=628672,1849430=710842} 2026-02-25T08:46:41.867 INFO:tasks.rados.rados.0.trial023.stdout:1140: writing trial02342524-48 from 642166 to 1270838 tid 1 2026-02-25T08:46:41.873 INFO:tasks.rados.rados.0.trial023.stdout:1140: writing trial02342524-48 from 1849430 to 2560272 tid 2 2026-02-25T08:46:41.873 INFO:tasks.rados.rados.0.trial023.stdout:1141: write (excl) oid 22 current snap is 146 2026-02-25T08:46:41.873 INFO:tasks.rados.rados.0.trial023.stdout:1141: seq_num 539 ranges {658973=703921,1907912=570853,2985346=73627} 2026-02-25T08:46:41.878 INFO:tasks.rados.rados.0.trial023.stdout:1141: writing trial02342524-22 from 658973 to 1362894 tid 1 2026-02-25T08:46:41.882 INFO:tasks.rados.rados.0.trial023.stdout:1141: writing trial02342524-22 from 1907912 to 2478765 tid 2 2026-02-25T08:46:41.884 INFO:tasks.rados.rados.0.trial023.stdout:1141: writing trial02342524-22 from 2985346 to 3058973 tid 3 2026-02-25T08:46:41.884 INFO:tasks.rados.rados.0.trial023.stdout:1142: snap_remove snap 140 2026-02-25T08:46:42.857 INFO:tasks.rados.rados.0.trial023.stdout:1142: done (3 left) 2026-02-25T08:46:42.858 INFO:tasks.rados.rados.0.trial023.stdout:1143: delete oid 32 current snap is 146 2026-02-25T08:46:42.858 INFO:tasks.rados.rados.0.trial023.stdout:1141: finishing write tid 1 to trial02342524-22 2026-02-25T08:46:42.858 INFO:tasks.rados.rados.0.trial023.stdout:1139: finishing write tid 1 to trial02342524-49 2026-02-25T08:46:42.858 INFO:tasks.rados.rados.0.trial023.stdout:1141: finishing write tid 2 to trial02342524-22 2026-02-25T08:46:42.858 INFO:tasks.rados.rados.0.trial023.stdout:1139: finishing write tid 2 to trial02342524-49 2026-02-25T08:46:42.858 INFO:tasks.rados.rados.0.trial023.stdout:1141: finishing write tid 3 to trial02342524-22 2026-02-25T08:46:42.858 INFO:tasks.rados.rados.0.trial023.stdout:1141: finishing write tid 4 to trial02342524-22 2026-02-25T08:46:42.858 INFO:tasks.rados.rados.0.trial023.stdout:1139: finishing write tid 3 to trial02342524-49 2026-02-25T08:46:42.858 INFO:tasks.rados.rados.0.trial023.stdout:1141: finishing write tid 5 to trial02342524-22 2026-02-25T08:46:42.858 INFO:tasks.rados.rados.0.trial023.stdout:1141: oid 22 updating version 0 to 70 2026-02-25T08:46:42.859 INFO:tasks.rados.rados.0.trial023.stdout:1141: oid 22 version 70 is already newer than 67 2026-02-25T08:46:42.859 INFO:tasks.rados.rados.0.trial023.stdout:1141: oid 22 version 70 is already newer than 70 2026-02-25T08:46:42.859 INFO:tasks.rados.rados.0.trial023.stdout:1141: oid 22 version 70 is already newer than 70 2026-02-25T08:46:42.859 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 22 v 70 (ObjNum 539 snap 146 seq_num 539) dirty exists 2026-02-25T08:46:42.859 INFO:tasks.rados.rados.0.trial023.stdout:1141: left oid 22 (ObjNum 539 snap 146 seq_num 539) 2026-02-25T08:46:42.859 INFO:tasks.rados.rados.0.trial023.stdout:1140: finishing write tid 1 to trial02342524-48 2026-02-25T08:46:42.859 INFO:tasks.rados.rados.0.trial023.stdout:1139: finishing write tid 4 to trial02342524-49 2026-02-25T08:46:42.859 INFO:tasks.rados.rados.0.trial023.stdout:1139: finishing write tid 5 to trial02342524-49 2026-02-25T08:46:42.859 INFO:tasks.rados.rados.0.trial023.stdout:1139: oid 49 updating version 0 to 206 2026-02-25T08:46:42.859 INFO:tasks.rados.rados.0.trial023.stdout:1139: oid 49 version 206 is already newer than 206 2026-02-25T08:46:42.859 INFO:tasks.rados.rados.0.trial023.stdout:1139: oid 49 version 206 is already newer than 206 2026-02-25T08:46:42.859 INFO:tasks.rados.rados.0.trial023.stdout:1139: oid 49 version 206 is already newer than 206 2026-02-25T08:46:42.860 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 49 v 206 (ObjNum 537 snap 146 seq_num 537) dirty exists 2026-02-25T08:46:42.860 INFO:tasks.rados.rados.0.trial023.stdout:1139: left oid 49 (ObjNum 537 snap 146 seq_num 537) 2026-02-25T08:46:42.860 INFO:tasks.rados.rados.0.trial023.stdout:1140: finishing write tid 2 to trial02342524-48 2026-02-25T08:46:42.860 INFO:tasks.rados.rados.0.trial023.stdout:1140: finishing write tid 3 to trial02342524-48 2026-02-25T08:46:42.860 INFO:tasks.rados.rados.0.trial023.stdout:1140: finishing write tid 4 to trial02342524-48 2026-02-25T08:46:42.860 INFO:tasks.rados.rados.0.trial023.stdout:1140: oid 48 updating version 0 to 237 2026-02-25T08:46:42.860 INFO:tasks.rados.rados.0.trial023.stdout:1140: oid 48 version 237 is already newer than 237 2026-02-25T08:46:42.860 INFO:tasks.rados.rados.0.trial023.stdout:1140: oid 48 version 237 is already newer than 237 2026-02-25T08:46:42.860 INFO:tasks.rados.rados.0.trial023.stdout:update_object_version oid 48 v 237 (ObjNum 538 snap 146 seq_num 538) dirty exists 2026-02-25T08:46:42.860 INFO:tasks.rados.rados.0.trial023.stdout:1140: left oid 48 (ObjNum 538 snap 146 seq_num 538) 2026-02-25T08:46:42.895 INFO:tasks.rados.rados.0.trial023.stdout:1139: done (3 left) 2026-02-25T08:46:42.896 INFO:tasks.rados.rados.0.trial023.stdout:1140: done (2 left) 2026-02-25T08:46:42.896 INFO:tasks.rados.rados.0.trial023.stdout:1141: done (1 left) 2026-02-25T08:46:42.896 INFO:tasks.rados.rados.0.trial023.stdout:1143: done (0 left) 2026-02-25T08:46:42.896 INFO:tasks.rados.rados.0.trial023.stdout:1144: read oid 19 snap -1 2026-02-25T08:46:42.896 INFO:tasks.rados.rados.0.trial023.stdout:1144: expect (ObjNum 529 snap 143 seq_num 529) 2026-02-25T08:46:42.896 INFO:tasks.rados.rados.0.trial023.stdout:1145: delete oid 16 current snap is 146 2026-02-25T08:46:42.910 INFO:tasks.rados.rados.0.trial023.stdout:1145: done (1 left) 2026-02-25T08:46:42.911 INFO:tasks.rados.rados.0.trial023.stdout:1146: snap_create 2026-02-25T08:46:42.915 INFO:tasks.rados.rados.0.trial023.stdout:1144: expect (ObjNum 529 snap 143 seq_num 529) 2026-02-25T08:46:42.916 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:46:42.983 INFO:tasks.rados.rados.0.trial023.stdout:seeking to 0 2026-02-25T08:46:43.125 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~303s 2026-02-25T08:46:43.125 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2026-02-25T08:46:43.452 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:46:43.529 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:43.526+0000 7f956445a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:43.531 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:46:43.529+0000 7f956445a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:43.548 INFO:teuthology.orchestra.run.trial023.stderr:Error ENXIO: problem getting command descriptions from osd.1 2026-02-25T08:46:43.576 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:43.576 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:43.627 INFO:tasks.ceph.osd.0:Sent signal 15 2026-02-25T08:46:43.628 INFO:tasks.ceph.osd.2:Sent signal 15 2026-02-25T08:46:43.628 INFO:tasks.ceph.osd.3:Sent signal 15 2026-02-25T08:46:43.629 INFO:tasks.ceph.mon.a:Sent signal 15 2026-02-25T08:46:43.629 INFO:tasks.ceph.mgr.x:Sent signal 15 2026-02-25T08:46:43.629 INFO:tasks.daemonwatchdog.daemon_watchdog:Killing thrasher OSDThrasher 2026-02-25T08:46:43.629 INFO:tasks.ceph.mon.a.trial023.stderr:2026-02-25T08:46:43.628+0000 7f5868bfc640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 40631) UID: 0 2026-02-25T08:46:43.629 INFO:tasks.ceph.mon.a.trial023.stderr:2026-02-25T08:46:43.628+0000 7f5868bfc640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2026-02-25T08:46:43.640 INFO:tasks.ceph.mgr.x.trial030.stderr:2026-02-25T08:46:43.639+0000 7f5fa9bf8640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster ceph -i x (PID: 39627) UID: 0 2026-02-25T08:46:43.640 INFO:tasks.ceph.mgr.x.trial030.stderr:2026-02-25T08:46:43.639+0000 7f5fa9bf8640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2026-02-25T08:46:46.202 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:46:46.273 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:46:46.276 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:46:46.277 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:46:51.278 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:46:51.350 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:46:51.353 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:46:51.353 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:46:56.354 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:46:56.425 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:46:56.427 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:46:56.428 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:47:01.429 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:47:01.501 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:47:01.503 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:47:01.504 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:47:06.506 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:47:06.578 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:47:06.581 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:47:06.582 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:47:11.583 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:47:11.654 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:47:11.657 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:47:11.658 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:47:16.659 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:47:16.731 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:47:16.734 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:47:16.735 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:47:21.735 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:47:21.804 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:47:21.807 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:47:21.807 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:47:26.808 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:47:26.879 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:47:26.882 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:47:26.883 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:47:31.884 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:47:31.954 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:47:31.957 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:47:31.957 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:47:36.958 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:47:37.030 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:47:37.033 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:47:37.034 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-25T08:47:42.035 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-25T08:47:42.105 INFO:teuthology.orchestra.run.trial023.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:47:42.108 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:47:42.112 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 192, in wrapper return func(self) ^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1513, in _do_thrash self.choose_action()() File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 562, in revive_osd self.ceph_manager.revive_osd( File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 3236, in revive_osd self.wait_run_admin_socket('osd', osd, File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2085, in wait_run_admin_socket raise Exception('timed out waiting for admin_socket ' Exception: timed out waiting for admin_socket to appear after osd.1 restart 2026-02-25T08:47:42.112 ERROR:tasks.thrashosds.thrasher:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1384, in do_thrash self._do_thrash() File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 192, in wrapper return func(self) ^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1513, in _do_thrash self.choose_action()() File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 562, in revive_osd self.ceph_manager.revive_osd( File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 3236, in revive_osd self.wait_run_admin_socket('osd', osd, File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2085, in wait_run_admin_socket raise Exception('timed out waiting for admin_socket ' Exception: timed out waiting for admin_socket to appear after osd.1 restart 2026-02-25T08:47:42.114 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2026-02-25T08:47:42.114 INFO:tasks.daemonwatchdog.daemon_watchdog:Killing remote process ceph-test-rados-ceph 2026-02-25T08:47:42.114 DEBUG:tasks.rados:Stopping ceph-test-rados-ceph due to exception The WatchDog barked due to Daemon ceph.osd.1 has failed 2026-02-25T08:47:42.114 INFO:tasks.rados:Stopping instance 0 2026-02-25T08:47:42.121 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 352, in thread raise watched_process.exception tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.1 has failed 2026-02-25T08:47:42.123 ERROR:teuthology.run_tasks:Manager failed: rados Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.1 has failed 2026-02-25T08:47:42.125 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2026-02-25T08:47:42.149 INFO:tasks.thrashosds:joining thrashosds 2026-02-25T08:47:42.150 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2026-02-25T08:47:42.150 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-25T08:47:42.150 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-25T08:47:42.236 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:47:42.232+0000 7fced5cc2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:49:42.192 DEBUG:teuthology.orchestra.run:got remote process result: 124 2026-02-25T08:49:42.193 ERROR:teuthology.run_tasks:Manager failed: thrashosds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/thrashosds.py", line 214, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.1 has failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 158, in __exit__ self.gen.throw(value) File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2970, in wait_for_all_osds_up while not self.are_all_osds_up(): ^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2960, in are_all_osds_up x = self.get_osd_dump() ^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2670, in get_osd_dump return self.get_osd_dump_json()['osds'] ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2662, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1713, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1704, in run_cluster_cmd return self.controller.run(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/remote.py", line 596, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 461, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial023 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2026-02-25T08:49:42.198 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-25T08:49:42.209 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-25T08:49:42.210 DEBUG:teuthology.orchestra.run.trial023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:49:42.287 INFO:teuthology.orchestra.run.trial023.stderr:2026-02-25T08:49:42.283+0000 7f82a4e11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:51:42.241 DEBUG:teuthology.orchestra.run:got remote process result: 124 2026-02-25T08:51:42.241 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph.py", line 2000, in task yield File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/thrashosds.py", line 214, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.1 has failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph.py", line 2010, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2936, in wait_for_clean num_active_clean = self.get_num_active_clean() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2715, in get_num_active_clean pgs = self.get_pg_stats() ^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2481, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1713, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1704, in run_cluster_cmd return self.controller.run(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/remote.py", line 596, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 461, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial023 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2026-02-25T08:51:42.243 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-25T08:51:42.243 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-25T08:51:42.243 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-25T08:51:42.243 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T08:51:42.331 INFO:tasks.ceph.osd.0:Stopped 2026-02-25T08:51:42.331 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-25T08:51:42.331 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T08:51:42.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:51:42.331 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph.py", line 2000, in task yield File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/thrashosds.py", line 214, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.1 has failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph.py", line 1525, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph.py", line 2010, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2936, in wait_for_clean num_active_clean = self.get_num_active_clean() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2715, in get_num_active_clean pgs = self.get_pg_stats() ^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2481, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1713, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1704, in run_cluster_cmd return self.controller.run(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/remote.py", line 596, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 461, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial023 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 485, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial023 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log' 2026-02-25T08:51:42.333 INFO:tasks.ceph.osd.1:Stopped 2026-02-25T08:51:42.333 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-25T08:51:42.333 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T08:51:42.412 INFO:tasks.ceph.osd.2:Stopped 2026-02-25T08:51:42.412 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2026-02-25T08:51:42.412 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T08:51:42.484 INFO:tasks.ceph.osd.3:Stopped 2026-02-25T08:51:42.484 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-25T08:51:42.484 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-25T08:51:42.484 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T08:51:42.485 INFO:tasks.ceph.mgr.x:Stopped 2026-02-25T08:51:42.485 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-25T08:51:42.485 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-25T08:51:42.485 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T08:51:42.485 INFO:tasks.ceph.mon.a:Stopped 2026-02-25T08:51:42.485 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-25T08:51:42.486 DEBUG:teuthology.orchestra.run.trial023:> sudo grep -E '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | grep -E -v '\(MDS_ALL_DOWN\)' | grep -E -v '\(MDS_UP_LESS_THAN_MAX\)' | grep -E -v '\(POOL_APP_NOT_ENABLED\)' | grep -E -v 'but it is still running' | grep -E -v 'objects unfound and apparently lost' | grep -E -v 'overall HEALTH_' | grep -E -v '\(OSDMAP_FLAGS\)' | grep -E -v '\(OSD_' | grep -E -v '\(PG_' | grep -E -v '\(POOL_' | grep -E -v '\(CACHE_POOL_' | grep -E -v '\(SMALLER_PGP_NUM\)' | grep -E -v '\(OBJECT_' | grep -E -v SLOW_OPS | grep -E -v '\(REQUEST_SLOW\)' | grep -E -v '\(TOO_FEW_PGS\)' | grep -E -v 'slow request' | grep -E -v 'timeout on replica' | grep -E -v 'late reservation from' | grep -E -v MON_DOWN | grep -E -v OSDMAP_FLAGS | grep -E -v OSD_DOWN | grep -E -v PG_DEGRADED | grep -E -v PG_AVAILABILITY | grep -E -v POOL_APP_NOT_ENABLED | grep -E -v 'mons down' | grep -E -v 'mon down' | grep -E -v 'out of quorum' | grep -E -v noscrub | grep -E -v nodeep-scrub | grep -E -v 'Degraded data redundancy' | grep -E -v 'is down' | grep -E -v 'osds down' | grep -E -v 'pg .*? is .*?degraded.*?, acting' | grep -E -v 'pg .*? is stuck' | grep -E -v 'pg degraded' | grep -E -v PG_BACKFILL_FULL | grep -E -v 'Low space hindering backfill .*? backfill_toofull' | grep -E -v OSD_ROOT_DOWN | grep -E -v 'pgs degraded' | grep -E -v 'pgs undersized' | grep -E -v 'is active.*backfill_toofull.*' | head -n 1 2026-02-25T08:51:42.515 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial023.front.sepia.ceph.com 2026-02-25T08:51:42.515 DEBUG:teuthology.orchestra.run.trial023:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-25T08:51:42.588 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial023.front.sepia.ceph.com 2026-02-25T08:51:42.588 DEBUG:teuthology.orchestra.run.trial023:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-25T08:51:42.653 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial030.front.sepia.ceph.com 2026-02-25T08:51:42.654 DEBUG:teuthology.orchestra.run.trial030:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-25T08:51:42.685 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@trial030.front.sepia.ceph.com 2026-02-25T08:51:42.685 DEBUG:teuthology.orchestra.run.trial030:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2026-02-25T08:51:42.753 INFO:tasks.ceph:Archiving mon data... 2026-02-25T08:51:42.755 DEBUG:teuthology.misc:Transferring archived files from trial023:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69065/data/mon.a.tgz 2026-02-25T08:51:42.755 DEBUG:teuthology.orchestra.run.trial023:> mktemp 2026-02-25T08:51:42.773 INFO:teuthology.orchestra.run.trial023.stdout:/tmp/tmp.2J5LZBEVow 2026-02-25T08:51:42.773 DEBUG:teuthology.orchestra.run.trial023:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.2J5LZBEVow 2026-02-25T08:51:42.882 DEBUG:teuthology.orchestra.run.trial023:> sudo chmod 0666 /tmp/tmp.2J5LZBEVow 2026-02-25T08:51:42.964 DEBUG:teuthology.orchestra.remote:trial023:/tmp/tmp.2J5LZBEVow is 384KB 2026-02-25T08:51:43.056 DEBUG:teuthology.orchestra.run.trial023:> rm -fr /tmp/tmp.2J5LZBEVow 2026-02-25T08:51:43.072 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-25T08:51:43.072 DEBUG:teuthology.orchestra.run.trial023:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-25T08:51:43.115 DEBUG:teuthology.orchestra.run.trial030:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-25T08:51:43.191 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-25T08:51:43.207 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-25T08:51:43.207 INFO:tasks.ceph:Archiving crash dumps... 2026-02-25T08:51:43.209 DEBUG:teuthology.misc:Transferring archived files from trial023:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69065/remote/trial023/crash 2026-02-25T08:51:43.210 DEBUG:teuthology.orchestra.run.trial023:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-25T08:51:43.256 DEBUG:teuthology.misc:Transferring archived files from trial030:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69065/remote/trial030/crash 2026-02-25T08:51:43.258 DEBUG:teuthology.orchestra.run.trial030:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-25T08:51:43.284 INFO:tasks.ceph:Compressing logs... 2026-02-25T08:51:43.284 DEBUG:teuthology.orchestra.run.trial023:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-25T08:51:43.301 DEBUG:teuthology.orchestra.run.trial030:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-25T08:51:43.322 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50869.log 2026-02-25T08:51:43.323 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47507.log 2026-02-25T08:51:43.337 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.50869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43724.log 2026-02-25T08:51:43.337 INFO:teuthology.orchestra.run.trial023.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.50869.log.gz 2026-02-25T08:51:43.337 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-25T08:51:43.337 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.47507.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47507.log.gz 2026-02-25T08:51:43.337 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.43724.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43724.log.gz 2026-02-25T08:51:43.337 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47129.log 2026-02-25T08:51:43.337 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48560.log 2026-02-25T08:51:43.337 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.47129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43346.log 2026-02-25T08:51:43.337 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.48560.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47129.log.gz 2026-02-25T08:51:43.337 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51247.log 2026-02-25T08:51:43.337 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48560.log.gz 2026-02-25T08:51:43.337 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.43346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42115.log 2026-02-25T08:51:43.337 INFO:teuthology.orchestra.run.trial023.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.43346.log.gz 2026-02-25T08:51:43.337 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.51247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43994.log 2026-02-25T08:51:43.337 INFO:teuthology.orchestra.run.trial023.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.51247.log.gz 2026-02-25T08:51:43.337 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.42115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47264.log 2026-02-25T08:51:43.337 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42115.log.gz 2026-02-25T08:51:43.337 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.43994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40720.log 2026-02-25T08:51:43.337 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43994.log.gz 2026-02-25T08:51:43.337 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.47264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47723.log 2026-02-25T08:51:43.337 INFO:teuthology.orchestra.run.trial023.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.47264.log.gz 2026-02-25T08:51:43.337 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.40720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50545.log 2026-02-25T08:51:43.337 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40720.log.gz 2026-02-25T08:51:43.337 INFO:teuthology.orchestra.run.trial023.stderr:gzip/var/log/ceph/ceph-client.admin.47723.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41001.log 2026-02-25T08:51:43.337 INFO:teuthology.orchestra.run.trial023.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.47723.log.gz 2026-02-25T08:51:43.337 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.50545.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-25T08:51:43.337 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50545.log.gz 2026-02-25T08:51:43.337 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.41001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41233.log 2026-02-25T08:51:43.337 INFO:teuthology.orchestra.run.trial023.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41001.log.gz 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44831.log 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.41233.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41233.log.gz 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46940.log 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45833.log 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.44831.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44831.log.gz 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.46940.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46940.log.gz 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50680.log 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.45833.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45833.log.gz 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44318.log 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42017.log 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.50680.log: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.50680.log.gz 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.44318.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44318.log.gz 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43805.log 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.42017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48290.log 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42017.log.gz 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.43805.log: 50.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.43805.log.gz -5 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr: --verbose -- /var/log/ceph/ceph-client.admin.48236.log 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.48290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47966.log 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48290.log.gz 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.48236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51898.log 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48236.log.gz 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.47966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42886.log 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47966.log.gz 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.51898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45616.log 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.51898.log.gz 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.42886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49546.log 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42886.log.gz 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.45616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41968.log 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45616.log.gz 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.49546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46022.log 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.49546.log.gz 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.41968.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41968.log.gz 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47318.log 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.46022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41133.log 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.46022.log.gz 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.47318.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47318.log.gz 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49762.log 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.41133.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41133.log.gz 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42317.log 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.49762.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.49762.log.gz 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48182.log 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.42317.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42317.log.gz 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40975.log 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.48182.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48182.log.gz 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51935.log 2026-02-25T08:51:43.338 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.40975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42470.log 2026-02-25T08:51:43.339 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40975.log.gz 2026-02-25T08:51:43.339 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.51935.log: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.51935.log.gz 2026-02-25T08:51:43.339 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48831.log 2026-02-25T08:51:43.339 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.42470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50086.log 2026-02-25T08:51:43.339 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42470.log.gz 2026-02-25T08:51:43.339 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.48831.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48831.log.gz 2026-02-25T08:51:43.339 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50734.log 2026-02-25T08:51:43.339 INFO:teuthology.orchestra.run.trial023.stderr:gzip/var/log/ceph/ceph-client.admin.50086.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.48506.log 2026-02-25T08:51:43.339 INFO:teuthology.orchestra.run.trial023.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.50086.log.gz 2026-02-25T08:51:43.339 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.50734.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50734.log.gz 2026-02-25T08:51:43.339 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50275.log 2026-02-25T08:51:43.339 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.48506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44048.log 2026-02-25T08:51:43.339 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48506.log.gz 2026-02-25T08:51:43.339 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.50275.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.50275.log.gz 2026-02-25T08:51:43.339 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48642.log 2026-02-25T08:51:43.339 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.44048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47912.log 2026-02-25T08:51:43.339 INFO:teuthology.orchestra.run.trial023.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44048.log.gz 2026-02-25T08:51:43.339 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.48642.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48642.log.gz 2026-02-25T08:51:43.339 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46400.log 2026-02-25T08:51:43.339 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43535.log 2026-02-25T08:51:43.339 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.47912.log: 50.4%/var/log/ceph/ceph-client.admin.46400.log: -- replaced with /var/log/ceph/ceph-client.admin.47912.log.gz 2026-02-25T08:51:43.339 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46400.log.gz 2026-02-25T08:51:43.339 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41706.log 2026-02-25T08:51:43.339 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.43535.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43535.log.gz 2026-02-25T08:51:43.339 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51058.log 2026-02-25T08:51:43.339 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.41706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47183.log 2026-02-25T08:51:43.339 INFO:teuthology.orchestra.run.trial023.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41706.log.gz 2026-02-25T08:51:43.339 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.51058.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.51058.log.gz 2026-02-25T08:51:43.339 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44183.log 2026-02-25T08:51:43.341 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.47183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43265.log 2026-02-25T08:51:43.341 INFO:teuthology.orchestra.run.trial023.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47183.log.gz 2026-02-25T08:51:43.341 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.44183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45048.log 2026-02-25T08:51:43.341 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44183.log.gz 2026-02-25T08:51:43.341 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.43265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.42524.log 2026-02-25T08:51:43.341 INFO:teuthology.orchestra.run.trial023.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43265.log.gz 2026-02-25T08:51:43.341 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.45048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50140.log 2026-02-25T08:51:43.341 INFO:teuthology.orchestra.run.trial023.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45048.log.gz 2026-02-25T08:51:43.341 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.0.42524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43454.log 2026-02-25T08:51:43.341 INFO:teuthology.orchestra.run.trial023.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.0.42524.log.gz 2026-02-25T08:51:43.341 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.50140.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50140.log.gz 2026-02-25T08:51:43.341 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43075.log 2026-02-25T08:51:43.341 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.43454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47453.log 2026-02-25T08:51:43.341 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43454.log.gz 2026-02-25T08:51:43.341 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.43075.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.43075.log.gz 2026-02-25T08:51:43.341 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41509.log 2026-02-25T08:51:43.341 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.47453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48885.log 2026-02-25T08:51:43.341 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47453.log.gz 2026-02-25T08:51:43.341 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.41509.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41509.log.gz 2026-02-25T08:51:43.341 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41507.log 2026-02-25T08:51:43.341 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.48885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46535.log 2026-02-25T08:51:43.341 INFO:teuthology.orchestra.run.trial023.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.48885.log.gz 2026-02-25T08:51:43.341 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.41507.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41507.log.gz 2026-02-25T08:51:43.341 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44372.log 2026-02-25T08:51:43.341 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.46535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45102.log 2026-02-25T08:51:43.341 INFO:teuthology.orchestra.run.trial023.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.46535.log.gz 2026-02-25T08:51:43.341 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.44372.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44372.log.gz 2026-02-25T08:51:43.341 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48696.log 2026-02-25T08:51:43.342 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.45102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47588.log 2026-02-25T08:51:43.342 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45102.log.gz 2026-02-25T08:51:43.342 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.48696.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48696.log.gz 2026-02-25T08:51:43.342 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49290.log 2026-02-25T08:51:43.342 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.47588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43589.log 2026-02-25T08:51:43.342 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47588.log.gz 2026-02-25T08:51:43.342 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.49290.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.49290.log.gz 2026-02-25T08:51:43.342 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44453.log 2026-02-25T08:51:43.342 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.43589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50005.log 2026-02-25T08:51:43.342 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43589.log.gz 2026-02-25T08:51:43.342 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.44453.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44453.log.gz 2026-02-25T08:51:43.342 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49074.log 2026-02-25T08:51:43.343 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.50005.log: gzip -5 49.6% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.50005.log.gz /var/log/ceph/ceph-client.admin.49627.log 2026-02-25T08:51:43.343 INFO:teuthology.orchestra.run.trial023.stderr: 2026-02-25T08:51:43.343 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.49074.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49074.log.gz 2026-02-25T08:51:43.343 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45941.log 2026-02-25T08:51:43.343 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.49627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44912.log 2026-02-25T08:51:43.343 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49627.log.gz 2026-02-25T08:51:43.343 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.45941.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45941.log.gz 2026-02-25T08:51:43.343 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49479.log 2026-02-25T08:51:43.343 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.44912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47399.log 2026-02-25T08:51:43.343 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44912.log.gz 2026-02-25T08:51:43.344 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.49479.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49479.log.gz 2026-02-25T08:51:43.344 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42940.log 2026-02-25T08:51:43.344 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.47399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50221.log 2026-02-25T08:51:43.344 INFO:teuthology.orchestra.run.trial023.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47399.log.gz 2026-02-25T08:51:43.344 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.42940.log: 48.4% -- replaced with /var/log/ceph/ceph-client.admin.42940.log.gz 2026-02-25T08:51:43.344 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42316.log 2026-02-25T08:51:43.344 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.50221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48371.log 2026-02-25T08:51:43.344 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50221.log.gz 2026-02-25T08:51:43.345 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.42316.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42316.log.gz 2026-02-25T08:51:43.345 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46265.log 2026-02-25T08:51:43.345 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.48371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41870.log 2026-02-25T08:51:43.345 INFO:teuthology.orchestra.run.trial023.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48371.log.gz 2026-02-25T08:51:43.345 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.46265.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.46265.log.gz 2026-02-25T08:51:43.345 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.39569.log 2026-02-25T08:51:43.345 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.41870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41062.log 2026-02-25T08:51:43.345 INFO:teuthology.orchestra.run.trial023.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.41870.log.gz 2026-02-25T08:51:43.346 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph.tmp-client.admin.39569.log: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.39569.log.gz 2026-02-25T08:51:43.346 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47642.log 2026-02-25T08:51:43.346 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.41062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42421.log 2026-02-25T08:51:43.346 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41062.log.gz 2026-02-25T08:51:43.346 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.47642.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47642.log.gz 2026-02-25T08:51:43.346 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46805.log 2026-02-25T08:51:43.346 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.42421.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42421.log.gz 2026-02-25T08:51:43.347 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50464.log 2026-02-25T08:51:43.347 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.46805.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46805.log.gz 2026-02-25T08:51:43.347 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41651.log 2026-02-25T08:51:43.347 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.50464.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50464.log.gz 2026-02-25T08:51:43.347 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48966.log 2026-02-25T08:51:43.348 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.41651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46157.log 2026-02-25T08:51:43.348 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41651.log.gz 2026-02-25T08:51:43.348 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.48966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48750.log 2026-02-25T08:51:43.348 INFO:teuthology.orchestra.run.trial023.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.48966.log.gz 2026-02-25T08:51:43.348 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.46157.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46157.log.gz 2026-02-25T08:51:43.348 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50788.log 2026-02-25T08:51:43.348 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.48750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50329.log 2026-02-25T08:51:43.348 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48750.log.gz 2026-02-25T08:51:43.349 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.50788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45887.log 2026-02-25T08:51:43.349 INFO:teuthology.orchestra.run.trial023.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.50788.log.gz 2026-02-25T08:51:43.349 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.50329.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50329.log.gz 2026-02-25T08:51:43.349 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50923.log 2026-02-25T08:51:43.349 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44642.log 2026-02-25T08:51:43.349 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.45887.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45887.log.gz 2026-02-25T08:51:43.349 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.50923.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-25T08:51:43.349 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50923.log.gz 2026-02-25T08:51:43.350 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.44642.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.44642.log.gz 2026-02-25T08:51:43.350 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48047.log 2026-02-25T08:51:43.350 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41331.log 2026-02-25T08:51:43.350 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.48047.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.48047.log.gz 2026-02-25T08:51:43.350 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45562.log 2026-02-25T08:51:43.350 INFO:teuthology.orchestra.run.trial023.stderr: 87.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-25T08:51:43.351 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.41331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43211.log 2026-02-25T08:51:43.351 INFO:teuthology.orchestra.run.trial023.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41331.log.gz 2026-02-25T08:51:43.351 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.45562.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45562.log.gz 2026-02-25T08:51:43.351 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43670.log 2026-02-25T08:51:43.351 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.43211.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43211.log.gz 2026-02-25T08:51:43.351 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42066.log 2026-02-25T08:51:43.351 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.43670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51112.log 2026-02-25T08:51:43.351 INFO:teuthology.orchestra.run.trial023.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43670.log.gz 2026-02-25T08:51:43.352 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.42066.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42066.log.gz 2026-02-25T08:51:43.352 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44588.log 2026-02-25T08:51:43.352 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.51112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44507.log 2026-02-25T08:51:43.352 INFO:teuthology.orchestra.run.trial023.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.51112.log.gz 2026-02-25T08:51:43.352 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.44588.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44588.log.gz 2026-02-25T08:51:43.352 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48101.log 2026-02-25T08:51:43.352 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.44507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43021.log 2026-02-25T08:51:43.352 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44507.log.gz 2026-02-25T08:51:43.353 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.48101.log: 48.4% -- replaced with /var/log/ceph/ceph-client.admin.48101.log.gz 2026-02-25T08:51:43.353 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43913.log 2026-02-25T08:51:43.353 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51382.log 2026-02-25T08:51:43.353 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.43021.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43021.log.gz 2026-02-25T08:51:43.353 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.43913.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43913.log.gz 2026-02-25T08:51:43.353 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41919.log 2026-02-25T08:51:43.353 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.51382.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-25T08:51:43.353 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51382.log.gz 2026-02-25T08:51:43.354 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.41919.log: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.41919.log.gz 2026-02-25T08:51:43.354 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43400.log 2026-02-25T08:51:43.354 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46346.log 2026-02-25T08:51:43.354 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.43400.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43400.log.gz 2026-02-25T08:51:43.354 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49951.log 2026-02-25T08:51:43.355 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.46346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50410.log 2026-02-25T08:51:43.355 INFO:teuthology.orchestra.run.trial023.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.46346.log.gz 2026-02-25T08:51:43.355 INFO:teuthology.orchestra.run.trial030.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-25T08:51:43.355 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.49951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41380.log 2026-02-25T08:51:43.355 INFO:teuthology.orchestra.run.trial023.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.49951.log.gz 2026-02-25T08:51:43.355 INFO:teuthology.orchestra.run.trial030.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39727.log 2026-02-25T08:51:43.355 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.50410.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50410.log.gz 2026-02-25T08:51:43.355 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41508.log 2026-02-25T08:51:43.355 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.41380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49020.log 2026-02-25T08:51:43.355 INFO:teuthology.orchestra.run.trial023.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41380.log.gz 2026-02-25T08:51:43.356 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2026-02-25T08:51:43.356 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.39727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39825.log 2026-02-25T08:51:43.356 INFO:teuthology.orchestra.run.trial030.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39727.log.gz 2026-02-25T08:51:43.356 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39879.log 2026-02-25T08:51:43.356 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.41508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44237.log 2026-02-25T08:51:43.356 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41508.log.gz 2026-02-25T08:51:43.356 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.49020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45481.log 2026-02-25T08:51:43.356 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49020.log.gz 2026-02-25T08:51:43.357 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.39825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39774.log 2026-02-25T08:51:43.357 INFO:teuthology.orchestra.run.trial030.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39825.log.gz 2026-02-25T08:51:43.357 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.39879.log: 50.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.39879.log.gz 2026-02-25T08:51:43.357 INFO:teuthology.orchestra.run.trial030.stderr: -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-25T08:51:43.357 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-client.admin.39774.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39774.log.gz 2026-02-25T08:51:43.357 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.44237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42318.log 2026-02-25T08:51:43.357 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44237.log.gz 2026-02-25T08:51:43.357 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.45481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43859.log 2026-02-25T08:51:43.357 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45481.log.gz 2026-02-25T08:51:43.357 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.42318.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42318.log.gz 2026-02-25T08:51:43.357 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40634.log 2026-02-25T08:51:43.357 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.43859.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43859.log.gz 2026-02-25T08:51:43.357 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47858.log 2026-02-25T08:51:43.357 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.40634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49870.log 2026-02-25T08:51:43.357 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40634.log.gz 2026-02-25T08:51:43.358 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.47858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46994.log 2026-02-25T08:51:43.358 INFO:teuthology.orchestra.run.trial023.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.47858.log.gz 2026-02-25T08:51:43.358 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.49870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41705.log 2026-02-25T08:51:43.358 INFO:teuthology.orchestra.run.trial023.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.49870.log.gz 2026-02-25T08:51:43.358 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.46994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44696.log 2026-02-25T08:51:43.358 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46994.log.gz 2026-02-25T08:51:43.358 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.41705.log: gzip -5 50.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.41705.log.gz /var/log/ceph/ceph-client.admin.41652.log 2026-02-25T08:51:43.358 INFO:teuthology.orchestra.run.trial023.stderr: 2026-02-25T08:51:43.359 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.44696.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44696.log.gz 2026-02-25T08:51:43.359 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51518.log 2026-02-25T08:51:43.359 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.41652.log: gzip 50.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.41652.log.gz --verbose 2026-02-25T08:51:43.359 INFO:teuthology.orchestra.run.trial023.stderr: -- /var/log/ceph/ceph-client.admin.40767.log 2026-02-25T08:51:43.359 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.51518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49209.log 2026-02-25T08:51:43.359 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51518.log.gz 2026-02-25T08:51:43.359 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.40767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45237.log 2026-02-25T08:51:43.360 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40767.log.gz 2026-02-25T08:51:43.360 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.49209.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49209.log.gz 2026-02-25T08:51:43.360 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51463.log 2026-02-25T08:51:43.361 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.45237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46616.log 2026-02-25T08:51:43.361 INFO:teuthology.orchestra.run.trial023.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45237.log.gz 2026-02-25T08:51:43.361 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.51463.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51463.log.gz 2026-02-25T08:51:43.361 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49425.log 2026-02-25T08:51:43.361 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.46616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46724.log 2026-02-25T08:51:43.361 INFO:teuthology.orchestra.run.trial023.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.46616.log.gz 2026-02-25T08:51:43.362 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.49425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44966.log 2026-02-25T08:51:43.362 INFO:teuthology.orchestra.run.trial023.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.49425.log.gz 2026-02-25T08:51:43.362 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.46724.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46724.log.gz 2026-02-25T08:51:43.362 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45751.log 2026-02-25T08:51:43.362 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.44966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42568.log 2026-02-25T08:51:43.362 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44966.log.gz 2026-02-25T08:51:43.362 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.45751.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45751.log.gz 2026-02-25T08:51:43.362 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42191.log 2026-02-25T08:51:43.362 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.42568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47777.log 2026-02-25T08:51:43.363 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42568.log.gz 2026-02-25T08:51:43.363 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.42191.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42191.log.gz 2026-02-25T08:51:43.363 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51328.log 2026-02-25T08:51:43.363 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.47777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41510.log 2026-02-25T08:51:43.363 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47777.log.gz 2026-02-25T08:51:43.363 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.51328.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51328.log.gz 2026-02-25T08:51:43.363 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43129.log 2026-02-25T08:51:43.363 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.41510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51004.log 2026-02-25T08:51:43.363 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41510.log.gz 2026-02-25T08:51:43.363 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.43129.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43129.log.gz 2026-02-25T08:51:43.364 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41821.log 2026-02-25T08:51:43.364 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.51004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49344.log 2026-02-25T08:51:43.364 INFO:teuthology.orchestra.run.trial023.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.51004.log.gz 2026-02-25T08:51:43.364 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.41821.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41821.log.gz 2026-02-25T08:51:43.364 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46859.log 2026-02-25T08:51:43.364 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.49344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45427.log 2026-02-25T08:51:43.364 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49344.log.gz 2026-02-25T08:51:43.364 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.46859.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46859.log.gz 2026-02-25T08:51:43.365 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-25T08:51:43.365 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.45427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41282.log 2026-02-25T08:51:43.365 INFO:teuthology.orchestra.run.trial023.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45427.log.gz 2026-02-25T08:51:43.365 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50599.log 2026-02-25T08:51:43.365 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.41282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51193.log 2026-02-25T08:51:43.365 INFO:teuthology.orchestra.run.trial023.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41282.log.gz 2026-02-25T08:51:43.366 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.50599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45156.log 2026-02-25T08:51:43.366 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50599.log.gz 2026-02-25T08:51:43.366 INFO:teuthology.orchestra.run.trial023.stderr: 87.9% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-25T08:51:43.366 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.51193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42627.log 2026-02-25T08:51:43.366 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51193.log.gz 2026-02-25T08:51:43.366 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.45156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46481.log 2026-02-25T08:51:43.366 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45156.log.gz 2026-02-25T08:51:43.366 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.42627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45373.log 2026-02-25T08:51:43.366 INFO:teuthology.orchestra.run.trial023.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42627.log.gz 2026-02-25T08:51:43.367 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.46481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49681.log 2026-02-25T08:51:43.367 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46481.log.gz 2026-02-25T08:51:43.367 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.45373.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45373.log.gz 2026-02-25T08:51:43.367 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48425.log 2026-02-25T08:51:43.367 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.49681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44777.log 2026-02-25T08:51:43.367 INFO:teuthology.orchestra.run.trial023.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.49681.log.gz 2026-02-25T08:51:43.367 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.48425.log: gzip -5 --verbose 50.0% -- /var/log/ceph/ceph-client.admin.46670.log -- replaced with /var/log/ceph/ceph-client.admin.48425.log.gz 2026-02-25T08:51:43.367 INFO:teuthology.orchestra.run.trial023.stderr: 2026-02-25T08:51:43.368 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.44777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41160.log 2026-02-25T08:51:43.368 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44777.log.gz 2026-02-25T08:51:43.368 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.46670.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46670.log.gz 2026-02-25T08:51:43.368 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49155.log 2026-02-25T08:51:43.368 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.41160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49816.log 2026-02-25T08:51:43.368 INFO:teuthology.orchestra.run.trial023.stderr: 66.9% -- replaced with /var/log/ceph/ceph-client.admin.41160.log.gz 2026-02-25T08:51:43.368 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.49155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45291.log 2026-02-25T08:51:43.368 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49155.log.gz 2026-02-25T08:51:43.369 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.49816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44129.log 2026-02-25T08:51:43.369 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49816.log.gz 2026-02-25T08:51:43.369 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.45291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45697.log 2026-02-25T08:51:43.369 INFO:teuthology.orchestra.run.trial023.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45291.log.gz 2026-02-25T08:51:43.369 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.44129.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44129.log.gz 2026-02-25T08:51:43.369 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46076.log 2026-02-25T08:51:43.369 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.45697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46211.log 2026-02-25T08:51:43.370 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45697.log.gz 2026-02-25T08:51:43.370 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.46076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47048.log 2026-02-25T08:51:43.370 INFO:teuthology.orchestra.run.trial023.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46076.log.gz 2026-02-25T08:51:43.370 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.46211.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46211.log.gz 2026-02-25T08:51:43.370 INFO:teuthology.orchestra.run.trial023.stderr:/var/log/ceph/ceph-client.admin.47048.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47048.log.gz 2026-02-25T08:51:43.387 INFO:teuthology.orchestra.run.trial030.stderr:/var/log/ceph/ceph-mgr.x.log: 91.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-25T08:51:43.467 INFO:teuthology.orchestra.run.trial023.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-25T08:51:43.473 INFO:teuthology.orchestra.run.trial023.stderr: 89.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-25T08:51:44.271 INFO:teuthology.orchestra.run.trial030.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-25T08:51:44.623 INFO:teuthology.orchestra.run.trial030.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2026-02-25T08:51:44.624 INFO:teuthology.orchestra.run.trial030.stderr: 2026-02-25T08:51:44.624 INFO:teuthology.orchestra.run.trial030.stderr:real 0m1.278s 2026-02-25T08:51:44.624 INFO:teuthology.orchestra.run.trial030.stderr:user 0m2.093s 2026-02-25T08:51:44.624 INFO:teuthology.orchestra.run.trial030.stderr:sys 0m0.137s 2026-02-25T08:51:44.965 INFO:teuthology.orchestra.run.trial023.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-25T08:51:44.967 INFO:teuthology.orchestra.run.trial023.stderr: 2026-02-25T08:51:44.967 INFO:teuthology.orchestra.run.trial023.stderr:real 0m1.652s 2026-02-25T08:51:44.967 INFO:teuthology.orchestra.run.trial023.stderr:user 0m1.861s 2026-02-25T08:51:44.967 INFO:teuthology.orchestra.run.trial023.stderr:sys 0m0.168s 2026-02-25T08:51:44.968 INFO:tasks.ceph:Archiving logs... 2026-02-25T08:51:44.968 DEBUG:teuthology.misc:Transferring archived files from trial023:/var/log/ceph to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69065/remote/trial023/log 2026-02-25T08:51:44.970 DEBUG:teuthology.orchestra.run.trial023:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-25T08:51:45.304 DEBUG:teuthology.misc:Transferring archived files from trial030:/var/log/ceph to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69065/remote/trial030/log 2026-02-25T08:51:45.304 DEBUG:teuthology.orchestra.run.trial030:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-25T08:51:45.530 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph.py", line 2000, in task yield File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/thrashosds.py", line 214, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.1 has failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 158, in __exit__ self.gen.throw(value) File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph.py", line 2010, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2936, in wait_for_clean num_active_clean = self.get_num_active_clean() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2715, in get_num_active_clean pgs = self.get_pg_stats() ^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2481, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1713, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1704, in run_cluster_cmd return self.controller.run(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/remote.py", line 596, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 461, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial023 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2026-02-25T08:51:45.531 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-25T08:51:45.544 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/task/install/__init__.py", line 644, in task yield File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph.py", line 2000, in task yield File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/thrashosds.py", line 214, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.1 has failed 2026-02-25T08:51:45.660 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-25T08:51:45.672 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/task/install/__init__.py", line 644, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.1 has failed 2026-02-25T08:51:45.673 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-25T08:51:45.673 DEBUG:teuthology.orchestra.run.trial023:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-25T08:51:45.675 DEBUG:teuthology.orchestra.run.trial030:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-25T08:51:45.700 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-25T08:51:45.712 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-25T08:51:45.712 DEBUG:teuthology.orchestra.run.trial023:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-25T08:51:45.740 DEBUG:teuthology.orchestra.run.trial030:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-25T08:51:45.752 INFO:teuthology.orchestra.run.trial023.stderr:bash: line 1: ntpq: command not found 2026-02-25T08:51:45.755 INFO:teuthology.orchestra.run.trial030.stderr:bash: line 1: ntpq: command not found 2026-02-25T08:51:45.792 INFO:teuthology.orchestra.run.trial023.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-25T08:51:45.792 INFO:teuthology.orchestra.run.trial023.stdout:=============================================================================== 2026-02-25T08:51:45.792 INFO:teuthology.orchestra.run.trial023.stdout:^- hou1.us.ntp.li 2 9 207 146 +2079us[+2079us] +/- 59ms 2026-02-25T08:51:45.792 INFO:teuthology.orchestra.run.trial023.stdout:^- ip216-250-115-174.pbiaas> 2 9 377 20 -530us[ -530us] +/- 113ms 2026-02-25T08:51:45.792 INFO:teuthology.orchestra.run.trial023.stdout:^* ntp4.glypnod.com 2 8 377 275 -1534us[-1502us] +/- 16ms 2026-02-25T08:51:45.792 INFO:teuthology.orchestra.run.trial023.stdout:^- 172-232-15-202.ip.linode> 2 6 377 13 -6879us[-6879us] +/- 76ms 2026-02-25T08:51:45.794 INFO:teuthology.orchestra.run.trial030.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-25T08:51:45.795 INFO:teuthology.orchestra.run.trial030.stdout:=============================================================================== 2026-02-25T08:51:45.795 INFO:teuthology.orchestra.run.trial030.stdout:^+ ip216-250-115-174.pbiaas> 2 9 377 17 +476us[ +476us] +/- 113ms 2026-02-25T08:51:45.795 INFO:teuthology.orchestra.run.trial030.stdout:^- ntp2.zeus.frumentum.media 2 7 377 20 -104ms[ -104ms] +/- 130ms 2026-02-25T08:51:45.795 INFO:teuthology.orchestra.run.trial030.stdout:^+ 172-104-28-175.ip.linode> 2 6 377 17 -5141us[-5141us] +/- 57ms 2026-02-25T08:51:45.795 INFO:teuthology.orchestra.run.trial030.stdout:^* ns2.fortrockdc.com 2 6 377 149 +3008us[+3187us] +/- 40ms 2026-02-25T08:51:45.795 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-25T08:51:45.807 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-25T08:51:45.807 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-25T08:51:45.821 DEBUG:teuthology.orchestra.run.trial023:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-25T08:51:45.889 DEBUG:teuthology.orchestra.run.trial030:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-25T08:51:45.945 DEBUG:teuthology.orchestra.run.trial023:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-25T08:51:45.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:51:45.974 DEBUG:teuthology.orchestra.run.trial030:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-25T08:51:46.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:51:46.000 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-25T08:51:46.019 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-25T08:51:46.031 INFO:teuthology.task.internal:Duration was 2829.777500 seconds 2026-02-25T08:51:46.031 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-25T08:51:46.049 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-25T08:51:46.049 DEBUG:teuthology.orchestra.run.trial023:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-25T08:51:46.051 DEBUG:teuthology.orchestra.run.trial030:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-25T08:51:46.086 INFO:teuthology.orchestra.run.trial023.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-25T08:51:46.087 INFO:teuthology.orchestra.run.trial030.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-25T08:51:46.353 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-25T08:51:46.353 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial023.front.sepia.ceph.com 2026-02-25T08:51:46.353 DEBUG:teuthology.orchestra.run.trial023:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-25T08:51:46.373 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial030.front.sepia.ceph.com 2026-02-25T08:51:46.373 DEBUG:teuthology.orchestra.run.trial030:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-25T08:51:46.410 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-25T08:51:46.410 DEBUG:teuthology.orchestra.run.trial023:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-25T08:51:46.415 DEBUG:teuthology.orchestra.run.trial030:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-25T08:51:46.512 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-25T08:51:46.513 DEBUG:teuthology.orchestra.run.trial023:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-25T08:51:46.515 DEBUG:teuthology.orchestra.run.trial030:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-25T08:51:46.536 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T08:51:46.537 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T08:51:46.538 INFO:teuthology.orchestra.run.trial023.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-25T08:51:46.538 INFO:teuthology.orchestra.run.trial023.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-25T08:51:46.538 INFO:teuthology.orchestra.run.trial023.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-25T08:51:46.539 INFO:teuthology.orchestra.run.trial030.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T08:51:46.539 INFO:teuthology.orchestra.run.trial030.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T08:51:46.539 INFO:teuthology.orchestra.run.trial030.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-25T08:51:46.540 INFO:teuthology.orchestra.run.trial030.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-25T08:51:46.540 INFO:teuthology.orchestra.run.trial030.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-25T08:51:46.547 INFO:teuthology.orchestra.run.trial023.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 83.9% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-25T08:51:46.548 INFO:teuthology.orchestra.run.trial030.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 82.3% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-25T08:51:46.549 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-25T08:51:46.562 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-25T08:51:46.562 DEBUG:teuthology.orchestra.run.trial023:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-25T08:51:46.612 DEBUG:teuthology.orchestra.run.trial030:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-25T08:51:46.638 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-25T08:51:46.654 DEBUG:teuthology.orchestra.run.trial023:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-25T08:51:46.657 DEBUG:teuthology.orchestra.run.trial030:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-25T08:51:46.679 INFO:teuthology.orchestra.run.trial023.stdout:kernel.core_pattern = core 2026-02-25T08:51:46.701 INFO:teuthology.orchestra.run.trial030.stdout:kernel.core_pattern = core 2026-02-25T08:51:46.720 DEBUG:teuthology.orchestra.run.trial023:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-25T08:51:46.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:51:46.753 DEBUG:teuthology.orchestra.run.trial030:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-25T08:51:46.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:51:46.776 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-25T08:51:46.788 INFO:teuthology.task.internal:Transferring archived files... 2026-02-25T08:51:46.789 DEBUG:teuthology.misc:Transferring archived files from trial023:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69065/remote/trial023 2026-02-25T08:51:46.789 DEBUG:teuthology.orchestra.run.trial023:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-25T08:51:46.824 DEBUG:teuthology.misc:Transferring archived files from trial030:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69065/remote/trial030 2026-02-25T08:51:46.824 DEBUG:teuthology.orchestra.run.trial030:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-25T08:51:46.860 INFO:teuthology.task.internal:Removing archive directory... 2026-02-25T08:51:46.861 DEBUG:teuthology.orchestra.run.trial023:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-25T08:51:46.863 DEBUG:teuthology.orchestra.run.trial030:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-25T08:51:46.908 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-25T08:51:46.927 INFO:teuthology.task.internal:Not uploading archives. 2026-02-25T08:51:46.927 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-25T08:51:46.945 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-25T08:51:46.945 DEBUG:teuthology.orchestra.run.trial023:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-25T08:51:46.948 DEBUG:teuthology.orchestra.run.trial030:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-25T08:51:46.964 INFO:teuthology.orchestra.run.trial023.stdout: 920834 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 25 08:51 /home/ubuntu/cephtest 2026-02-25T08:51:46.965 INFO:teuthology.orchestra.run.trial030.stdout: 920834 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 25 08:51 /home/ubuntu/cephtest 2026-02-25T08:51:46.966 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-25T08:51:46.985 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-25T08:51:47.004 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: BarkError: The WatchDog barked due to Daemon ceph.osd.1 has failed 2026-02-25T08:51:47.008 INFO:teuthology.run:Summary data: description: crimson-rados/thrash/{0-crimson_install 0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} clusters/{crimson-fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm} thrashers/{default} thrashosds-health workloads/snaps-few-objects} duration: 2829.7774996757507 failure_reason: The WatchDog barked due to Daemon ceph.osd.1 has failed flavor: debug owner: scheduled_matan@soko04.front.sepia.ceph.com status: fail success: false 2026-02-25T08:51:47.009 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-25T08:51:47.122 INFO:teuthology.run:FAIL